Scaling-up evidence-based practices in child welfare services systems
Authors: Greg Aarons, PhD and Mark Chaffin, PhD
Thousands of families annually receive services through the child welfare system. Many have mental health or other service needs (Burns, Phillips, Wagner, Barth, Kolko, Campbell & Landsverk, 2004). Child welfare systems may refer clients to traditional mental health or substance abuse services that are not operated by child welfare. However, there is also a sizeable service system that is directly managed by child welfare, often under a contracted network of community-based organizations (CBOs) that provide special purpose programs. For example, child welfare may define the specific program being purchased, such as a home visiting family preservation program or a parenting skills program, and purchase that service via bids and contracts to multiple CBOs. In many cases, the child welfare system not only finances networks of CBOs, but also controls referral into the system and audits quality and outcomes. This constitutes a large and little studied non-traditional public sector service system, and one that is rapidly moving toward evidence-based practice (EBP) implementation and scale-up. In this article, we will briefly outline some of the themes for scaling up EBPs in these systems. Most of the work discussed here takes place in the child welfare system but predominantly in CBOs that provide an array of services including child abuse and neglect interventions, mental health and substance abuse treatment, and other allied health care services. These services may take place in child welfare, mental health, substance abuse treatment, primary care or school settings. As such, this work occurs in broad public sector service settings that involve various relationships among systems, agencies, providers, and most importantly, clients that may receive services in multiple sectors (Garland, Hough, Landsverk, & Brown, 2001; Horwitz, Hurlburt, Goldhaber-Fiebert, 2012). Indeed child welfare system involvement may serve as a gateway for identification of, and services for mental health problems of youth and parents.
Several features of child welfare services systems facilitate EBP uptake and diffusion. Services are purchased and managed at scale—these often are countywide or statewide initiatives. Specific programs are purchased (e.g. a parenting program) rather than ad hoc or individualized services. Contractors must accept specific service parameters, potentially including agreement to implement a designated EBP model and participate in its quality control protocol. Because the system is not necessarily tied to traditional financing streams (e.g. unit-rate fee-for-service billing, which incentivizes volume rather than quality or outcomes), financing can be more agile and fitted precisely to the EBP model (e.g. a pay-for-value approach, financed to accommodate a designated, quality controlled protocol). And because child welfare can be both the designer and consumer of services, high utilization can be achieved without marketing or grooming referral networks.
Other system features may create implementation challenges. Few EBPs were designed with the child welfare system in mind, and adaptation may be required. For example, an EBP parenting model designed for disruptive child behavior may need adaptation to be effective with abusive parents (Chaffin, et al., 2011). We recently proposed a scale-up model designed to deal with uncertainty during the implementation process and to guide adaptation of system, organization and intervention elements during scale up (Aarons, Green, Palinkas, Self-Brown, Whitaker, Lutzker, Silovsky, Hecht & Chaffin, 2012). Systems can be vulnerable to disruption by policy changes, class-action lawsuits and frequent leadership changes. Child welfare systems attract frequent and often negative scrutiny from the media and the public, making innovation risky. For example, even if a new EBP model results in better outcomes, a single high-profile negative outcome in the press might jeopardize not only the scale-up but current agency administration.
A review of EBPs used by child welfare is beyond the scope of this article, but we believe that the current evidence is encouraging, including at least one statewide scale-up in Oklahoma (Chaffin, et al., 2012), and a multi-county scale-up in California (Aarons, Green, et al., 2012). Knowledge about how to scale-up has relied on “lessons learned” and anecdote, but is rapidly becoming informed by research (Aarons, Fettes, Sommerfeld, Palinkas, 2012).
Theme One — Less can be more
When scaling-up within an existing network, the less change required, the more implementation may occur. Implementation conceptual models highlight complexity and multiple levels at which change occurs — the individual provider or workforce, the organization, and the outer system context (Aarons, Hurlburt, Horwitz, 2011; Damschroder, et al., 2009; Glisson & Schoenwald, 2005; Greenhalgh et al., 2004; Southam-Gerow, Rodriguez, Chorpita, & Daleiden, 2012). Successful scale-up requires at least some adjustment across all levels. Achieving change at some levels is relatively simple — for example, training providers. At other levels, it can be complicated — for example, creating a new quality control structure within an organization; changing organizational culture; or changing service expectations among court systems. Still other changes may be so difficult that they would derail implementation, such as changing the minimum degree qualifications for a large existing workforce. One recommendation is to select EBPs that have not only good empirical support, but also “fit” the existing workforce, organizations, cultures, financial resources, referral and utilization patterns, and service modalities (Damschroder et al., 2009; Klein & Sorra, 1996). For example, Oklahoma and San Diego County, Calif., implemented the SafeCare© model within a large existing home-based child welfare services network. This required limited or no change in service modality (home-based), workforce qualifications, service dose, budgets, utilization, capacity or the general ways in which the CBOs construed their mission. While some changes were required in funding streams and contract language, the largest changes were limited to provider behavior (i.e. the techniques used) and quality control.
Theme Two — Politics and relationships matter
Collaboration is important, but collaboration is perhaps a sanitized term that fails to capture the full process. Scaling-up can involve political and interpersonal processes. There will be shared goals, but also competing interests and agendas, including competing interests over who controls services and resources. The key is being able to craft a solution that the majority will endorse, that the minority will tolerate, and that leaves as few as possible feeling left out or dissatisfied with the outcome. At some juncture, this may involve exercise of leadership authority along with compromise and recognition that political processes are complex (Aarons, Hurlburt, Willging, et al., In review).
Theme Three — Leadership matters
Leadership matters for any implementation effort, but especially for scaling up. Leadership is required to ensure that necessary resources are provided, that financing is adequate and to unfreeze policies that hinder innovation. Beyond material or policy supports, leadership styles and attitudes directly influence buy-in. Transformational leadership, which refers to leaders who inspire and stimulate vigorous interest in the scale-up, creates more favorable front-line provider attitudes toward EBP and innovation (Aarons & Sommerfeld, 2012).
Theme Four — Be persistent
Even if there is an initially skeptical response, EBP implementation can win over most front-line providers. This can be thought of as a culture change which takes time (Glisson, Hemmelgarn, Dukes, Atkinson, & Williams, 2012). Scaling-up may involve hundreds of individual front-line providers. Most will have had little influence on the EBP adoption decision that will profoundly impact how they do their own jobs. For many, the adoption decision will feel like, “just another requirement handed down from on high,” a reaction that sets the stage for implementing on paper, but not in actual practice. Plus, because EBPs invariably involve more structure, quality control, closer scrutiny and possibly more documentation, change may seem burdensome at first. In our experience, negative reactions are real but not permanent. An unexpected benefit found in the Oklahoma scale-up was that front-line providers assigned to implement the EBP under greater quality control scrutiny (live practice observation) had lower job turnover than those assigned to usual services or quality control (Aarons, Sommerfeld, Hecht, Silovsky & Chaffin, 2009), and experienced less emotional exhaustion (Aarons, Fettes, Flores & Sommerfeld, 2009). EBP scale-up promotes professional growth and self-efficacy, especially if providers eventually see improved client outcomes (Aarons & Palinkas, 2007). Growth and efficacy drive job satisfaction. Communicating these messages about the implementation may be critical, along with persisting until satisfaction eclipses skepticism.
Theme Five — Localize expertise
System and CBO leaders dislike long-term dependency on remote developers or purveyors. Systems and agencies want to develop their own internal capacity for new-hire training and quality control. It is important to create a sense of local EBP ownership. At the same time, systems and CBOs desire some connection with EBP researchers and developers in order to keep abreast of new developments and to make their own contribution to a larger community of innovation. One of the benefits of scaling up is that scale creates powerful laboratories for research and development via research-practice partnerships. The key is achieving a balance between localized semi-autonomous expertise, and sustained engagement with researchers and developers. One example of this is the development of local inter-agency collaborative teams (ICTs), which are groups of highly expert providers, based at different agencies within a network, who serve local training and quality control functions and who liaison between their fellow front-line providers and EBP researchers or organizations (Hurlburt, et al., In review).
Theme Six — Stay in it for the long haul
Planning is important, but scaling up is a process that requires ongoing attention. Implementations traverse a series of developmental stages, from exploration to preparation to active implementation to sustainment (Aarons, Hurlburt & Horwitz, 2011). During exploration and preparation stages, system leadership plays a key role, often in concert with model experts or consultants. As planning moves to include logistics and quality control, CBO leadership and supervisors may become more involved. Once training begins, front-line providers take the focus. Immediately after training, clients become the critical element — there must be sufficient EBP cases ready to go so that providers can practice newly learned skills. It is important to anticipate problems at each stage of this process. This includes obstacles that the EBP itself may create, but which were not anticipated. Scale-up disrupts existing system equilibrium and it is likely that no amount of planning will anticipate every problem. Addressing emergent problems to craft solutions requires agility and the capacity to convene troubleshooting “implementation resource teams” that engage stakeholders at all levels of the overall contextual ecology (Aarons, Green et al., 2012).
Theme Seven — Formalize and institutionalize
Scaling up may bring an injection of new one-time start-up resources. CBOs and systems may sign-on to the implementation in order to capture these start-up resources. Ultimately, it is not desirable for systems to chase a perpetual chain of demonstration grants, where each new project supplants a prior one. For the scale-up to be sustained, it should endure without needing a series of new budget supplements. It should become institutionalized in policy, be specified in funding contracts, and be utilized habitually. Institutionalization takes place at system, organization, and clinical process levels. This can be facilitated by having formal structures (e.g., sites, hierarchies), policies (e.g., positions supporting EBP use), contracts that require EBP use, tools (e.g., fidelity monitoring) and processes (e.g., fidelity feedback and coaching) (Stirman, Kimberly, Cook, Calloway, Castro, & Charns, 2012). To facilitate institutionalization, and formalization, sustainment should begin at the inception of the process and continue through the exploration and preparation stages and through the implementation phase. That is, begin with sustainment in mind.
We have summarized some of our experiences and findings in scaling up EBPs across two states and multiple counties with different service systems. Although our focus has been on service networks managed by child welfare, the findings are not necessarily limited to this context and there is precedent for considering the broader public sector service system as the “context” (Aarons, Hurlburt, & Horwitz, 2011). As noted above, child welfare, mental health, substance abuse and health concerns span service systems, just as clients and patients often receive services in more than one service sector. Consolidating findings across studies and systems can allow us to better identify, understand and describe the complex implementation process. Then we can tailor current implementation strategies and develop new and innovative implementation strategies to improve care and outcomes for children and families across service systems, organizations and health concerns.
Aarons, G. A., Fettes, D. L., Flores Jr, L. E., & Sommerfeld, D. H. (2009). Evidence-based practice implementation and staff emotional exhaustion in children’s services. Behaviour research and therapy, 47(11), 954.
Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R.Silovsky, J. F., Hecht, D.B., & Chaffin, M.J. (2012). Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science, 7(1), 32.
Aarons, G. A., Hurlburt, M., Willging, C., Fettes, D., Gunderson, L., Chaffin, M., & Palinkas, L. (2013). Collaboration, Negotiation, and Coalescence for Interagency-Collaborative Teams to Scale-up Evidence-Based Practice. Manuscript submitted for publication. (Journal of Clinical Child & Adolescent Psychology).
Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77(2), 270.
Burns, B. J., Phillips, S. D., Wagner, H. R., Barth, R. P., Kolko, D. J., Campbell, Y., & Landsverk, J. (2004). Mental health need and access to mental health services by youths involved with child welfare: A national survey. Journal of the American Academy of child and Adolescent Psychiatry, 43(8), 960-970.
Chaffin, M., Funderburk B., Bard, D., Valle, L., & Gurwitch, R. (2011). A Motivation-PCIT Package Reduces Child Welfare Recidivism in a Randomized Dismantling Field Trial, Journal of Consulting and Clinical Psychology, 84-95.
Chaffin, M., Hecht, D., Bard, D., Silovsky, J. F., & Beasley, W. H. (2012). A statewide trial of the SafeCare home-based services model with parents in Child Protective Services. Pediatrics, 129(3), 509-515.
Damashek, A., Bard, D., & Hecht, D. (2012). Provider cultural competency, client satisfaction, and engagement in home-based programs to treat child abuse and neglect. Child maltreatment, 17(1), 56-66.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4(1), 50.
Garland, A. F., Hough, R. L., Landsverk, J. A., & Brown, S. A. (2001). Multi-sector complexity of systems of care for youth with mental health needs. Children's Services: Social Policy, Research, and Practice, 4(3), 123-140.
Glisson, C. Hemmelgarn, A., Green, P., Dukes, D., Atkinson, S., & Williams, N.J. (2012). Randomized Trial of the Availability, Responsiveness, and Continuity (ARC) Organizational Intervention With Community-based Mental Health Programs and Clinicians Serving Youth. Journal of the American Academy of Child and Adolescent Psychiatry, 51(8), 780-787.
Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments. Mental health services research, 7(4), 243-259.
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly, 82(4), 581-629.
Horwitz, S. M., Hurlburt, M. S., Goldhaber-Fiebert, J. D., Heneghan, A. M., Zhang, J., Rolls-Reutz, J. Fisher, E., Landsverk, J., & Stein, R. E. (2012). Mental health services use by children investigated by child welfare agencies. Pediatrics, 130(5), 861-869.
Hurlburt, M., Aarons, G. A., Fettes, D., Willging, C., Gunderson, L., Palinkas, L., & Chaffin, M. (2013). Interagency Collaborative Teams for Capacity Building to Scale-Up of Evidence-Based Practice: Structure and Process. Manuscript in submitted for publication.
Southam-Gerow, M. A., Rodríguez, A., Chorpita, B. F., & Daleiden, E. (2012). Dissemination and implementation of evidence based treatments for youth: Challenges and recommendations. Professional Psychology: Research and Practice, 43, 527-534.
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science, 7(1), 17.
Gregory Aarons, PhD, is a psychologist and professor of psychiatry at the University of California, San Diego. His current work, funded by National Institute of Mental Health focuses on developing and testing approaches to evidence-based practice implementation and sustainment in health, mental health and social service settings. Aarons’ implementation studies examine evidence-based practice implementation for child neglect and the use of community engagement and collaboration to support appropriate adaptation of system, organization and intervention characteristics during implementation. His other current projects examine the effectiveness of promoting interagency collaboration to scale-up an evidence-based practice a study focusing on how to effectively develop local expertise to implement an evidence-based HIV preventive intervention in community health centers in Mexico, and optimizing leadership and organizational development strategies to promote effective evidence-based practice implementation in healthcare and allied healthcare settings. Aarons is an associate editor of the journal Implementation Science and on the editorial board of Administration and Policy in Mental Health and Mental Health Services Research.
Mark Chaffin, PhD is a psychologist and professor of pediatrics at the University of Oklahoma Health Sciences Center in Oklahoma City. He directs research for the Section of Developmental and Behavioral Pediatrics and the Center on Child Abuse and Neglect. His work centers around child maltreatment, including development, adaptation, implementation and scale-up of evidence based prevention and intervention models.