Authors: Lisa Saldana, PhD, and Patti Chamberlain, PhD

Over the last decade, the field of implementation science has gained increasing traction, moving research beyond the initial stages of theory development to expanded foci, including evaluating the effectiveness of implementation and sustainability strategies (see NIH Dissemination and Implementation Research in Health study section). In part, this movement has resulted from the well-documented evidence that it takes 17 years on average for research to translate into practice, and less than 15 percent of evidence-based practices (EBPs) are implemented in real-world settings (Balas & Boren, 2000). Children’s mental health lags even further behind other medical and behavioral fields in moving EBPs into real-world settings according to some reports (Hoagwood, Olin, & Cleek, 2013), despite strong evidence that usual care mental health services tend to produce poor outcomes (Garland et al., 2013).

Here we describe our research and experience implementing and scaling up two evidence-based practices (EBPs) into large public service systems: Multidimensional Treatment Foster Care (MTFC; Chamberlain, 2003), an alternative to congregate care for children and adolescents with severe behavioral problems, and KEEP (Chamberlain et al., 2008), a training and support program for foster parents that aims to decrease placement disruption and speed reunification and permanency. These interventions are theoretically based and have demonstrated positive outcomes in multiple RCTs, as is the case with all EBPs. Both MTFC and KEEP have shown relatively high adoption rates, which is somewhat unique among child and family focused EBPs, with several notable exceptions (e.g., MST, FFT, Incredible Years, Triple P). Currently, more than 120 MTFC teams exist worldwide. More recently developed, KEEP is implemented in 31 sites in the U.S. and in Europe.

MTFC implementation

We focus first on MTFC because we have implemented this model in a variety of community settings since 2001 and have learned from both our successes and failures. The MTFC model is complex to implement, involving a well-coordinated treatment team that provides services to the youth (through individual therapy and skills training) and family (family therapy), all on the backbone of the child’s placement in a well-supported and trained foster home. The team utilizes a point-and-level system that focuses on reinforcement for normative behavior and staff role stratification to support the youth and parents (foster and biological). Foster parents provide positive adult support and mentoring, close supervision and consistent limit setting. Unlike many foster parenting models, MTFC focuses on training in behavior management strategies that have been demonstrated to be effective through research (Chamberlain, 1998). Foster parents attend weekly support group meetings with the program supervisor to receive peer-to-peer support and problem-solve youth behavioral problems.

The model’s multi-faceted nature makes it difficult to implement; however, MTFC’s target population has consistently experienced poor and even iatrogenic outcomes in usual group/residential care settings. In addition, such placements are expensive and violate contemporary norms such as serving youth in least restrictive settings with high involvement from families and relatives. These factors (cost savings, better outcomes, and adherence with best practice values) all appear to make it “worth” implementing MTFC despite the complexity and intensity of the model.

Although much is known about what it takes to run an effective MTFC program, less is known about what it takes to start a program, or about the cost of implementing MTFC. What is needed in terms of planning, training, and upfront costs? How long will the program need to be on start-up life support to become viable and self-sustaining?

These questions are complicated by the fact that EBPs are implemented primarily in agencies/sites that are early adopters. In these instances agencies that are willing to embrace innovation are the ones that adopt EBPs. This creates a needs-innovation paradox in which the most organized, capable settings take advantage of advances in research while sites that potentially need EBPs the most do not. Over the last seven years, we have studied what it takes to implement MTFC and KEEP in non-early adopting agencies. We recently completed a large randomized trial of MTFC in non-early adopting counties in California and Ohio, and are currently piloting the implementation of KEEP across a large urban foster care system in New York City.

In the California/Ohio trial, 51 counties were randomized to one of two implementation strategies: Community Development Teams (CDT, experimental condition; Sosna & Marsenich, 2006) or Individual Implementation (IND, control condition), which is the typical strategy used with early adopting sites. Participation in the CDT condition involved implementing MTFC in concert with six to seven other counties in peer-to-peer networks. Stakeholders, including system leaders and agencies directors, were brought together for a series of meetings and phone calls led by a CDT facilitator well versed in MTFC. Participation in the IND condition involved delivery of the standard MTFC package, including a readiness process involving six planning calls and an on-site stakeholder meeting prior to implementation. In both conditions, all MTFC program staff received the typical MTFC quality assurance protocol, which includes five days of clinical training and weekly consultation with an MTFC expert.

The importance of meeting a local need

We found that county “need” for foster care services significantly predicted whether or not the county achieved successful program start up regardless of study condition. Need was determined by the number of youth in out-of-home care in the county on two snapshot days during the year preceding the study. This finding is consistent with our experience implementing MTFC in early adopting communities. Because the MTFC model was developed specifically to address real world outcomes that are important to systems (e.g., arrests, living in community settings), communities who typically implement MTFC are those who have identified a need to adopt EBPs to address specific gaps or deficits.

Understanding and measuring process and milestone achievement

To evaluate the success/failure of implementation, we developed the Stages of Implementation Completion (SIC; Chamberlain, Brown, & Saldana, 2011), an 8-stage assessment tool that measures a site’s progress toward successful implementation of MTFC. The SIC measures both implementation process (the progression through pre-implementation, implementation and sustainability) and achievement of milestones (e.g., serving first client). The stages range from Engagement with the developers to practitioner Competency and each stage is populated with sub-activities (e.g., date staff are hired, date completed fidelity tracking training). As the purveyors monitor the implementation progress of the adopting site, they track the dates that the site completes each of the activities.

Three scores are derived from the SIC: Duration, Proportion and Stage. The Duration score, or the amount of time that a site takes in each stage, allows us to understand how long it takes for sites to move through the different stages and phases of implementation. The Proportion score assesses the degree of thoroughness by capturing the percentage of activities that the site completed while moving through each stage. Finally, the Ssage score provides information regarding the final stage that each site attained during their implementation.

Currently, we are conducting research on adapting the SIC for other child mental health EBPs (Saldana; R01 MH097748-01A1). Through this process, we hope to understand more about which implementation activities are essential and common across implementation strategies for different EBPs.

Administrative turnover matters

Using the SIC, we followed California and Ohio participants including system leaders, agency directors and practitioners through the implementation process. We examined the influence of staff turnover on implementation success and learned that both the “who” and the “when” of turnover matters. Counties that experienced system leader turnover during the pre-implementation phase —when they are in the process of undergoing readiness planning — were significantly less likely to achieve program start up. Turnover in the system leader role became less important after the program commenced. On the other hand, turnover in the agency director role during the implementation phase—when the EBP is underway and progressing through the consultation process — was predictive of program failure. Surprisingly, turnover in practitioners once the program began did not have a significant influence on short-term MTFC program sustainability. Although these findings need to be replicated with other EBPs, they suggest the value of different roles throughout the implementation process in terms of achieving implementation success. 

Pre-implementation behavior matters

Also using the SIC, we found that successful program start-up was predicted by site performance during the pre-implementation phase. Three clusters, or types, of counties were identified: those who (1) completed a high proportion of implementation activities, and did so quickly; (2) completed a low proportion of implementation activities, and did so slowly; and (3) did not complete pre-implementation. The sites who completed pre-implementation activities thoroughly and at a relatively steady but rapid pace were more likely to successfully start a program than those who skipped recommended implementation activities, or proceeded either in a rush or at a slow pace (Saldana et al., 2012). Although several sites that were extremely slow during pre-implementation did eventually start programs, these programs were not successfully sustained.

Costing implementation is feasible

Standardized methods for assessing EBP implementation costs currently do not exist. Using the SIC, we were able to successfully cost the amount of resources used by sites to implement MTFC. We found that the greatest degree of variation among sites occurred during pre-implementation. Perhaps counter intuitively, the amount of time that it takes to complete an implementation activity does not necessarily relate to the cost associated with that activity. Some agency directors might choose to lag behind in completing activities, preferring instead to use low-wage staff who are less efficient in completing implementation tasks. Given the finding that time is of the essence for successful program start up, it is important for decision-makers to consider timing as well as resources when developing budgets and fiscal strategies. Importantly, we demonstrated that implementation costs can indeed be assessed and calculated in a standardized manner and, therefore, our future research will focus on determining if there are common costs and resources that can be expected by adopters independent of the EBP being implemented.

KEEP implementation: CSNYC

The relatively high level of interest in KEEP adoption has afforded us the opportunity to evaluate KEEP implementation efforts and examine if the knowledge gained from the MTFC implementation trial can help to increase the potential for success. KEEP is a preventive model that evolved from using social learning theory-based interventions with foster parents in MTFC. The curriculum includes a 16-week foster/kinship parent group to help caregivers manage emotional behavior problems of children in regular foster care. Multiple randomized trials showed that participation in KEEP reduced placement disruptions, improved child mental health problems and speeded reunification to permanency. KEEP has been implemented in England and Sweden, and across several states in the U.S. 

Currently, the largest implementation is occurring in New York City as part of a larger Administration of Children’s Services foster care system reform effort (Child Success NYC; CSNYC) that also includes parent management training for biological and adoptive parents, and skills coaching for youth aged 14 years and older. The goals of CSNYC include decreasing the amount of time youth spend in out-of-home care and reducing the number of placement disruptions. These linked interventions are implemented and monitored by our team in Oregon including the authors, Marion Forgatch and her team at Implementation Sciences International, Peter Sprengelmeyer and his team at OSLC Community Programs, and Sylvia Rowlands at the New York Foundling. The implementation includes tracking model fidelity and delivering ongoing consultation to agencies using web-based technologies.

Lessons learned

To maximize the opportunities for EBP adoption into usual practice settings, we have gleaned several take-away messages from our experiences. First, it is advantageous to develop programs/interventions with the ultimate agency or system level in mind. MTFC was designed, developed and tested within the juvenile justice system, and KEEP within the child welfare system. Second, program outcomes should be easily observable and measurable without causing undue burden to providers. It is ideal to map onto outcomes and data that the agency/system already tracks. Programs that save money in the long run are often worth the investment despite the resources needed for start up and for maintenance with fidelity. Finally, further study is needed using measures such as the SIC and others to better understand how to design and test successful and cost-effective implementations. These projects demonstrate some important lessons for the field of implementation science to consider as researchers, policymakers and systems grapple with the complexities of scaling up EBPs in diverse service systems to achieve better public health outcomes for vulnerable children and families.

References

Balas, E., & Boren, S. (2000). Managing clinical knowledge for health care improvement. In Bemmel J, McCray AT (Eds). Section 1: health and clinical management. Yearbook of Medical Informatics: Patient Centered Systems (p. 65-70).Stuttgart, Germany: Schattauer Verlagsgesellschaft.

Chamberlain, P. (1998). Treatment Foster Care. Family Strengthening Series. Washington, D.C.: U. S. Department of Justice. (OJJDP Bulletin NCJ l734211).

Chamberlain, P. (2003). The Oregon Multidimensional Treatment Foster Care model: Features, outcomes, and progress in dissemination. In S. Schoenwald & S. Henggeler (Series Eds.), Moving evidence-based treatments from the laboratory into clinical practice. Cognitive and Behavioral Practice, 10, 303-312.

Chamberlain, P., & Brown, C. H., & Saldana, L. (2011). Observational Measure of Implementation Progress: The Stages of Implementation Completion (SIC). Implementation Science, 6, 116.

Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. A., & Reid, J. B. (2008). Prevention of behavior problems for children in foster care: Outcomes and mediation effects. Prevention Science, 9, 17-27.

Garland, A. F., Haine-Schlagel, R., Brookman-Frazee, L., Baker-Ericzen, M., Trask, E., & Fawley-King, K. (2013). Improving community-based mental health care for children: Translating knowledge into action. Adminstration and Policy in Mental Health Research, 40(1), 6-22.

Hoagwood, K., Olin, S., & Cleek, A. (2013). Beyond context to the skyline: Thinking in 3D. Adminstration and Policy in Mental Health Research, 40(1), 23-28.

Saldana, L., Chamberlain, P., Wang, W., & Brown, H. (2012). Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health Research, 39, 419-425.

Sosna, T., & Marsenich, L. (2006). Community development team model: Supporting the model adherent implementation of programs and practices. Sacramento: California Institute for Mental Health.

U.S. Department of Health and Human Services, National Institutes of Health (2013). Dissemination and Implementation Research in Health (R01-PAR-10-038).

Authors’ bios

Lisa Saldana, PhD, has a doctorate in clinical psychology with a research and clinical emphasis in child welfare populations and evidence-based practice. She is a co-investigator on the implementation trial described in this article, examining two implementation strategies to scale-up Multidimensional Treatment Foster Care (MTFC), for youth in foster care. She recently obtained an R01 to examine measurement of implementation processes and milestones for EBPs in the children's mental health service sectors. She is working on research grants focused on the economic evaluation of evidence-based practices and, with her colleagues, developed the Stages of Implementation Completion, a tool for measuring the process of implementing practices in real-world community settings. Saldana also is working on the development of preventive interventions to address the needs of families involved in the child welfare system. With funding from NIDA, she developed an integrative treatment for maternal substance abuse and child neglect (the FAIR project), and is co-developer of the Parenting and Visitation Enhancement (PAVE) model being piloted for biological parents working towards reunification.

Patricia Chamberlain, PhD, is a senior research scientist at the Oregon Social Learning Center and the Center for Research to Practice in Eugene, Ore. Chamberlain developed the Multidimensional Treatment Foster Care (MTFC) and KEEP intervention models. She is the project director for Child Success NYC, a collaboration with New York City’s Administration of Children’s Services. She has been the principal investigator on nine randomized clinical trials examining the feasibility, efficacy and effectiveness of community-based intervention approaches. Chamberlain also founded OSLC Community Programs, a nonprofit organization dedicated to providing services to Oregon children and families, and is a founding partner of Treatment Foster Care Consultants (TFCC), a company created to disseminate the MTFC model to interested communities. Her research interests include gender influences on outcomes for adolescents and young adults, methods for collecting “real time” data on child functioning in public service system settings, and methods for training and supporting relevant agencies/entities in counties and states to bring evidence-based interventions up to scale so that they are part of routine practice.  She has authored three books and more than 70 journal articles and book chapters on evidence-based treatment approaches, treatment process, outcome research, methodology, foster care and related topics. In 2007, she received the Science to Practice Award from the Society for Prevention Research.