Evidence-informed care for everyone, including the uncoverable: Scaling Up Evidence-Based systems requires carefully choosing evidence-based treatment (EBT) programs for the youths they cover and leveraging EBT knowledge to serve the rest

Examples of two methods for selecting a service array to provide EBT options for as many youths as possible.

Authors: Adam Bernstein, PhD, Christine Bae, Eric L. Daleiden, PhD, Michael A. Southam-Gerow, PhD and Bruce F. Chorpita, PhD

The evidence-based practice (EBP) movement of the past two decades represents a revolution in the science of psychosocial treatment service quality. Nonetheless, the complexity of leveraging science to optimize or even just improve service has proven substantial. Though many challenges involved are well known and have been written about for decades, other aspects of the science-practice gap have come into focus more gradually as systems struggle to make use of the bounty of the evidence-based practice movement to serve complex client populations with diverse clinical needs. A principal challenge among those is the growing realization that packaged evidence-based (EBT) products, e.g., programs developed and tested successfully for individual groups, do not —and simply cannot — exist for every child and context (cf, Kazdin, 2008).

This predicament is illustrated below by examples of two methods for selecting a service array to provide EBT options for as many youths as possible. We then present an approach to extending the EBT paradigm in light of this limitation.

The conventional approach to treatment array selection: committee solution making

With well over a hundred prevention and intervention EBTs in existence (Substance Abuse and Mental Health Services Administration [SAMSHA], 2012), typical service systems must select only a fraction of those to serve their needs. Various lists and catalogues of EBTs summarize the available options (e.g., the National Registry of Effective Practices and Programs [NREPP]); SAMSHA, 2012) but provide no formal guidance regarding how to select the treatments most relevant to a local population. Moreover, even for highly resourced service systems, the number of EBTs that can be successfully integrated and managed is likely to be small due to factors including the burden of training and maintaining a staff skilled in multiple EBTs, and the demands of administrative, financial, supervision and evaluation structures, each typically unique for every EBT.

The Los Angeles County Department of Mental Health (LACDMH), amidst an ambitious initiative to dramatically increase the use of EBTs in the county, recently took on the challenge of carefully determining its service array. With a new state funding source dedicated in part to Prevention and Early Intervention (PEI) practices, the county began an extensive stakeholder review process covering a two-year period and involving multiple county departments, provider agencies, clinical staff, consumers and other stakeholders. The process was admirably open, public and grounded in transparent information. The plan that initially resulted included the implementation of 51 EBTs, promising practices and community defined evidence practices in eight different geographic regions within Los Angeles County. However, upon state approval of the plan, the economic crisis largely eliminated prior mental health service funding sources, forcing earlier decisions to be revisited with marked decrease in the number of programs to implement.

To prevent system collapse, the county proactively transformed the mental health system through rapid implementation of a modified PEI plan. Both directly-operated county clinics and contracted mental health providers were given the opportunity to transform and develop a PEI program and implement one or more of the following six EBTs: Seeking Safety, Trauma Focused Cognitive Behavioral Therapy, Child-Parent Psychotherapy, Cognitive Behavioral Intervention for Trauma in Schools, Positive Parenting Program (Triple P) and Group Cognitive Behavioral Therapy of Major Depression. 

Considering the goal of expanding the spectrum of care and the limited selection of EBTs available for implementation, there were obvious gaps inadvertently created by this immediate transformation. After presenting a quantitative model of array selection, we will return to the LACDMH example to discuss an evidence-informed approach to filling these gaps.

The relevance mapping approach to treatment array selection

We recently introduced an empirical approach to support such decisions, called relevance mapping (Chorpita, Bernstein, & Daleiden, 2011). Relevance mapping gauges the generalizability of any treatment or set of treatments to a given population, and in doing so, addresses the question of how a set of selected treatments can and should work together to provide an evidence-based option for as many youth and families as possible.

An example is the analysis done for Hawai‘i’s statewide mental health system, the Child and Adolescent Mental Health Division (CAMHD; Chorpita et al., 2011). Relevance mapping procedures were used to compare (a) demographic and diagnostic data from 1,781 youths receiving services during a recent fiscal year, with (b) the same characteristics of participants in 437 randomized clinical trials (RCTs) corresponding to 98 treatment types. Each youth was compared to each study to determine which EBTs were a “match” for that client. We required that EBTs match youths on just three dimensions — primary problem, age and gender — however ethnicity and treatment setting were also considered, and any other available parameter deemed important by CAMHD stakeholders could have been added. The procedure then looked at each possible combination of treatments to find the smallest set that together matched as many youths as possible. A treatment program already available in the CAMHD system, Multisystemic Therapy (MST; Henggeler,  Schoenwald, Borduin,  Rowland, & Cunningham, 1998) was also entered into the analysis so that solutions identified would complement — not duplicate — the youths already covered by MST.

From among countless possible combinations, relevance mapping identified precisely the eight treatments that along with MST covered the maximum number of youths: 71 percent. That is, with those eight treatments plus the one already in place, as many youths would be coverable as if all of the 98 treatment types were available. Additionally, the results showed that among the eight treatments identified, some applied to a very small percentage of youths, indicating that if resources are tight, those may be the first to consider omitting from implementation.

These results would provide a powerful information source to inform a committee selection process as described in the section above. With such results in hand, a committee would be primed to select a set of EBTs applicable to more youths, and more families would thus have an EBT program available. However, though the relevance mapping approach appears to present major advantages to human decision making unaided by such supports, the root problem persists: a large portion of youths remains (29 percent for CAMHD, even if all 98 EBTs from 437 RCTs were available) for whom there is no specific EBT product.

What to do for everyone else: evidence-informed care in the absence of an ideal EBT

Even the best empirically informed EBT array planning thus leaves the need for a “catchall” service to provide for the considerable number of youths with no matching EBT. However, a choice remains:  the catchall service can be empirically informed by knowledge from the hundreds of studies considered above, or not (i.e., “usual care”).

How can an empirically informed catchall service be achieved? One solution is to structure key clinical processes (selecting an intervention, session planning, supervision, etc.) to make routine use of knowledge drawn from the evidence base and from the client case at hand. Returning to the example of LACDMH, this goal was approached by building a service called Managing and Adapting Practices (MAP). MAP has four foci:

(a) Common practices. MAP therapists and supervisors receive training in a library of Practice Guides, two-page summaries of the most common procedures drawn from EBTs (e.g., psychoeducation, relaxation, monitoring).

(b) Common processes. Providers are similarly trained in a set of Process Guides, one-page outlines or flowcharts that represent core concepts abstracted from EBTs regarding how to organize the timing, sequence, and logic of how care is delivered (e.g., session structure, treatment course planning).

(c) Measurement and feedback. A clinical dashboard visually summarizes evidence relevant to ongoing decision making during treatment, including progress (measures of targeted outcomes) and practices delivered. When progress is poor, instrumental feedback is available. An online searchable database (the PracticeWise Evidence-Based Services [PWEBS] database provides summaries of the practices most commonly found in EBTs targeting specific problems or diagnoses for persons of a particular age, gender or ethnicity, or in a specific setting).

(d) Delivering information. The MAP planning model is built around making key information resources available at decision times to bias toward the best evidence. For example, making the dashboard available for clinical review anchors progress evaluation in data and cues consideration of whether practices align with evidence from the PWEBS. Similarly, sessions are primed to follow best evidence via provider review of the particular Practice Guide to be implemented, along with session and treatment planners and recent case-specific history on the dashboard.

Can it work in a live system?

The LACDMH network of service providers advocated that MAP be added to the menu of available PEI practices in order to better meet the needs of the communities they serve and to close some of the gaps in available treatment. After six months of implementation of the selected PEI practices, LACDMH added MAP as an option for providers and their PEI programs to serve youths experiencing symptoms of depression, anxiety, disruptive behavior and trauma.

Preliminary results from LACDMH show that MAP has been very popular, with provider demand for MAP driving rapid development and scaling. In two years, more than 1,600 direct providers and over 100 agency supervisors have been trained. In fiscal year 2011-12, MAP accounted for 22 percent of countywide billing claims, and served 12,360 unique clients, more than any EBT program in LACDMH, encouraging indicators of acceptability. Regarding outcomes, preliminary results from California Institute of Mental Health evaluations show promising improvement rates for MAP, with effect sizes on the Youth Outcome Questionnaire (Burlingame, Wells, & Lambert, 1996) in the large (caregiver-report; Cohen’s d = 0.81) to moderate (youth-report; Cohen’s d = 0.70) ranges. Importantly though, the ideas above are larger than MAP, which represents one particular implementation of these concepts. The same principles underlay Hawai‘i’s expansive evidence-based services initiative, for example, with resulting dramatic improvements in outcomes and efficiency (Daleiden, Chorpita, Donkervoet,  Arensdorf, & Brogan, 2006).

For system stakeholders and policy makers faced with the challenge of creating the best service array, two key questions are: what treatments should we make available to help as many as possible of the families we serve? And, how will we serve those whom our best research (e.g., all the EBTs listed by NREPP) leaves uncovered? We have shown that empirical approaches like relevance mapping can help with the treatment array design decision, and structured knowledge drawn from evidence can help us to address the inevitable problem of those left out.


Burlingame, G. M., Wells, M. G., & Lambert, M. J. (1996). Youth Outcome Questionnaire. Stevenson, MD: American Professional Credentialing Services.

Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children's mental health services. Journal of Consulting and Clinical Psychology, 79(4), 470-480.

Daleiden, E. L., Chorpita, B. F., Donkervoet, C. M., Arensdorf, A. A., & Brogan, M. (2006). Getting better at getting them better: Health outcomes and evidence-based practice within a system of care. Journal of the American Academy of Child and Adolescent Psychiatry, 45, 749-756.

Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (1998). Multisystemic treatment for antisocial behavior in chilrdren and adolescents. New York: Guilford Press.

Kazdin, A. E. (2008). Evidence-based treatments and delivery of psychological services: shifting our emphases to increase impact. Psychological Services, 5, 201–215.

Substance Abuse and Mental Health Services Administration (2012). National registry of effective programs and practices.

Authors’ Bios

Adam Bernstein, PhDAdam Bernstein, PhD, recently completed his doctorate in clinical psychology at University of California, Los Angeles, and also holds a BA in Computer Science and MA in psychology from Stanford University. His research has focused on empirical tools to help service systems optimally select and coordinate multiple evidence-based treatments. He serves on the Research & Evaluation, Professional Development, and Service & Products Development teams at PracticeWise, LLC.


Christine BaeChristine Bae is a licensed clinical social worker with a Master Degree in Public Health. Currently, she is a partner and consultant with Seedling Consulting Group, LLC working on projects in advocacy, development and implementation.  She is also a lecturer in the Social Work Graduate Program at California State University, Los Angeles.  Recently, Ms. Bae served as a Mental Health Clinical Program Head at the Los Angeles County Department of Mental Health with Prevention and Early Intervention Program Administration.  She has worked with Los Angeles County Departments of Health and Mental Health for over 13 years with experience in specific geographic initiatives and programs, contract negotiations, strategic planning, program development and administration for the children's mental health system, and fiscal analysis and monitoring of Medicaid programs.


Eric Daleiden, PhDEric Daleiden, PhD, is the chief operating officer of PracticeWise, LLC. He has worked extensively in strategic and operational management and has held leadership positions in the academy, the private sector, and government, including serving as a research evaluation specialist for Hawai‘i’s Child and Adolescent Mental Health Division. Daleiden holds a PhD from The Ohio State University and has numerous scientific papers and technical reports. His personal mission is to improve the lives of youth and families through better understanding and delivery of behavioral health services.


Michael Southam-Gerow, PhD Michael Southam-Gerow, PhD, is associate professor in the departments of psychology and pediatrics at Virginia Commonwealth University (VCU), and the Director of Quality, Performance and Training for PracticeWise, LLC. Southam-Gerow received his bachelor’s degree at the University of Michigan and his PhD at Temple University in Philadelphia and his research focuses on the dissemination of evidence-based treatments (EBTs) for child/adolescent mental health problems, treatment integrity and emotion regulation. He has received research funding from the National Institute of Mental Health (NIMH) and is the author of dozens of scholarly papers.


Bruce Chorpita, PhDBruce Chorpita, PhD, is currently professor of psychology, at the University of California, Los Angeles and President of PracticeWise, LLC. He received his PhD in psychology from the University at Albany, State University of New York and held a faculty position with the department of psychology at the University of Hawai‘i from 1997 to 2008. From 2001 to 2003, Chorpita served as the clinical director of the Hawai‘i Department of Health’s Child and Adolescent Mental Health Division. His work focuses on treatment and system designs that enhance effectiveness of children's mental health services in large systems. With over 150 publications on children’s mental health, he has been the recipient of multiple awards and honors for his work. Chorpita has held research and training grants from the National Institute of Mental Health, the Hawai‘i Departments of Education and Health, the John D. and Catherine T. MacArthur Foundation, and the Annie E. Casey Foundation.