Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jan 29.
Published in final edited form as: J Evid Based Soc Work. 2014 Jan;11(0):84–96. doi: 10.1080/15433714.2013.842440

Statewide Implementation of the 4 Rs and 2 Ss for Strengthening Families

Geetha Gopalan 1, Lydia M Franco 2, Kara Dean-Assael 3, Mandy McGuire-Schwartz 4, Anil Chacko 5,6,7, Mary McKay 8
PMCID: PMC3905247  NIHMSID: NIHMS388528  PMID: 24405134

Abstract

Embedding evidence-informed practices for children with mental health needs into “real-world” community settings has proven challenging. In this paper, we discuss how the Practical, Robust, Implementation, and Sustainability Model (PRISM) guided statewide implementation of an evidence-informed intervention targeting families and youth with oppositional defiant and conduct disorders, collectively referred to as disruptive behavior disorders. Additionally, we present qualitative findings based upon provider reports of integrating a novel, evidence-informed intervention within their respective community mental health setting. Finally, we discuss a process of modifying the intervention to address agency-level factors, as well as inform more widespread implementation efforts.

Introduction

Over the past two decades, significant advancements have been made in the development of effective practices for children with mental health needs. However, many have failed during implementation efforts in “real-world” community settings. As a result, the majority of children receiving their care in “real world” clinics are not offered or will not receive treatments with established evidence of effectiveness. Implementing any new innovation including evidence-informed practices requires attention to systemic, as well as agency- and consumer-level factors. In this paper, we discuss the utility of the Practical, Robust, Implementation, and Sustainability Model (PRISM; Feldstein & Gow, 2008) as a guide for widespread implementation of an evidence-informed practice, the 4 Rs and 2 Ss for Strengthening Families Program (4 Rs Program) (Franco et al., 2008; Gopalan & Franco, 2009; McKay, Gonzales, Quintana, Kim, & Abdul-Adil, 1999; McKay, Gonzales, Stone, Ryland, & Kohner,1995; McKay, Harrison, Gonzales, Kim, & Quintana, 2002; McKay et al., 2010; 2011), which targets youth and their families with disruptive behavior disorders across community-based child mental health clinics in New York State.

Based on public health constructs from quality improvement, PRISM considers how program design, external environment, implementation infrastructure, and program beneficiaries influence whether an evidence-informed practice will ultimately be adopted and sustained by community-based agencies. This paper explores how PRISM could inform the state-wide implementation of the 4 Rs Program. In addition, we present qualitative findings based upon provider and administrator reports regarding the benefits and challenges of integrating the 4 Rs Program within their respective community mental health clinic. Finally, we describe a process whereby the intervention was subsequently modified to address agency-level (i.e., clinic and providers) resources and constraints in order to prepare for more widespread implementation.

Background

Two decades of research have yielded a number of efficacious interventions for children's mental health treatment (Kazdin & Weisz, 2003). Evidence-informed practices in children's mental health include psychosocial interventions for childhood autism (Rogers & Vismara, 2008); mood and anxiety disorders (David-Ferdon & Kaslow, 2008; Silverman, Pina, & Viswesvaran, 2008; Weisz, Hawley, & Doss, 2004), as well as children's behavioral difficulties (Eyberg, Nelson, & Boggs, 2008; Pelham & Fabiano, 2008; Weisz et al., 2004). Psychotropic medications have also demonstrated effectiveness in treating Attention Deficit Hyperactivity Disorder (ADHD), mood, and anxiety disorders in children and adolescents (McClellan & Werry, 2003; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001). Finally, successful strategies have been developed to support service delivery, including interventions to promote consumer engagement, diagnostic screening, clinical assessment protocols, and measurement feedback systems (Bickman, 2008; Kazdin & Weisz, 2003; Meyer et al., 2001; SAMHSA, 2010).

However, it has been humbling to note that few practices supported by evidence have been able to penetrate community-based child mental health clinics, with widespread adoption and sustainability of evidence-informed practices proving elusive (New Freedom Commission on Mental Health, 2003). In fact, some estimates indicate that translating an efficacious intervention from the research setting to the “real world” can take up to 17 years (IOM, 2001; USPHS, 2000). Within the field of children's mental health, sustained supports for implementing evidence-informed practices remain scarce (Kazak et al., 2010). This is particularly problematic as accumulated evidence suggests that care offered in typical community-based child mental health settings may not be associated with the positive outcomes so frequently associated with evidence-informed care (Bickman, 1996; Weisz & Jensen, 2001; Weisz, Jensen-Doss, & Hawley, 2006).

Further, service delivery exists within a context where shrinking resources may affect care in general and implementation of evidence-informed practices in particular. Unfortunately, increasing state budget shortfalls are resulting in severe fiscal constraints across the mental health service spectrum (NRI, 2010). Given that the treatment of children's mental health disorders ranks among the top 5 most costly medical conditions (AHRQ, 2009; Soni, 2006), implementing evidence-informed practices may be economically sound, where one state reports a 56% return on investment (Aos, Mayfield, Miller, & Yen, 2006). However, when viewed by “real world” providers and administrators, evidence-informed practices often require too much investment in an atmosphere of change and scarce resources.

It is within this somewhat confused, often stressful, context that public systems across the country are proceeding with efforts to have community-based clinics provide evidence-informed care. In fact, over 20 states are actively attempting to integrate effective psychosocial therapies and medication practices into their systems (Bruns & Hoagwood, 2008; Ganju, 2003). Efforts to support system transformation include statewide reform efforts, such as the National Association of State Mental Health Program Directors (NASMHPD) consortium (Ganju, 2003; NASMHPD, 2005; Yannacci, Rivard, & Ganju, 2005), redirection of national funding priorities to incentivize the adoption of evidence-informed practices (Kazak et al., 2010), consortiums focused on sharing evaluation and implementation lessons (Bruns & Hoagwood, 2008; Bruns et al., 2008), as well as toolkits to promote uptake (Burns et al., in press; Torrey et al., 2001).

New York State, in particular, has invested $33 million to implement three quality improvement activities focusing on implementing practices supported with evidence through the “Achieving the Promise” initiative (Cavaleri, Hoagwood, & McKay, 2009). These include the (1) Evidence-Based Treatment Dissemination Center (EBTDC), which provided training in Cognitive Behavioral Therapy (CBT) to clinicians and supervisors across 186 clinics (Gleacher et al., 2010); (2) Family and Child Clinic Plus, which provided 36,065 early detection screenings across 108 clinics (Essock et al., 2009); and (3) PSYCKES, a medication management system to reduce polypharmacy in 234 clinics (Essock et al., 2009; Finnerty et al., 2002).

Despite the momentum to support evidence-informed practices within “real-world” settings, clinic staff and families themselves often have difficulty accepting new and tested practices. At the agency level, tensions emerge related to the effect that adopting novel practices will have on caseloads and supervisory practices, how to integrate evidence-informed practices into existing management and organizational configurations, how to monitor outcomes and provide feedback, and whether the costs of training and skill-building are prohibitive within a context of underfunded public mental health programs (Kazak et al., 2010). Implementation efforts must also contend with low “absorptive capacity”, which can entail barriers for providers to learn new skills, as well as limited infrastructure to utilize new knowledge and practices (Aarons et al., 2011). As a consequence, difficulties often arise when engaging agency providers to utilize new practices supported with evidence. Recent quality improvement initiatives in New York State demonstrated significant challenges to scaling-up across the state, manifested in uneven agency uptake which has hampered the efficiency of roll-outs in New York (Gleacher et al., 2011) as well as other states (NRI, 2010). For example, challenges to uptake in New York included issues of balance in providing enough support for supervisors and staff without adding burden, differences in consultation quality, uneven clinic-level support for staff, and differences in clinician skill set (Gleacher et al., 2011).

Children's Technical Assistance Center (CTAC)

In response to significant implementation challenges, the Children's Technical Assistance Center (CTAC) at Columbia University and the New York State Psychiatric Institute was established to provide training and support on quality improvement (QI) strategies, including clinical and organizational skills, to all New York State child-serving mental health clinics, with the overall goal of improving mental health outcomes for children and their families. QI supports were offered to over 300 clinics, specifically designed to directly impact clinic operations, improve efficiency, effectiveness, and value of services provided at every level of the clinic system. To accomplish this goal, a set of evidence-informed trainings targeting the enhancement of clinical skills, family support practices, and organizational business practices are in the process of being developed and deployed.

In consideration of the practice exigencies of multi-level state evidence-informed practice implementation, CTAC staff drew upon the PRISM model to drive implementation efforts. PRISM is based on public health constructs drawn from quality improvement, chronic care, and diffusion of innovations (Feldstein & Glasgow, 2008). An adapted version of the PRISM model (see Figure 1) emphasizes how quality improvement efforts to support the integration and adoption of evidence-informed practices are influenced by: (1) the State system context (e.g. implementation supports and constraints; financing; implementation and sustainability infrastructure), (2) consumers, and (3) agencies (i.e., clinics and providers).

Figure 1.

Figure 1

Open in a new tab

As indicated by PRISM, the state context is a powerful predictor of success in implementation and sustainability, with attention to be paid to market forces, such as payor satisfaction of initiatives which can help to accelerate change (Feldstein & Glasgow, 2008). In addition, evidence-informed practice initiatives must meet requirements of regulatory, accreditation, and reporting authorities, fit within existing reimbursement and coverage factors, and have available community resources to assist with interventions. Finally, an infrastructure for implementation and sustainability must be available which includes a dedicated implementation team, routine performance measurement and data sharing, protocols and procedures which can be adapted at the local level, and forums for sharing best implementation, training, and support practices (Feldstein & Glasgow, 2008). Consequently, CTAC has created an infrastructure to (1) develop and disseminate a range of trainings via regional learning opportunities; (2) reinforce and support skill acquisition for providers, and (3) provide personalized technical assistance to facilitate implementation of those specific practices meeting New York State requirements for regulatory, financing, and overall healthcare reform.

PRISM indicates that intervention implementation is maximized when it is able to reach important consumer subgroups, with attention paid to how local and individual barriers affect program participation (Feldstein & Glasgow, 2008). For instance, it is not uncommon for most low-income, minority families to attend 2 or 3 treatment sessions before prematurely dropping out of community-based child mental health services (McKay & Bannon, 2004). Typical barriers to engagement include parent stress, lack of social support, insufficient concrete resources (e.g., lack of childcare, transportation), as well as motivational barriers (e.g., stigma, prior negative experiences with service providers; Alvidrez, 1999; Kazdin & Whitley, 2003; Wahler & Dumas, 1989). Therefore, it is important that innovations for child mental health service delivery can overcome these service utilization barriers.

In response to these issues, CTAC sought to help clinics increase parental engagement and empowerment within a service delivery strategy addressing the most common reason for referral to child mental health services: child oppositional defiant and conduct difficulties evidenced either at home or at school (Angold & Costello, 2001; Frick, 1998; Loeber et al., 2000). In this way, consumer-level (i.e., families and children) factors of the PRISM model could be addressed. Consequently, CTAC chose the 4 Rs Program to implement across New York State. Previously called the Multiple Family Group (MFG) Service Delivery Model to Reduce Child Disruptive Behaviors (Franco et al., 2008; Gopalan & Franco, 2009; Gopalan et al., in press; McKay et al., 1995; 1999; 2002; 2010; 2011), the 4 Rs Program is a protocol-driven approach specifically designed to improve service use and mental health outcomes for urban, low-income children of color. As its primary service delivery modality, weekly sessions of multiple family groups (involving 6–8 families) are convened to foster within family and between family learning.

Using a common elements approach (Chorpita, Becker, & Daleiden, 2007), the core treatment components of the 4 Rs Program are based on empirically supported family-level influences on disruptive behavior disorders (Dishion et al., 1995; Egeland et al., 1990; Keiley, 2002; Kilgore et al., 2000; Kumpfer & Alvarado, 2003; Loeber & Farrington, 1998; Loeber & Stouthamer-Loeber, 1987; Patterson et al., 1992; Reid et al., 1999; Sampson & Laub, 1994; Tremblay et al., 1991; Tolan et al., 1996), incorporating treatment strategies from behavioral parent training and family therapy. Developed in collaboration with families of youth with disruptive behavior disorders and mental health providers, the resulting curriculum involves sessions which are co-facilitated by clinicians and parent advocates (parents who have previously navigated through the child mental health service system). To date, effectiveness outcome studies indicate benefits in terms of reduced child behavioral difficulties and parent stress, as well as increased rates of overall retention compared to usual care (McKay et al., 2010; 2011).

As the 4 Rs Program addresses consumer-level issues, CTAC focused remaining efforts to address agency (i.e., clinic and providers) level contexts, which are crucial components for successful implementation of evidence-informed practices. According to the PRISM model, innovations should be able to meet local clinic needs, require minimal burden on agency staff, be adaptable/modifiable to improve its usability, can be embedded within work flow, can be easily stopped if needed, and allow for staff to observe results (Feldstein & Glasgow, 2008). Next, we present feedback data from clinic providers and directors regarding the benefits and challenges of integrating the 4 Rs Program within community mental health clinics. Based on this feedback, modifications to the intervention are described in order to facilitate agency fit and overall statewide implementation.

Methods

This study presents provider and director qualitative data gathered as part of a large scale test of the 4Rs Program involving families with children (aged 7–11 years old) meeting diagnostic criteria for oppositional defiant disorder or conduct disorder. Families were recruited from 13 community child mental health clinics within the New York City metropolitan area between October 2006 and October 2010.

Participants

Following implementation of the 4 Rs Program at each child mental health clinic sites, the research team asked intervention facilitators and directors to provide systematic feedback via a brief structured survey tool. Survey tools were sent out to both intervention facilitators (n = 67) and clinic directors (n = 13) in early 2010. At least one survey was collected from 12 of the 13 sites. In all, data was collected from a total of n = 39 respondents (30 facilitators and 9 directors).

Measures

The survey tool was developed by the research team to collect structured feedback from facilitators and directors about their experiences with the implementation of the program. Individual items were created based on key areas identified in the existing literature as being important influences on uptake, adoption and sustainability of practice innovations (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). Separate forms were developed for facilitators and directors. The facilitators' form included five main questions focusing on: (1) perspectives on collaboration, (2) experiences working with the 4 Rs Program, (3) challenges experienced, (4) issues related to diversity among facilitators, and (5) thoughts about the model of pairing clinicians and parent advocates as intervention co-facilitators. The first three questions of the directors' form were similar to the first three questions on the facilitators' form, while the fourth question focused on recommendations for improvement.

Data Analysis

Data from the structured survey tools were analyzed via a multi-phase, iterative process. Each survey was read in-depth by one of the co-authors of this paper with a preliminary code identified. Then, the same staff member re-read each survey and re-coded as necessary to more fully capture prevalent themes across respondents. Next, definitions of each thematic code were created by consensus among research staff raters. Each survey tool was re-coded by two raters separately using the same code definitions as a guide. Each respondent's answer to each question was labeled with as many codes as appropriate. Based upon the complete set of coded surveys, interrater reliability was determined by calculating the percentage of accurate, concurrent identification of codes from 2 raters out of the total number of codes present. Ten randomly selected feedback forms coded by both raters were compared with interrater reliability determined to be 84%. Major themes were then subsequently collapsed and summarized in terms of Benefits, as well as Recommendations for Improvement and Modifications.

Following a participatory collaborative model used to develop the original 4 Rs Program (Hoagwood, Jenson, McKay, & Olin, 2010), a group of facilitators (parent advocates and clinicians), and research staff reviewed the original intervention, addressed concerns and made modifications to the intervention guided by key PRISM-based areas of agency-level factors (Organizational Readiness, Attitudes/Expectations, Workforce Capacity, Resources; see Figure 1). Common themes from qualitative data are underlined below.

Results and Modifications

Benefits

Facilitators and directors reported an overall positive experience in implementing the 4 Rs Program. Many facilitators and several directors described the programs as beneficial for clients. Several facilitators commented on collaboration being useful in working with families, mentioning benefits of collaboration related to working with, relating to, or understanding clients or families.

“Working with entire families was extremely effective because all members were part of the process.”

Many also reported that the intervention was helpful and relevant to clients' needs.

“Difficult to engage families received treatment they may not have gotten … Families experienced support from other families”

Some specifically mentioned the usefulness of the parent advocates' inclusion in the model, with particular emphasis on the parent advocates' unique relationship to parent clients.

“Helpful to have parent advocate who could work with families in a different way.”

Other positive feedback focused on building of facilitator skills (e.g., group process, parenting, teamwork, engagement) through the use of the model.

“I learned a lot about the group process and how to work with co-facilitators. I was also surprised by how much I learned about parenting …”

Lastly, eight of the facilitators specifically mentioned the strength of the model, including references to the strengths of the materials, trainers, support, or overall curriculum.

“I think the curriculum was great, and interactive, and gave the families lots to collaborate on with each other and their own families.”

Recommendations for Improvement and Modifications

Readiness and Expectations

Facilitators highlighted specific challenges related to collaboration, such as lack of role clarity between co-facilitators or between the agency and the research team, communication problems, and organizational climate issues.

“It has been a challenge to work with people of different backgrounds and education levels, but I believe the personalities played more of a role in running the group.”

Each of these concerns stressed the need for supporting agency readiness. As a result, CTAC organized Learning Collaboratives (LCs) to support the next phase of more widespread implementation. LCs are organized around a 9-month period, involving groups of interested agencies working regularly together with a leader to problem-solve agency-level barriers to implementation as well as foster idea exchange across members (IHI, 2003; Kilo, 1998). LCs have been used successfully to implement evidence-based engagement strategies within child mental health clinics (Cavaleri et al., 2006; 2010). At this next phase, each agency was asked to organize a 3-person Quality Improvement Team (QIT; agency director, clinical supervisor, and data manager) to build support from within the agency. In doing so, the QIT managed agency-level barriers to facilitation while intervention facilitators can focus solely on clinical delivery of the model.

Further, several facilitators commented on the need for more support or training, including wanting or expecting more or different support, training, or supervision than was provided. Many facilitators also noted challenges with clients, such as difficulty engaging families, staying on time, and managing members' behaviors within the group.

“Feeling frustration at times with a parent's seeming unmotivation to participate & make any changes.”

In order to better prepare clinic facilitators to implement the 4R's and 2S's model successfully, training was expanded from 3 to 5 hours to provide greater support to staff, and focused on orienting staff to the model and materials, and enhancing experiential activities. Modeling of appropriate group management and facilitation were completed by the trainers who are past clinician and parent advocate facilitators. An important focus also included addressing clinician anxiety in facilitating the groups, problem-solving common areas of concern, building co-facilitation skills, and clarifying expectations. Specific questions around group management were integrated throughout the training along with practice of these skills through mock role-plays. Clinical supervisors were also invited to attend the training to understand the model better and support their staff. In this way, less time would be spent with supervision of the 4 Rs program outside of providers' existing work tasks, and instead, be integrated within the existing organizational and supervision structure.

Workforce capacity

Many facilitators and several directors also noted concerns about time, scheduling, productivity pressures, or the amount of work or energy required. As a result, despite strong enthusiasm for the 4R's and 2S's, clinical practice exigencies hindered full actualization of the model.

“It was challenging to maintain the energy level of the early sessions and commit an evening to the group every week for 16 weeks”

Concerns around time and materials indicated a deterring complexity of the original model, and therefore, became a primary focus of the collaborative revisions committee. Separate facilitator and client materials were streamlined and repackaged into one manual used by both clients and facilitators, modeled after materials used in a successful dissemination project for adult mental health (Salerno et al., 2011), with an additional small facilitator guidebook available for review of key core competencies. Based on the typical client population characteristics, the main manual was revised to simplify structure, language, and activities. As a result, less preparation time outside of the group would be necessary. Reducing session length from 90 minutes to 60 minutes allowed for better fit into current insurance billing models. To reduce the burden of time spent due to intervention-specific supervision, existing clinic supervisors were concurrently instructed in the 4 Rs Program along with facilitators, and were given recommendations for incorporating support for facilitators in regular clinic supervision. Finally, co-facilitators were encouraged to split weekly between-session calls. Additionally, parent advocates were encouraged to make calls as they were often better able to engage families and did not have the same productivity demands as clinicians. Facilitators were instructed that calls should be quick check-ins to encourage completion of the homework and participation in the next session.

Resources and Flexibility

Other areas of improvement included providing ongoing support and increasing flexibility through the development of 4 Rs Program resources. Although not specifically addressed through the respondent surveys, anecdotal experiences by research staff through their interactions with clinic facilitators and directors indicated the need for additional modifications. A number of web-based resources were developed for facilitators to download, including a video of past facilitators describing their experiences in addressing a number of frequently asked questions. Further, personalizable materials to promote the 4 Rs Program for clients were developed along with guidance on client treatment planning and progress notes specific to the model. The facilitators were also provided additional opportunities to talk with the model trainers about half way through their first round of groups through a series of conference calls. A significant concern in today's climate is the changing regulatory guidelines for clinics leaving many unsure how to adapt to them. Specific to this model, a number of online resources are available, including online documents, webinars, and the LC process to address how best to integrate the model in the current fiscal climate.

Given that every clinic functions somewhat differently, flexibility in the implementation of the 4 Rs Program is vital. Consequently, additional resources include guidance on areas for further adaptation, such that facilitators maintain the key, evidence-informed treatment components (Rs and Ss), but can be flexible in how components are presented. The manual is unscripted, focusing on key talking points, discussion questions, and activities in order to allow facilitators the flexibility in presenting the information to best meet their clients' needs. Moreover, flexibility has been introduced in the targeted population with “minimum requirement” of an age range and broad behavioral issues, with encouragement to use clinician judgment. Lastly, although we recommended an ideal pairing of a clinician and parent advocate as co-facilitators, clinics are allowed to determine which makes the most fiscal and clinical sense.

Current status

The current implementation of the 4 Rs Program is underway, with 28 agencies across New York State participating in five Learning Collaboratives beginning the first round of intervention groups. Through the LC process, two more core areas based on the PRISM model have been introduced. First, a focus on observable results has led to the development of an online portal system allowing the tracking of performance indicators which helps clinics determine whether the model adds value to their programs. These are straightforward indicators of client involvement and outcomes (e.g., attendance, discontinuation, client outcomes, facilitator feedback, and fidelity). Through the portal system, LC participating clinics will input and track their performance indicators, as well as instantly produce downloadable reports and graphs to facilitate ongoing service monitoring. For example, seeing attendance patterns over time while the 4 Rs Program is implemented can help facilitators determine whether or not they need to increase engagement efforts.

Second, each of the above mentioned modifications helps to integrate the model into regular clinic practice allowing for sustainability. During the LC meetings, discussion of barriers to sustaining the model will be addressed with clinics creating plans for future implementation. The practice of integrating as many components as possible within regular agency contexts helps to eliminate the need for outside implementation supports. Moreover, the development of simple standalone materials will greatly influence future use of the model.

Conclusion

To advance the first statewide implementation of the 4 Rs Program, CTAC staff utilized the PRISM conceptual framework to guide model modifications serving to enhance agency-level uptake. A collaborative approach (Hoawood, Jenson, McKay, & Olin, 2010) guided model revisions such that the resulting program manual was co-created by family members, providers, and research staff to promote the existing evidence base in “user friendly” language. While the main components of the model based on the evidence-base remain the same, the revised version is streamlined to cut down on complexity, reduce the number of materials, better fit into current clinic structures, and is supported through a learning collaborative process.

As a potential limitation to current efforts, the larger effectiveness study of the original intervention did not collect client feedback on their experience of the 4 Rs Program. However, a main component of the 4 Rs Program has always been to enhance engagement of families in child mental health service use, from the utilization of parent advocates in development and facilitation through to language of the intervention materials. Of particular importance is the model's focus on key concepts of engagement from past research, such as addressing motivational (e.g., assessing stigma in services) and concrete barriers (e.g., transportation issues) to care, as well as making mid-week calls to families (McKay & Bannon, 2004; McKay, Stoewe, McCadam, & Gonzales, 1998; McKay et al., 2004). Consequently, consumer factors related to service use barriers and intervention acceptability are unlikely to impede implementation efforts.

It is anticipated that the collective experience, feedback, and resources resulting from this statewide implementation process will serve as an example for future efforts within the child mental health field. Additionally, the inclusion of brief performance indicators will also provide data on the extent to which modifications made to the 4 Rs Program result in attenuated outcomes relative to the original clinical trial. If successful, the PRISM model could further serve as a guide in the development phase of evidence-informed practices to pre-empt many of these common agency, provider, and consumer level challenges of implementation.

Acknowledgments

This project was supported by award number R01MH072649 from the National Institute of Mental Health (NIMH). Salary support (only) for this study also came from NIMH (F32MH090614). The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institute of Mental Health or the National Institutes of Health.

References

  1. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health. 2011;38:4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Agency for Healthcare Research and Quality. (AHRQ) AHRQ News and Numbers. Author; Rockville, MD: Apr 22, 2009. Mental disorders among most costly conditions in children. Retrieved from http://www.ahrq.gov/news/nn/nn042209.htm. [Google Scholar]
  3. Alvidrez J. Ethnic variations in mental health attitudes and service use among low-income African American, Latina, and European American young women. Community Mental Health Journal. 1999;35(6):515–530. doi: 10.1023/a:1018759201290. [DOI] [PubMed] [Google Scholar]
  4. Angold A, Costello EJ. The epidemiology of disorders of conduct: Nosological issues and comorbidity. In: Hill J, Maughan B, editors. Conduct disorders in childhood and adolescence. Cambridge University Press; New York, NY: 2001. pp. 126–168. [Google Scholar]
  5. Aos S, Mayfield J, Miller M, Yen W. Evidence-based treatment of alcohol, drug, and mental health disorders: Potential benefits, costs, and fiscal impacts for Washington State. Washington State Institute for Public Policy; Olympia, WA: 2006. Publication No. 06-06-3901. [Google Scholar]
  6. Bickman L. A continuum of care: More is not always better. American Psychologist. 1996;51(7):689–701. doi: 10.1037//0003-066x.51.7.689. [DOI] [PubMed] [Google Scholar]
  7. Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry. 2008;47(10):114–1119. doi: 10.1097/CHI.0b013e3181825af8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bruns EJ, Hoagwood KE. State implementation of evidence-based practice for youths, part I: Responses to the state of the evidence. Journal of the American Academy of Child & Adolescent Psychiatry. 2008;47:369–373. doi: 10.1097/CHI.0b013e31816485f4. [DOI] [PubMed] [Google Scholar]
  9. Bruns EJ, Hoagwood KE, Rivard JC, Wotring J, Marsenich L, Carter B. State implementation of evidence-based practice for youths, part II: Recommendations for research and policy. Journal of the American Academy of Child & Adolescent Psychiatry. 2008;47:499–504. doi: 10.1097/CHI.0b013e3181684557. [DOI] [PubMed] [Google Scholar]
  10. Burns BJ, Fisher SK, Ganju V, Lane GM, Nazzaro MB, Rivard JC, Roberts K. A guide for selecting and adopting evidence-based practices for children and adolescents with disruptive behavior disorders. Substance Abuse and Mental Health Services Administration; Center for Mental Health Services; Child, Adolescent and Family Branch, U.S. Department of Health and Human Services; Rockville, MD: in press. [Google Scholar]
  11. Burns BJ, Hoagwood K, editors. Evidence-based practice part I: A research update. Child and Adolescent Psychiatric Clinics of North America. 2004;12:11–13. doi: 10.1016/j.chc.2004.11.001. [DOI] [PubMed] [Google Scholar]
  12. Cavaleri MA, Gopalan G, McKay MM, Appel A, Bannon WM, Bigley MF, Fazio M, Harrison M, Nayowith G, Salerno A, Sher T, Thalor S. Impact of a learning collaborative to improve child mental health service use. Best Practices in Mental Health: An International Journal. 2006;2(2):67–79. [Google Scholar]
  13. Cavaleri MA, Gopalan G, McKay MM, Messam T, Elwyn L. The effect of a learning collaborative to improve engagement in child mental health services. Children and Youth Services Review. 2010;32(2):281–285. [Google Scholar]
  14. Cavaleri MA, Hoagwood KH, McKay MM. Examining the implementation of the Child and Family Clinic Plus Initiative [Unpublished monograph] 2009 [Google Scholar]
  15. Chorpita BF, Becker KD, Daleiden EL. Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of the American Academy of Child and Adolescent Psychiatry. 2007;46(5):647–652. doi: 10.1097/chi.0b013e318033ff71. [DOI] [PubMed] [Google Scholar]
  16. David-Ferdon C, Kaslow NJ. Evidence-based psychosocial treatments for child and adolescent depression. Journal of Clinical Child & Adolescent Psychology. 2008;37(1):62–104. doi: 10.1080/15374410701817865. [DOI] [PubMed] [Google Scholar]
  17. Dishion TJ, French DC, Patterson GR. The development and ecology of antisocial behavior. In: Cicchetti D, Cohen DJ, editors. Developmental psychopathology, vol. 2: Risk, disorder, and adaptation. John Wiley & Sons; Oxford, England: 1995. pp. 421–471. [Google Scholar]
  18. Egeland B, Kalkoske M, Gottesman N, Erickson MF. Preschool behavior problems: Stability and factors accounting for change. Journal of Child Psychology and Psychiatry. 1990;31(6):891–909. doi: 10.1111/j.1469-7610.1990.tb00832.x. [DOI] [PubMed] [Google Scholar]
  19. Essock SM, Covell NH, Leckman-Westin E, Lieberman JA, Sederer LI, Kealey E, Finnerty MT. Identifying clinically questionable psychotropic prescribing practices for Medicaid recipients in New York State. Psychiatric Services. 2009;60:1595–1602. doi: 10.1176/ps.2009.60.12.1595. [DOI] [PubMed] [Google Scholar]
  20. Eyberg SM, Nelson MM, Boggs SR. Evidence—based psychosocial treatments for children and adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology. 2008;37(1):215–237. doi: 10.1080/15374410701820117. [DOI] [PubMed] [Google Scholar]
  21. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) Joint Commission Journal on Quality and Patient Safety. 2008;34(4):228–243. doi: 10.1016/s1553-7250(08)34030-6. [DOI] [PubMed] [Google Scholar]
  22. Finnerty M, Altmansberger R, Bopp J, Carpinello S, Docherty JP, Fisher W, Felton C. Using state administrative and pharmacy data bases to develop a clinical decision support tool for schizophrenia guidelines. Schizophrenia Bulletin. 2002;28:85–94. doi: 10.1093/oxfordjournals.schbul.a006929. [DOI] [PubMed] [Google Scholar]
  23. Fixsen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F. Implementation research: A synthesis of the literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; Tampa, F.L.: 2005. FMHI Publication #231. [Google Scholar]
  24. Franco LM, Dean-Assael KM, McKay MM. Multiple family groups to reduce youth behavioral difficulties. In: LeCroy CW, editor. Handbook of evidence-based treatment manuals for children and adolescents. Oxford University Press; New York, NY: 2008. pp. 546–590. [Google Scholar]
  25. Frick PJ. Conduct disorders and severe antisocial behavior. Plenum Press; New York, NY: 1998. [Google Scholar]
  26. Ganju V. Implementation of evidence-based practices in state mental health systems: Implications for research and effectiveness studies. Schizophrenia Bulletin. 2003;29(1):125–131. doi: 10.1093/oxfordjournals.schbul.a006982. [DOI] [PubMed] [Google Scholar]
  27. Gleacher AA, Nadeem E, Moy AJ, Whited AL, Albano A, Radigan M, Wang Rui, Chassman J, Myrhol-Clarke B, Hoagwood KE. Statewide CBT training for clinicians and supervisors treating youth: The New York State Evidence Based Treatment Dissemination Center. Journal of Emotional and Behavioral Disorders. 2011;19(3):182–192. doi: 10.1177/1063426610367793. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Gopalan G, Bannon WM, Dean-Assael K, Fuss A, Gardner L, LaBarbara B, McKay M. Multiple Family Groups: An engaging mental health intervention for child welfare involved families. Child Welfare. In Press. [PMC free article] [PubMed] [Google Scholar]
  29. Gopalan G, Franco L. Multiple family groups to reduce disruptive behaviors. In: Gitterman A, Salmon R, editors. Encyclopedia of social work with groups. Routledge; New York, NY: 2009. [Google Scholar]
  30. Hoagwood K, Burns BJ, Kiser L, Ringeisen H, Schoenwald SK. Evidence-based practice in child and adolescent mental health services. Psychiatric Services. 2001;52(9):1179–1189. doi: 10.1176/appi.ps.52.9.1179. [DOI] [PubMed] [Google Scholar]
  31. Hoagwood KE, Jensen PS, McKay M, Olin SS, editors. Children's mental health research: The power of partnerships. Oxford University Press; New York, NY: 2010. [Google Scholar]
  32. Institute for Healthcare Improvement. (IHI) The breakthrough series: IHI's collaborative model for achieving breakthrough improvement in innovation series. Institute for Healthcare Improvement; Boston, MA: 2003. [Google Scholar]
  33. Institute of Medicine. (IOM) Crossing the quality chasm: A new health system for the 21st century. National Academy Press; Washington, D.C.: 2001. [PubMed] [Google Scholar]
  34. Kazak AE, Hoagwood K, Weisz JR, Hood K, Kratochwill TR, Vargas LA, Banez GA. A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist. 2010;65(2):85–97. doi: 10.1037/a0017784. [DOI] [PubMed] [Google Scholar]
  35. Kazdin AE, Weisz JR, editors. Evidence-based psychotherapies for children and adolescents. Guilford Press; New York, NY: 2003. [Google Scholar]
  36. Kazdin AE, Whitley MK. Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior. Journal of Consulting and Clinical Psychology. 2003;71(3):504–515. doi: 10.1037/0022-006x.71.3.504. [DOI] [PubMed] [Google Scholar]
  37. Keiley MK. The development and implementation of an affect regulation and attachment intervention for incarcerated adolescents and their parents. The Family Journal. 2002;10(2):177–189. [Google Scholar]
  38. Kilgore K, Snyder J, Lentz C. The contribution of parental discipline, parental monitoring, and school risk to early-onset conduct problems in African American boys and girls. Developmental Psychology. 2000;36(6):835–845. doi: 10.1037//0012-1649.36.6.835. [DOI] [PubMed] [Google Scholar]
  39. Kilo CM. A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvement's breakthrough series. Quality Management in Health Care. 1998;6:1–13. doi: 10.1097/00019514-199806040-00001. [DOI] [PubMed] [Google Scholar]
  40. Kumpfer KL, Alvarado R. Family-strengthening approaches for the prevention of youth problem behaviors. American Psychologist. 2003;58(6–7):457–465. doi: 10.1037/0003-066X.58.6-7.457. [DOI] [PubMed] [Google Scholar]
  41. Loeber R, Burke JD, Lahey BB, Winters A, Zera M. Oppositional defiant and conduct disorder: A review of the past 10 years, part I. Journal of the American Academy of Child & Adolescent Psychiatry. 2000;39(12):1468–1484. doi: 10.1097/00004583-200012000-00007. [DOI] [PubMed] [Google Scholar]
  42. Loeber R, Farrington DP. Serious & violent juvenile offenders: Risk factors and successful interventions. Sage Publications, Inc.; Thousand Oaks, CA: 1998. [Google Scholar]
  43. Loeber R, Stouthamer-Loeber M. The prediction of delinquency. In: Quay HC, editor. Handbook of juvenile delinquency. Wiley; New York, NY: 1987. pp. 325–416. [Google Scholar]
  44. McClellan JM, Werry JS. Evidence-based treatments in child and adolescent psychiatry: An inventory. Journal of the American Academy of Child & Adolescent Psychiatry. 2003;42(12):1388–1400. doi: 10.1097/01.chi.0000092322.84052.88. [DOI] [PubMed] [Google Scholar]
  45. McKay MM, Bannon W. Engaging families in child mental health services. Child and Adolescent Psychiatric Clinics of North America. 2004;13:905–921. doi: 10.1016/j.chc.2004.04.001. [DOI] [PubMed] [Google Scholar]
  46. McKay MM, Gonzales J, Quintana E, Kim L, Abdul-Adil J. Multiple family groups: An alternative for reducing disruptive behavioral difficulties of urban children. Research on Social Work Practice. 1999;9(5):593–607. [Google Scholar]
  47. McKay MM, Gonzales JJ, Stone S, Ryland D, Kohner K. Multiple family therapy groups: A responsive intervention model for inner city families. Social Work with Groups. 1995;18(4):41–56. [Google Scholar]
  48. McKay MM, Gopalan G, Franco L, Dean-Assael K, Chacko A, Jackson J, Fuss A. A collaboratively designed child mental health service model: Multiple family groups for urban children with conduct difficulties. Research in Social Work Practice. 2011;21(6):664–674. doi: 10.1177/1049731511406740. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. McKay MM, Gopalan G, Franco LM, Kalogerogiannis KN, Olshtain-Mann O, Bannon W, Elwyn L, Goldstein L, Umpierre M. It takes a village to deliver and test child and family-focused services. Research in Social Work Practice. 2010;20(5):476–482. doi: 10.1177/1049731509360976. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. McKay MM, Harrison ME, Gonzales J, Kim L, Quintana E. Multiple-family groups for urban children with conduct difficulties and their families. Psychiatric Services. 2002;53(11):1467–1468. doi: 10.1176/appi.ps.53.11.1467. [DOI] [PubMed] [Google Scholar]
  51. McKay MM, Hibbert R, Hoagwood K, Rodriguez J, Murray L, Legerski J, Fernandez D. Integrating evidence-based engagement interventions into “real world” child mental health settings. Brief Treatment and Crisis Intervention. 2004;4(2):177–186. [Google Scholar]
  52. McKay MM, Stoewe J, McCadam K, Gonzales J. Increasing access to children mental health services for urban children and their caregivers. Health & Social Work. 1998;23(1):9. doi: 10.1093/hsw/23.1.9. [DOI] [PubMed] [Google Scholar]
  53. Meyer GJ, Finn SE, Eyde LD, Kay GG, Moreland KL, Dies RR, Reed GM. Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist. 2001;56:128–165. doi:10/1037/0003-066X.56.2.128. [PubMed] [Google Scholar]
  54. National Association of State Mental Health Program Directors. (NASMHPD) Integrating behavioral health and primary care services: Opportunities and challenges for state mental health authorities. Author; Alexandria, VA: 2005. Retrieved from http://www.nasmhpd.org/publicationsmeddir.cfm. [Google Scholar]
  55. National Association of State Mental Health Program Directors Research Institute, Inc. (NRI) The impact of the state fiscal crisis on the public mental health system. 2010 Feb 24; [Congressional Briefing] Retrieved from http://www.nri-inc.org/reports_pubs/2010/NRI_SMHA_Budget_2010.pdf.
  56. New Freedom Commission on Mental Health Achieving the promise: Transforming mental health in America, final report. 2003 [DHHS Publication SMA-03-3832]. Retrieved from http://www.mentalhealthcommission.gov/reports/FinalReport/downloads/FinalReport.pdf.
  57. Patterson GR, Reid JB, Dishion TJ. A social learning approach: IV. antisocial boys. Castalia; Eugene, OR: 1992. [Google Scholar]
  58. Pelham WE, Fabiano GA. Evidence-based psychosocial treatments for attention-deficit/hyperactivity disorder. Journal of Clinical Child & Adolescent Psychology. 2008;37(1):184–214. doi: 10.1080/15374410701818681. [DOI] [PubMed] [Google Scholar]
  59. Reid JB, Eddy JM, Fetrow RA, Stoolmiller M. Description and immediate impacts of a preventive intervention for conduct problems. American Journal of Community Psychology. 1999;27(4):483–517. doi: 10.1023/A:1022181111368. [DOI] [PubMed] [Google Scholar]
  60. Rogers SJ, Vismara LA. Evidence-based comprehensive treatments for early autism. Journal of Clinical Child & Adolescent Psychology. 2008;37(1):8–38. doi: 10.1080/15374410701817808. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Salerno T, Margolies P, Cleek A, Pollock M, Gopalan G, Jackson C. Wellness self-management: An adaptation of the Illness Management and Recovery Practice in New York State. Psychiatric Services. 2011;62(5) doi: 10.1176/appi.ps.62.5.456. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Sampson RJ, Laub JH. Urban poverty and the family context of delinquency: A new look at structure and process in a classic study. Child Development. 1994;65(2):523–540. [PubMed] [Google Scholar]
  63. Silverman WK, Hinshaw SP. The second special issue on evidence-based psychosocial treatments for children and adolescents: A ten-year update. Journal of Clinical and Child Adolescent. 2008;37(1):1–7. [Google Scholar]
  64. Silverman WK, Pina AA, Viswesvaran C. Evidence-based psychosocial treatments for phobic and anxiety disorders in children and adolescents. Journal of Clinical Child & Adolescent Psychology. 2008;37(1):105–130. doi: 10.1080/15374410701817907. [DOI] [PubMed] [Google Scholar]
  65. Soni A. The five most costly children's conditions, 2006: Estimates for the U.S. civilian noninstitutionalized children, Ages 0–17. Agency for Healthcare Research and Quality; Rockville, MD: Apr, 2009. Statistical Brief # 242. http://www.meps.ahrq.gov/mepsweb/data_files/publications/st242/stat242.pdf. [PubMed] [Google Scholar]
  66. Substance Abuse and Mental Health Services Administration's National Registry of Evidence-based Programs and Practices. (SAMHSA) U.S. Department of Health and Human Services Substance Abuse & Mental Health Services Administration; 2010. Retrieved from http://nrepp.samhsa.gov/ [Google Scholar]
  67. Tolan PH, Henry D. Patterns of psychopathology among urban poor children: Comorbidity and aggression effects. Journal of Consulting and Clinical Psychology. 1996;64:1094–1099. doi: 10.1037//0022-006x.64.5.1094. [DOI] [PubMed] [Google Scholar]
  68. Torrey WC, Drake RE, Dixon L, Burns BJ, Flynn L, Rush AJ, Klatzker D. Implementing evidence-based practices for persons with severe mental illnesses. Psychiatric Services. 2001;52:45–50. doi: 10.1176/appi.ps.52.1.45. [DOI] [PubMed] [Google Scholar]
  69. Tremblay RE, Loeber R, Gagnon C, Charlebois P. Disruptive boys with stable and unstable high fighting behavior patterns during junior elementary school. Journal of Abnormal Child Psychology. 1991;19(3):285–300. doi: 10.1007/BF00911232. [DOI] [PubMed] [Google Scholar]
  70. United States Public Health Service. (USPHS) Report of the Surgeon General's Conference on Children's Mental Health: A national action agenda; Rockville, MD: U.S. Department of Health and Human Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health; 2000. [Google Scholar]
  71. Wahler R, Dumas J. Attentional problems in dysfunctional mother-child interactions: An interbehavior model. Psychological Bulletin. 1989;105:116–130. doi: 10.1037/0033-2909.105.1.116. [DOI] [PubMed] [Google Scholar]
  72. Weisz JR, Hawley KM, Doss AJ. Empirically tested psychotherapies for youth internalizing and externalizing problems and disorders. Child & Adolescent Psychiatry Clinics of North America. 2004;13(4):729–815. doi: 10.1016/j.chc.2004.05.006. [DOI] [PubMed] [Google Scholar]
  73. Weisz JR, Jensen AL. Child and adolescent psychotherapy in research and practice contexts: Review of the evidence and suggestions for improving the field. European Journal of Child and Adolescent Psychiatry. 2001;10:12–18. doi: 10.1007/s007870170003. [DOI] [PubMed] [Google Scholar]
  74. Weisz JR, Jensen-Doss A, Hawley KM. Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist. 2006;67(7):671–689. doi: 10.1037/0003-066X.61.7.671. [DOI] [PubMed] [Google Scholar]
  75. Weisz JR, Weiss B, Han SS, Granger DA, Morton T. Effects of psychotherapy with children and adolescents revisited: A meta-analysis of treatment outcome studies. Psychiatric Bulletin. 1995;117:450–468. doi: 10.1037/0033-2909.117.3.450. [DOI] [PubMed] [Google Scholar]
  76. Yannacci J, Rivard JC, Ganju VK. State activities in implementing evidence-based programs for children, youth, and families. Paper presented at the 18th Annual Research Conference, A System of Care for Children's Mental Health: Expanding the Research Base; Tampa, FL. 2005, March 6–9; Retrieved from http://www.nriinc.org/reports_pubs/2005/EBPImpActsforKids2005.pdf. [Google Scholar]

RESOURCES