Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Mar 1.
Published in final edited form as: J Clin Child Adolesc Psychol. 2014 Jan 24;43(2):145–157. doi: 10.1080/15374416.2013.869749

Scaling up Evidence-based Practices for Children and Families in New York State: Towards Evidence-based Policies on Implementation for State Mental Health Systems

Kimberly Eaton Hoagwood 1, S Serene Olin 1, Sarah Horwitz 1, Mary McKay 2, Andrew Cleek 2, Alissa Gleacher 1, Eric Lewandowski 1, Erum Nadeem 1, Mary Acri 1, Ka Ho Brian Chor 1, Anne Kuppinger 3, Geraldine Burton 1,3, Dara Weiss 1, Samantha Frank 1, Molly Finnerty 4, Donna M Bradbury 4, Kristin M Woodlock 4, Michael Hogan 1
PMCID: PMC3954943  NIHMSID: NIHMS546526  PMID: 24460518

Abstract

Dissemination of innovations is widely considered the sine qua non for system improvement. At least two dozen states are rolling-out evidence-based mental health practices targeted at children and families using trainings, consultations, webinars, and learning collaboratives to improve quality and outcomes. In New York State (NYS) a group of researchers, policy-makers, providers and family support specialists have worked in partnership since 2002 to redesign and evaluate the children’s mental health system. Five system strategies driven by empirically-based practices and organized within a state-supported infrastructure have been used in the child and family service system with over 2,000 providers: (a) business practices; (b) use of health information technologies in quality improvement; (c) specific clinical interventions targeted at common childhood disorders; (d) parent activation; and (e) quality indicator development. The NYS system has provided a laboratory for naturalistic experiments. We describe these initiatives, key findings and challenges, lessons learned for scaling, and implications for creating evidence-based implementation policies in state systems.

Introduction

Over the past 25 years, research has identified numerous efficacious interventions for children with behavioral and emotional challenges, including cognitive behavioral therapies for internalizing disorders, parent management strategies for disruptive behavior disorders, medication therapies, and combined interventions. Reviews have also identified effective practices to support service delivery such as engagement strategies, diagnostic screens, standardized clinical assessments, and measurement feedback systems (Bickman, 2008; Burns & Hoagwood, 2004; Kazdin & Weisz, 2003; McCellan & Werry, 2003; Silverman & Hinshaw, 2008; Weisz, Weiss, Han, Granger, & Morton, 1995). States and healthcare systems are taking a leading role in supporting the adoption of empirically-validated interventions for the populations they serve. To date, more than twenty states are actively implementing evidence-based psychosocial therapies (Bruns, Hoagwood, & Hamilton, 2008) or medication practices (Essock et al., 2009). Some states (e.g., CA, CO, MI, OH) have contracts with purveyor organizations to implement specific evidence-based practices (EBPs) (Fixsen, Blase, Metz, & Van Dyke, 2013; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Substance Abuse and Mental Health Services Administration, accessed 12 December 2012). This growing demand for EBP implementation by states is matched by a demand for practical strategies, toolkits, and brief and feasible measurement systems (Ganju, 2003; Torrey et al., 2001).

Efforts to improve the quality of mental health services in states has immediate public health relevance: the Agency for Healthcare Research and Quality (AHRQ) (2009) reported that the treatment of children’s mental disorders is among the top 5 most costly medical conditions, constituting $8.9 billion annually, or 9% of total medical spending on children birth through 17. However, the public mental health system, which is comprised of a diverse array of community based treatment, rehabilitation and support services, is managed at the state and sometimes the county level, with financing largely dependent upon Medicaid and state general funds. Recent state budget shortfalls have led to significant reduction in resources to fund these important services. According to the Center on Budget and Policy Priorities, cumulative state budget gaps exceeded $350 billion for the period of FY 2009–11 (Miller, 2011).

A recent fiscal study of the economic impact of implementing EBPs in one state indicated that adoption can be economically sound, resulting in a 56 percent rate of return on investment (Aos, Mayfield, Miller, & Yen, 2006). Yet states face significant challenges in financing EBP implementation, primarily because they must often pool divergent funding streams (e.g. Medicaid, state general revenue, federal block grants, or private foundation funding). In some cases, states seek private foundation funding for startup costs (necessary for successful implementation), and find implementation is interrupted because the grant funding was time-limited (Preethy et al., 2008). With only a short time frame in which to get EBPs implemented, states scramble to find tools and strategies to install EBPs quickly into their systems. In the context of severe fiscal constraints, practical decision-making by state leadership, based upon empirical evidence and guided by feasible protocols is needed. Unfortunately, state dissemination and implementation efforts have been inconsistently and inadequately studied (Wisdom, Chor, Hoagwood, & Horwitz, 2013). Strategies to scale up EBPs in response to policy mandates have not been guided by empirical knowledge about best practices for successful implementation (McHugh & Barlow, 2010). Therefore, the benefits of EBPs have not been consistently realized. While significant resources have been channeled into EBP development, it takes 17 years for 14% of original research to change practice (Green, 2001). To more effectively reap the benefits of science in practice and scale up EBPs in states, different system strategies are needed.

Implementation frameworks for state systems scaling

With a series of grants from NIMH (R24MH068708, formerly P20MH068708; P20MH078178; P30MH090322) and support from the NYS Office of Mental Health (OMH), beginning in 2002, our research team initiated a series of partnered projects which created a unique state services research laboratory. This laboratory provided a platform upon which to launch a series of naturalistic experiments, largely focused on strategies for scaling up evidence-based practices for children and families. Our first one-year state planning grant (R24MH068708 from 2003–2004) enabled us to develop a vertical and horizontal network of state policymakers, agency directors, mid-management supervisors, clinicians, family partners, and researchers. The network has been sustained over time and is instrumental to maintaining momentum among the projects (described below). Our first NYS planning grant also supported development of a systems instrument to assess dimensions of organizational readiness (Schoenwald et al., 2008; Schoenwald & Hoagwood, 2001; Schoenwald, Hoagwood, Atkins, Evans, & Ringeisen, 2010) from the perspectives of each of the stakeholder groups mentioned above. The emphasis was on the differential salience of various factors (e.g., organizational, policy, research) for facilitating implementation of EBPs in the child-serving system.

Our second grant, P20MH078178 (from 2005 to 2010), focused on scaling up three large projects driven by NYS policy priorities: evidence-based trauma treatments subsequent to the events of September 11th (Gleacher et al., 2011; North et al., 2008); a school and clinic screening initiative to improve early detection of social-emotional development disruptions (New York State Office of Mental Health, 2009); and a training program for 400 family support specialists to improve parent engagement in children’s services (Olin, Hoagwood, Rodriguez, Radigan, et al., 2010; Rodriguez et al., 2011). Based on lessons learned from these studies about impediments to scaling up new practices in the state system, we sought and were awarded an Advanced Services Research Center (P30 MH090322) from NIMH in 2010. Our current Center [also known as the IDEAS Center: Advanced Center on Implementation-Dissemination Science in States for Children and Families] focuses on the development of practical system strategies to facilitate broad system-wide implementation of three major practice foci: clinical evidence-based treatments (EBTs), services (engagement and family support) and organizational (business) practices. One important goal is to assist state leadership in setting evidence-supported policies to improve the children’s mental health system.

Notably, since our Center was funded, a newly emerging challenge in working with states arose. This was associated with the passage of the Patient Protection and Affordable Care Act (ACA) in 2010 (P.L. 111-148). Building on the Dominici-Wellstone Mental Health Parity and Addiction Equity Act of 2008 (P.L. 110-343), the ACA included mental health within its changes to health care; and specifically changed the structure, financing and accountability standards by which mental health services are delivered and paid for in states. These acts also reframed mental health within a broader public health framework (Institute of Medicine, 2006; Institute of Medicine, 2009; Kohn, Corrigan, & Donaldson, 2000). One implication for state system implementation of children’s services is that the traditional model of a separate specialty mental health system that operates in isolation from the rest of healthcare is rapidly vanishing. Stand-alone specialized mental health services are being replaced with regionalized networks of healthcare providers (Kelleher, 2010). These changes are putting pressure on states to revamp their priorities to emphasize accountability, quality standards, and to reduce the use of costly services. The impact of these broad healthcare policy changes on mental health and public service sectors are yet unclear, but will likely have significant implications for strategies states employ to scale up change efforts.

This new emphasis on healthcare accountability and quality in conjunction with lessons learned from our earlier efforts to scale up new practices has challenged us to focus on effective and efficient system strategies. We have selected five: business practices; use of health information technologies (i.e., measurement feedback systems) in quality improvement; specific clinical interventions; parent activation; and quality indicator development. The first two: business practices and health information technologies, provide infrastructure support to the network of licensed clinics (N=346) serving the clinical needs of children and families. The clinical practices and parent activation initiatives constitute the core of the clinical and support services that NYS provides. The initiatives involving development of quality indicators are new approaches to better align the mental health system with the general healthcare system.

1. Business Practices

Effective business practices are critical for the long-term viability of state-supported mental health agencies given the requirements for fiscal reporting and payments for reimbursable services. Business practices are also relevant to the intra-organizational context as they affect the structure with which administrative billing systems function.

For years most New Yorkers enrolled in Medicaid have received their physical healthcare services in Medicaid Managed Care Plans. However, a ‘carve-out’ was in place for those with either a “serious mental illness” (adults) or a “serious emotional disturbance” (children); behavioral health services were offered via traditional fee for service Medicaid administrated by NYS. This ‘carve-out’ facilitated the development of a behavioral healthcare system for at-risk children and families. Providers were subsidized to provide care to families without insurance or insufficient commercial insurance through a separate mechanism.

In 2008 NYS OMH began the elimination of these subsidies and raised the average rates clinics received to provide care. Almost immediately following that policy decision, NYS OMH began a more drastic change in its Medicaid payments to address significant budget shortfalls brought on by the financial crisis. All children in New York’s publically funded system will soon be placed in managed care for the full range of health and behavioral health treatment. This will have an unpredictable impact on service funding and delivery.

The changing business environment is affecting behavioral health agencies as they shift the way that they provide clinic-based services. Agencies have begun to implement several strategies to improve business efficiencies. These include open access, centralized scheduling, and concurrent documentation (Lloyd, 1998). Open access is a business procedure that enables a client seeking an intake to be seen either immediately upon presenting for care or within 24 hours. Instead of clinicians being responsible for all interactions with a family (including documentation and scheduling), clinics are now moving towards a system called centralized scheduling that treats providers like medical doctors with back to back appointments and little time for paperwork, consultation, or other activities. Centralized scheduling is a process where administrative staff within the clinic complete all scheduling, thus freeing up additional time for clinicians to engage in billable hours of treatment and facilitating “back filling” appointments as cancelations are made. Concurrent documentation is becoming the norm, and clinicians are trained to complete all paperwork associated with an individual or family client collaboratively within each treatment session. This includes case notes, treatment plans, and assessments. Utilization management has also begun to take a more prominent role in clinic operation with services intensity being dictated by need rather than a traditional weekly session. Short-term and flexibly delivered (i.e., individual or group) therapies are being demanded because they rest on a business model that is viable. These business practices are being rolled-out via the Clinic Technical Assistance Center (CTAC), (Hoagwood & McKay, Co-Directors), funded by NYS OMH (see below) using a variety of implementation strategies.

Lessons Learned for Scaling

It is too soon to know whether the business practices will result in more efficient service delivery: an evaluation is underway. At the moment, more than 60 clinics from NYS are participating in multiple CTAC initiatives to improve their financial viability. They include county-run, free standing, and hospital-affiliated programs. A number of clinics participating in these projects have shown substantial progress in improving their productivity and fiscal health. Mixed methods studies are underway to assess the changes these clinics have made and any differences among these sets of clinics to inform future work.

2. Use of Health Information Technology (HIT) in Quality Improvement (QI)

NYS’s public mental health system has lagged considerably behind the general health care world in the adoption of HIT as a QI tool. Recently, the adoption rates of mental health programs using an electronic health record (EHR) have increased substantially in NYS, with 400 million dollars spent developing Regional Health Information Organizations (RHIOs) that allow providers to electronically share information across electronic platforms and practice settings. However, the efforts around information sharing currently apply only to adult systems. Federal meaningful use incentives, put in place under the ACA, provided funding for medical staff in healthcare practices to implement HIT systems in the NYS public health system; such efforts have only begun to focus on behavioral health practices (P.L. 111-148). Thus, development and installation of electronic platforms (outer context system infrastructure) for tracking children’s mental health functioning is greatly needed.

Within children’s mental health, NYS has been experimenting with smaller scale efforts around the use of electronic measurement systems. Adoption of such efforts to date best corresponds to installing measurement feedback systems as an EBP tool for guiding clinical practice through data use.

NYS OMH is committed to broadening the use of measurement feedback systems for clinical management within its licensed clinics. Through a pilot study, we examined implementation challenges of building clinical measurement feedback systems into four community-based mental health clinics. Specifically, we examined the uptake and use of the Contextualized Feedback System (CFS) (Bickman, Kelley, Breda, de Andrade, & Riemer, 2011) (R18HS18036-01, PI: Hoagwood). Due to a variety of factors (i.e., fiscal pressures, conflicting priorities, technological compatibility, innovation-values fit), one agency with two participating clinics discontinued participation in the project. The remaining two clinics that faced similar fiscal pressures built the CFS into daily clinic operations. Integrating CFS involved tailoring it to the specific needs of each clinic. Strategies included incorporating mandatory clinical documentation, determining how site routines could be modified to integrate CFS, and developing local project plans to manage CFS implementation. Weekly consultation calls with a CFS trainer strengthened appropriate use of the system. All users (e.g., leaders, supervisors, clinicians, administrative assistants) were involved in developing an implementation plan and coaching was provided for a year via calls and in person meetings led by a CFS trainer in conjunction with an on-site Master Partner.

To understand the implementation of CFS, an Implementation Index was created to account for two interrelated activities: having the respondents complete the questionnaires and having the clinician view the resulting feedback. Preliminary results indicate variations across clinics and clinicians’ implementation of CFS, with overall averages reflecting low CFS implementation. Interestingly, youth who received more of the CFS intervention (i.e., had higher implementation) showed the fastest improvement during the treatment phase. Results are being analyzed to understand facilitators and barriers associated with CFS implementation to inform future rollouts of MFS systems throughout the state.

To facilitate a broader uptake, NYS OMH is planning to roll-out the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden, 2009) in 2013. Like CFS, MAP is a MFS and offers a broad range of additional resources, such as access to the most current scientific information and to user-friendly measurement tools and clinical protocols (Chorpita & Daleiden, 2009). Using an online database, the MAP system can suggest formal evidence-based programs and can provide detailed recommendations about discrete components of evidence-based treatments relevant to a specific youth’s characteristics. The MAP system framework tracks outcomes and practices on a graphical “dashboard.” Based on a similar process used to train clinicians in the Los Angeles county-wide MAP rollout, NYS plans to scale up MAP to approximately 150 clinicians and their supervisors via webinars and in-person trainings. The initial training will be followed by at least 6 months of biweekly phone consultation for clinicians and supervisors. Calls with clinicians will focus on their ability to successfully deliver the practice support tools with individual clients. For supervisors, calls will focus on supervisor knowledge and guidance of supervisees in the MAP program.

Lessons Learned for Scaling

Electronic platforms for measuring and tracking specific clinical practices and outcomes are a critical component of the changing healthcare system. Behavioral health is lagging behind the rest of healthcare in implementing these technologies. The implementation challenges are similar to those encountered in clinical EBP installation, and the solutions are likely to be similar: targeted consultation and personalized supports to assist providers in the use of the new technologies, along with incentives to use them. We see the use of electronic platforms as providing essential scaffolding for future EBP rollouts.

3. Supports for Implementing EBPs

Clinical interventions are often selected for implementation by state system leadership (outer context) but instantiation of them in agencies is affected by a host of inner context variables including provider training attitudes, capacity, supervisory practices, and strength of executive leadership. In NYS, early and ongoing efforts focused on implementation of specific evidence-based treatments (EBTs).

(a) Evidence Based Treatments

In 2004 NYS OMH created the Evidence-based Treatment Dissemination Center (EBTDC) to train front-line clinicians and supervisors working within clinics, residential treatment centers, and inpatient hospitals on specific EBTs for youth. The EBTDC was created after the successful launch of the training and consultation provided by the Child and Adolescent Trauma Treatment Services (CATS) Consortium—a coordinated set of trauma services provided to 385 children, adolescents and their families affected by September 11th (CATS Consortium, 2007; Hoagwood et al., 2007).

The first two years of EBTDC trained clinicians and supervisors in cognitive behavioral therapy (CBT) for post-traumatic stress disorder (Cohen, Mannarino, & Deblinger, 2006) and depression (Stark, Curry, & Goldman, 2006). The second cycle focused on individual CBT and parent training for disruptive behavior disorders (DBDs) (Lochman & Wells, 2004). The third cycle focused on trauma training for providers working in residential treatment centers and state hospitals. The model involves Ph.D.-level consultants who conduct consultation phone calls every other week with clinicians and supervisors to hone the skills acquired in the in-person training. In addition, consultants meet with the treatment developers on a monthly basis to discuss specific protocols and problem-solve difficult issues and cases. To date, EBTDC has provided training and a full year of distance-learning consultation to 1,552 clinicians and supervisors across 186 of the 346 clinics. The therapies supported by the EBTDC include CBT for depression, trauma-focused CBT, Coping Power, and Parent Management Training for DBDs (Gleacher et al., 2011; Nadeem et al., In press; Pimentel, Hoagwood, Albano, & Regan, 2009).

(b) Multifamily groups for disruptive behavior disorders

Another clinical intervention adopted by NYS for a roll-out within clinics has been the Multifamily Group (MFG) intervention for children with disruptive behavior disorders and their caregivers (McKay et al., 2011), particularly those in socioeconomically disadvantaged communities. The MFG intervention was developed in collaboration with families of youth with DBDs. MFG takes a common elements approach (Chorpita & Daleiden, 2009; Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008) by integrating core components from the empirical literature regarding effective practices for DBDs (i.e., providing contingencies for supporting behavior change in youth, developing positive relationships within the family, and evidence-based engagement techniques to improve retention) (McKay, Nudelman, McCadam, & Gonzales, 1996; McKay, Stoewe, McCadam, & Gonzales, 1998). Core skills, processes, and methods are framed in a strength-based perspective and delivered in a flexible manner through a partnership model with a family peer advocate and a clinician. In addition, MFG relies on multiple generations within a family working collaboratively with other families in a group setting as a way of decreasing the stigma associated with receipt of mental health services. Thus, MFG increases engagement in services as well as provides an efficient service-delivery mechanism. Findings suggest that engagement strategies added to the implementation of the MFG intervention based on Parent Management Training (PMT) increased retention of youth and families (80% vs. 10% completion of treatment) and decreased child disruptive behavioral symptoms and parenting stress (McKay et al., 2011).

(c) Engagement training

As noted above, we have also implemented a set of empirically-based engagement strategies as adjunctive to clinical treatments via a web-based training called Training to Increase Engagement Skills (TIES). TIES uses web-based technology to train front-line staff with specific engagement skills. This approach enables trainers to customize, download, and present family engagement research and strategies to mental health clinicians. The modules include telephone engagement strategies, engagement interventions, clinical interviewing, family service retention, and transition to other services (McKay et al., 1996; McKay et al., 1998).

Lessons Learned for Scaling

The costs of implementing with fidelity even one EBT are enormous. Consequently it is important to be selective when choosing an EBT for implementation. NYS embarked on an ambitious agenda to train the workforce on a set of EBTs and training on supportive services (engagement). At this time, only training in MFG is being offered regularly. This is in part because MFG was envisioned as a package of EBTs that could be repackaged and repurposed for different contexts. There are group and individual models, as well as different trainer combinations (clinician and family partner or family partner only). Most likely the adaptability of the model has lent itself to continued uptake.

4. Parent Activation

The effective implementation of quality services matters only if parents and their children (consumers) use them. Thus, activation of parents to select, access, and use services is an important component of an effective system.

It is well documented that families’ access to and use of services in the public child mental health system is inconsistent and sporadic. Low income, ethnically diverse families in particular encounter significant obstacles to seeking and staying in services. Because parents are the key drivers of services for their children, peer services delivered by professional family support specialists are becoming part of the mental health workforce. Peer specialists are increasingly employed throughout the country in a variety of organizations (Walker & Crocker, 2008). Estimates from national surveys of family organizations suggest that the number of family support specialists in the children’s mental health field has grown with a growing number of states developing certification standards (Hoagwood, et al., 2008; Walker & Crocker, 2008).

Issues of workforce capacity have been challenging children’s mental health services for some time (Annapolis Coalition on Behavioral Health Workforce, 2007; Schoenwald et al., 2010). Through grants from the NIMH, we developed and tested a training program for family support specialists to provide skills training in evidence-based engagement techniques, motivational interviewing, psychoeducation, problem solving and collaborative skills. The Parent Empowerment Program (PEP) is a basic training program that was developed collaboratively with researchers, practitioners, parents, family support specialists, and policy makers to teach specific evidence-based skills. The conceptual framework combines science and advocacy by integrating behavior change theory and principles of parent support (Jensen & Hoagwood, 2008; Olin, Hoagwood, Rodriguez, Ramos, et al., 2010). The impact of such training on improving collaborative skills and mental health services self-efficacy of family support specialists was demonstrated in a pilot (Olin, Hoagwood, Rodriguez, Ramos, et al., 2010). This PEP training now serves as NYS OMH’s early-entry training standard for family support specialists working in OMH licensed organizations.

Because active, engaged, questioning and empowered parents are more likely to seek out and continue with effective services (Fristad, Gavazzi, & Mackinaw-Koons, 2003; Fristad, Goldberg-Arnold, & Gavazzi, 2002), we have embarked on a series of efforts to improve parent activation through family support services delivered by family support specialists in New York. With a research grant from the National Institute of Mental Health (R01MH085969) titled “Improving Family to Family Services in Children’s Mental Health” we are examining family support services (particularly the role of family support specialists) in Home and Community Based Services Waiver (Waiver) programs. The goal is to understand the contexts within which family support specialist operate and to help organizations more effectively integrate family support services provided by family support specialists to improve the quality of care for children and families.

The study involves two phases. In Phase I, 21 Waiver programs across NYS successfully participated in the study, where we profiled the structure and organizational context of Waiver programs and examined the process and content of family support services within these programs. Our research team, in collaboration with NYS OMH and other key stakeholders, identified program-level and individual family support service-level quality indicators of best practices for family support specialists (Olin et al., 2013). These quality indicators are significantly correlated to organizational climate and culture (i.e., family support service work environments) (Kutash et al., In press; Olin et al., In press). Data from Phase I will inform Phase II of the study, specifically the adaptation of an organizational level intervention, called ARC (Availability, Responsiveness, and Continuity) (Glisson, Dukes, & Green, 2006; Glisson & Schoenwald, 2005).

In Phase II, a cluster-randomized design is being used to test the ARC intervention on Waiver program level outcomes. The ARC intervention is a research-based organizational intervention that will be used to identify and address barriers to effective integration of family support services in Waiver programs. Compared to Waiver programs in the control group, Waiver programs in the ARC group are hypothesized to show improvements in program staff report on Waiver program culture and climate, quality indicators of family support services, and team alliance with the family support specialists. The impact of ARC on family support services and caregiver/youth outcomes will also be examined. Compared to caregivers and youth served in the control group, those caregivers served in Waiver programs that received ARC are hypothesized to receive higher quality family support services, show increased parent activation, improved working alliance, reduced caregiver strain, and youth will show stable or positive change in symptoms and functioning as well as decreased out-of-home placements.

Lessons Learned for Scaling

Ultimately, parent activation aligns closely with the philosophy of family centered care and patient oriented outcomes in service systems. The current study will improve understanding of whether parent activation is linked to better quality of care, improved outcomes, and decreased costs. The parent empowerment training has been scaled up statewide, with support from the state and from local agencies. However state support is likely to decrease in the next few years, and it is not yet clear whether the benefits outweigh the costs. Without these data it is unclear whether the model will evolve or be sustained.

5. Quality improvement

The development of quality indicators to create consistent standards of practice and define measurable approaches for tracking change is directly related to the healthcare changes refashioning the mental health system. As noted earlier, new federal health policies are promoting the development of quality measures for children (Zima et al., 2013). The series of Institute of Medicine reports (Institute of Medicine, 2001; Institute of Medicine, 2006) and the ACA (P.L. 111-148) emphasize the integration of primary care, mental health care, and substance use treatment; and the development of indicators to hold providers accountable for quality of care they provide. Under the Children’s Health Insurance Program Reauthorization Act of 2009 (CHIP-RA) (P.L. 111-113), AHRQ funded seven national Centers of Excellence to develop quality indicators across all areas of children’s health. The National Collaborative for Innovation in Quality (NCINQ) is one of the Centers headed by the National Committee for Quality Assurance (NCQA), Nationwide Children’s Hospital, NYS OMH, and New York University (NYU). Our team at NYU was charged with developing quality indicators for adolescent depression and antipsychotic medication prescribing practices for children.

Developing quality indicators for use in states and healthcare plans is labor-intensive and time-consuming. Nevertheless each indicator, once approved and used, will provide leverage to hold providers accountable for delivering evidence-based care and improve quality of services. To develop the indicators for depression, for example, we conducted an extensive search of clinical practice guidelines to identify empirically based practices for adolescent depression management. The product of this synthesis was a care pathway that traced essential practices for the management of depression from case identification through symptom remission and 10 candidate quality indicators that captured each step of the practice (Lewandowski et al., In press).

Our field-testing of this care pathway involves manually reviewing de-identified charts in at least three large healthcare systems (Kaiser, Group Health, Bellevue) to refine data elements and determine whether the quality indicators can be derived from electronic health records. Such feasibility testing is a critical step in the development of usable tools for tracking care quality that can be integrated system wide.

The second quality indicator under development relates to antipsychotic prescribing practices. It is based on the NYS Psychiatric Services and Clinical Knowledge Enhancement System (PSYCKES). PSYCKES-Medicaid is a secure, HIPAA-compliant web-based tool that provides access to and summarizes Medicaid claims and encounter data to support clinical decision-making and improve psychotropic prescribing practices. PSYCKES was first made available to NYS OMH clinics in fall 2008 and included two sets of indicators for children and adults in the Medicaid mental health population: (1) psychotropic polypharmacy; and (2) use of antipsychotics with a high/moderate risk for metabolic disturbance among individuals with cardiometabolic conditions. A scientific advisory board nominated these indicators, which were selected based on stakeholder input. A new indicator set has been developed exclusively for children and it includes: (1) psychotropic polypharmacy; (2) high dose psychotropic prescriptions in children; and (3) psychotropics in children five years or younger (New York State Office of Mental Health, 2011)(New York State Office of Mental Health, 2011)(New York State Office of Mental Health, 2011)(New York State Office of Mental Health, 2011)(New York State Office of Mental Health, 2011). These indicators are being used in NYS to screen the Medicaid population to identify quality concerns; identify clinic and prescriber performance relative to state and regional comparators; present actionable lists of youth with quality flags for provider review; and track medication starts and regimen changes.

Lessons Learned for Scaling

It is too soon to know if the extensive efforts involved in developing and testing quality indicators will yield readily usable measures. The absence of EHRs within clinics that contain relevant behavioral health information makes the testing of these measures very challenging and perhaps premature (Berenson, Pronovost, & Krumholz, 2013; Pincus, Spaeth-Rublee, & Watkins, 2011). However, once MFS systems are more widely used in the mental health system and behavioral health indices are documented in pediatric health clinics, use of quality indicators will be possible. With the state moving rapidly into a managed care environment for Medicaid populations, the development of targeted quality indicators to track child behavioral health outcomes is a high priority.

NYS OMH Infrastructure: Clinic Technical Assistance Center(CTAC)

The majority of the NYS rollouts described above are now being supported by NYS OMH via CTAC, a statewide training and dissemination network co-directed by Hoagwood and McKay. It was established in 2011 to disseminate effective mental health practices and assist the entire cohort of 346 licensed clinics throughout the state. CTAC provides training, consultation and support of evidence-based engagement strategies; MFG therapy; and effective business practices via webinars, performance improvement networks, web-based tools, in person trainings, and learning collaboratives. The latter model is derived from the Institute for Healthcare Improvement Breakthrough Series (Institute for Healthcare Improvement, 2003).

CTAC learning collaboratives usually last six to 15 months and bring together 12 to 16 multidisciplinary teams. Specific topics are selected and each organization forms a QI team to implement the practice. Data pertaining to a measurable goal, such as attendance, dropout and treatment completion rates are collected during action periods to determine the effectiveness of the intervention upon particular outcomes. Successful interventions and approaches are shared at subsequent learning sessions for other organizations to adopt and study at their own agencies.

Currently, two-thirds of all NYS OMH licensed child mental health clinics in New York City have participated in one or more CTAC activities. Based on the first year of CTAC offerings, we have indexed levels of adoption and identified regional differences in innovation uptake by these 346 child-serving clinics. Clinics can be categorized according to five levels of adoption: none, low-, medium-, high-, and super-. A study examining characteristics associated with different levels of adoption is underway. Findings will help state policy-makers plan how to target EBP rollouts to maximize efficiency.

DISCUSSION

Implementation of innovative practices within state systems shares some common elements with healthcare systems implementation. Both require a multi-phasic (over time) and multi-layered (stratified) conceptual framework that reflects the dynamism of complex systems (Aarons et al., 2012; Proctor et al., 2009; Stetler, McQueen, Demakis, & Mittman, 2008). Both require active participation by multiple stakeholders, typically including policy-makers, administrators, practitioners, researchers, and families or consumers (Fixsen & Blase, 2009). As Mittman (2011) said, implementation science is not a spectator sport; audience participation is required. Both types of implementation depend on methods that include documentation and diagnosis of site-specific implementation gaps and assessment of site-specific barriers. Small, rapid (and inexpensive) pilot tests and formative evaluations tend to be used in both to surface basic problems quickly and address them practically. Study designs (sampling, power, data collection, analysis) also tend to be similar in state implementation and in healthcare systems and these include anticipating and accounting for heterogeneity of treatment effects and documenting implementation processes and mechanisms.

But state implementation research and practice has unique challenges. State systems are by nature political. Change is not only inevitable but is driven by the machinery of power. There is usually little room for scientific input. In contrast to healthcare systems, usually driven by a commitment to evidence, state systems vary in their respect for evidence. Conducting science within a political environment necessitates clear boundaries about where compromises can or cannot be made. It requires translation and back translation between those crafting policy and those pursuing data-driven answers. Tensions may arise when policy-makers need to make rapid policy decisions but research findings are not ready for wide implementation. Misunderstandings can arise if there is not recognition that some questions cannot be answered at all, most are not answered quickly, and continuous evaluation is needed to assess the real impact of implementation efforts.

However, the goal of implementation research is to generate an empirical base to guide efforts to fit evidence-based interventions within real-world service systems. Within state systems, this goal can achieve broad public health benefit. Consequently, a very positive aspect of state implementation research is that state systems set policies that affect the broad public; if these policies are guided by research, then the well-acknowledged time delay of 17 years between research and practice can be foreshortened and perhaps more than the 14% of research findings may be used to change practice.

The state-academic partnership in NYS has approached statewide implementation in two ways: first, by capitalizing on existing empirical evidence to guide dissemination and implementation of evidence-based practices; second, by using the “laboratory” of real world clinics and agencies to field pilot tests and formative evaluations prior to wide scale dissemination. This combined approach has the advantage of reducing costly missteps sometimes associated with premature dissemination. In NYS, the five system strategies (business practices, health information technologies, clinical EBTs, parent activation and quality indicator development) were deliberately selected to accomplish both goals. We have focused on these to simultaneously deploy empirically-based practices to as many public mental health practitioners as possible and build a knowledge base to contribute to implementation science.

Our efforts have been uneven, imbalanced, and by necessity asymmetrical. A data infrastructure from which to launch these studies did not exist prior to any of these rollouts. In fact, we are still in the process of building it. The data that do exist are largely inaccessible or unsuitable for the purposes of answering questions about implementation effectiveness (i.e., largely Medicaid claims data, which is light on service information beyond medications). There is no statewide dataset that captures child symptom and functioning outcomes nor that tracks children over time and between systems. The lack of a systematic data infrastructure has constrained our ability to examine whether any of these rollouts are making any differences in children’s outcomes. Despite the significant investment NYS has made to deliver EBPs, the lack of a concomitant data infrastructure for evaluating the impact of such an investment is a glaring gap in a state with a clear commitment to evidence-based practices. Not unlike other states, the focus on improving the workforce capacity to deliver evidence-based practices has not been accompanied by a focus on the impact on service quality or child outcomes.

However, this is changing. Nationally, the ACA’s emphasis on EBPs and accountability incentivizes the creation of tracking and reporting requirements among health-care providers (P.L. 111-148). In the NYS public mental health system, data monitoring and tracking via electronic systems is catalyzed by the need to make a business case for remaining solvent. The use of data to inform program practice, planning and policy has been limited (McLellan, Carise, & Kleber, 2003; Pincus, Spaeth-Rublee, & Watkins, 2011; Wisdom et al., 2006) but is now being demanded by state authorities. Efficient data management systems are critical to improving service processes and outcomes and to demonstrate and document quality measures (Ducharme, Knudsen, Roman, & Johnson, 2007; Wisdom et al., 2007).

To date, NYS implementation has focused on five rollouts targeting, for the most part, change processes for agencies and for providers. Aside from PSYCKES (that examines medication management practices) and the CFS, both of which examined child level outcomes, the NYS strategies have focused on broader organizational and provider issues including workforce training. Despite the existence of state databases that could potentially be exploited to understand the impact of these broad implementation strategies, only recently has there been state interest in examining these data.

It is clear given the emphasis of ACA that EHRs will be assumed to be in place in all health and mental health systems over the next decade. The current investment by states in EHRs represents another opportunity to develop an efficient and usable infrastructure to guide policies and practices in healthcare. In NYS, our pilot to implement such systems (e.g., the CFS) has been challenging, even with training and assistance in initial implementation. Clinics with which we work vary tremendously in their sophistication with the use of EHRs and other health-related systems. Similar to what we have learned in implementing EBTs, the organizational barriers to adopting and sustaining electronic data systems are as onerous as those involved in installing new clinical practices. The strategies for overcoming these barriers are also similar: on-site and continuous consultation; innovation fit and compatibility, identification of an on-site champion or master user; and support by leadership for use of the system.

Our experience providing training and consultation for specific clinical treatments targeting specific conditions (trauma, depression, DBDs) has been shaped by the need for practical, ongoing distance learning consultation to augment face-to-face training. With respect to EBP implementation, NYS has taken an active stance by funding the EBTDC since 2004 and the more recent CTAC. Each of the EBPs for which training and intensive consultation is provided now, by necessity, incorporates fiscal implications—how to bill for these services, modify staffing patterns, change supervisory structures, etc. There is recognition that in order for children’s mental health services to be delivered with quality and in a fiscally viable manner, consideration of all of these factors is essential.

The focus on parent activation is aligned with the Institute of Medicine (2006) recommendation that shared decision-making is a key component of healthcare redesign. In mental health, shared decision-making is also one of ten components of the National Consensus Statement on Mental Health Recovery, as defined under empowerment (Substance Abuse and Mental Health Services Administration, 2006). Shared decision-making has been advocated in reviews (Leckman & King, 2007) and in guidelines (National Institute for Health and Clinical Excellence, 2005) related to the use of antidepressant medications in children and adolescents. Despite its promise in the general health field, professional, public, and legal and political forces have countered the movement towards shared decision-making in the mental health field for several reasons (Adams & Drake, 2006; Deegan & Drake, 2006). These include concerns about the capacity of persons with mental illnesses to make informed decisions, discomfort among some providers with the shift in roles required by shared decision-making, unease among some consumers regarding their ability to take on the responsibilities offered through shared decision-making, and lingering public fear and prejudice around persons with mental illnesses (Adams & Drake, 2006; Substance Abuse and Mental Health Services Administration, 2010). Such concerns may be amplified within children’s mental health, where professionals have historically viewed parents as obstacles to treatment, irrelevant to the treatment process, or causes of problems (Adams et al., 2000). Yet activated, empowered parents are more likely to engage and adhere to treatments for their children. The focus on parent activation is a step towards improving quality of care as outlined within healthcare redesign.

The NYS focus on development of quality indicators is similarly aligned with the general health care movement towards accountable care as exemplified by the ACA (P.L. 111-148). Large administrative datasets such as those available in states are valuable resources with which to examine the utility of proposed quality indicators in children’s mental health.

Conclusion

The dynamic political landscape within state systems is a challenging yet rich laboratory for promoting and testing innovations in healthcare. Steven Johnson in the History of Innovation (2010) identified principles that he had traced through engineering, geology, science, and technology over 500 years and found to underlie innovation. Among these were availability of “liquid networks” (i.e., loosely formed and multidisciplinary teams that work on the edges of new ideas and by virtue of their proximity and specialization generate new ideas). We have found that the networked partnership of policy-makers, researchers, agency directors, front-line providers, and family support specialists creates a fertile environment for generating new ideas. Another principle of innovation identified by Johnson was learning from errors: failing faster. This is a particular favorite in our work. As we have documented above, there have been many set-backs and consequently lessons learned. In fact, we and others involved in large-scale dissemination (Chamberlain, Bickman, Chaffin, Mittman) have suggested a set of “implementation salvage strategies” guided by the Aarons, Hurlburt & Horwitz (2011) model to avert or at least learn from failures (Hoagwood, Chaffin, Chamberlain, Bickman, & Mittman, 2011).

A third process identified by Johnson is the use of data platforms including publicly available state data for generating new solutions to public health problems. Our experience in state-academic partnerships has exemplified the utility of many of these principles. We believe that state implementation research can take advantage of its ethical responsibility to benefit the public by facilitating access to the large datasets generated with public dollars and incentivize their widespread use for solving public problems.

The fledging field of implementation science in state systems is shaped irrevocably by the changes that are occurring as the healthcare system assumes greater responsibility for mental health. These changes require states and providers to pay closer attention to data and evidence. This has been called the third paradigm shift (Gray, 2013), resulting in greater emphasis on population-based and personalized medicine. This shift, however, necessitates research methods, measures, designs, and strategies that are flexible and adaptable; and that focus on outcomes, accountability, and evidence. We believe that the growing use of data to inform decision-making and the drive to expand healthcare quality improvement will refashion the services research agenda to focus less on trials of narrow clinical interventions and more on designing, enacting, and sustaining large system changes. States are a fertile laboratory for such work.

Acknowledgments

The authors would like to acknowledge the National Institute of Mental Health (NIMH) grant P30MH090322-02, the Advanced Center for State Research to Scale up EBPs for Children.

References

  1. Aarons GA, Glisson C, Green P, Hoagwood KE, Kelleher K, Landsverk J The Research Network on Youth Mental Health. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implementation Science. 2012;7(1):56. doi: 10.1186/1748-5908-7-56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Adams J, Biss C, Mohammed VB, Meyers J, Slaton E Peer Technical Assistance Network. Learning from Colleagues: Family/Professional Partnerships. Washington, DC: Georgetown University; 2000. [Google Scholar]
  4. Adams J, Drake RE. Shared Decision-Making and Evidence-Based Practice. Community Mental Health Journal. 2006;42(1):87–105. doi: 10.1007/s10597-005-9005-8. [DOI] [PubMed] [Google Scholar]
  5. Agency for Healthcare Research and Quality. Mental disorders among most costly conditions in children. AHRQ News and Numbers. 2009 Apr 22; Retrieved January 17, 2013, from http://www.ahrq.gov/news/nn/nn042209.htm.
  6. Annapolis Coalition on Behavioral Health Workforce. Action plan on behavioral health workforce develoment. Washington, DC: Substance Abuse and Mental Health Services Administration; 2007. [Google Scholar]
  7. Aos S, Mayfield J, Miller M, Yen W. Washington State Evidence-Based Treatment of Alcohol, Drug, and Mental Health Disorders: Potential Benefits, Costs and Fiscal Impacts on Washington State. Washington: Washington State Institute for Public Policy; 2006. (Document ID: 06-06-3901) [Google Scholar]
  8. Berenson RA, Pronovost PJ, Krumholz HM. Timely Analysis of Immediate Health Policy Issues 6. 2013. Achieving the Potential of Health Care Performance Measures. [Google Scholar]
  9. Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry. 2008;47(10):1114. doi: 10.1097/CHI.0b013e3181825af8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Bickman L, Kelley SD, Breda C, de Andrade AR, Riemer M. Effects of Routine Feedback to Clinicians on Mental Health Outcomes of Youths: Results of a Randomized Trial. Psychiatric Services. 2011;62(12):1423–1429. doi: 10.1176/appi.ps.002052011. [DOI] [PubMed] [Google Scholar]
  11. Bruns EJ, Hoagwood KE, Hamilton JD. State Implementation of Evidence-Based Practice for Youths, Part I: Responses to the State of the Evidence. Journal of the American Academy of Child & Adolescent Psychiatry. 2008;47(4):369–373. doi: 10.1097/CHI.0b013e31816485f410. [DOI] [PubMed] [Google Scholar]
  12. Burns BJ, Hoagwood KE. Preface. Evidence-Based Practice, Part I: Research Update. Child and Adolescent Psychiatric Clinics of North America. 2004;13(4):xi–xiii. doi: 10.1016/S1056-4993(04)00063-X. [DOI] [PubMed] [Google Scholar]
  13. CATS Consortium. Implementing CBT for traumatized children and adolescents after September 11: Lessons learned from the Child and Adolescent Trauma Treatments and Services (CATS) Project. Journal of Clinical Child and Adolescent Psychology. 2007;36(4):581–592. doi: 10.1080/15374410701662725. [DOI] [PubMed] [Google Scholar]
  14. Child Health Insurance Program Reauthorization Act (CHIP-RA), P.L. 111-3 C.F.R. (2009).
  15. Chorpita BF, Daleiden EL. Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology. 2009;77(3):566. doi: 10.1037/a0014565. [DOI] [PubMed] [Google Scholar]
  16. Cohen JA, Mannarino AP, Deblinger E. Treating trauma and traumatic grief in children and adolescents. Guilford Press; 2006. [Google Scholar]
  17. Deegan PE, Drake RE. Shared decision making and medication management in the recovery process. Psychiatric Services. 2006;57(11):1636–1639. doi: 10.1176/ps.2006.57.11.1636. [DOI] [PubMed] [Google Scholar]
  18. Ducharme LJ, Knudsen HK, Roman PM, Johnson JA. Innovation adoption in substance abuse treatment: Exposure, trialability, and the Clinical Trials Network. Journal of Substance Abuse Treatment. 2007;32(4):321. doi: 10.1016/j.jsat.2006.05.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Essock S, Covell N, Leckman-Westin E, Lieberman J, Sederer L, Kealey E, Finnerty M. Identifying Clinically Questionable Psychotropic Prescribing Practices for Medicaid Recipients in New York State. Psychiatric Services. 2009;60(12):1595–1602. doi: 10.1176/ps.2009.60.12.1595. [DOI] [PubMed] [Google Scholar]
  20. Fixsen DL, Blase KA. NIRN implementation brief. 2009. Implementation: The missing link between research and practice; p. 1. [Google Scholar]
  21. Fixsen DL, Blase KA, Metz A, Van Dyke M. Statewide implementation of evidence-based programs. Exceptional Children. 2013;79(2):213–230. [Google Scholar]
  22. Fixsen DL, Naoom S, Blase KA, Friedman R, Wallace F. Implementation research: A synthesis of the literature (Louis de la Parte Florida Mental Health Institute Publication# 231) Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005. [Google Scholar]
  23. Fristad MA, Gavazzi SM, Mackinaw-Koons B. Family psychoeducation: an adjunctive intervention for children with bipolar disorder. Biological Psychiatry. 2003;53(11):1000–1008. doi: 10.1016/S0006-3223(03)00186-0. [DOI] [PubMed] [Google Scholar]
  24. Fristad MA, Goldberg-Arnold JS, Gavazzi SM. Multifamily psychoeducation groups (MFPG) for families of children with bipolar disorder. Bipolar Disorders. 2002;4(4):254–262. doi: 10.1034/j.1399-5618.2002.09073.x. [DOI] [PubMed] [Google Scholar]
  25. Ganju V. Implementation of Evidence-Based Practices in State Mental Health Systems: Implications for Research and Effectiveness Studies. Schizophrenia Bulletin. 2003;29(1):125–131. doi: 10.1093/oxfordjournals.schbul.a006982. [DOI] [PubMed] [Google Scholar]
  26. Garland AF, Hawley KM, Brookman-Frazee L, Hurlburt M. Identifying Common Elements of Evidence-Based Psychosocial Treatments for Children’s Disruptive Behavior Problems. Journal of the American Academy of Child & Adolescent Psychiatry. 2008;47(5):505–514. doi: 10.1097/CHI.0b013e31816765c2. [DOI] [PubMed] [Google Scholar]
  27. Gleacher AA, Nadeem E, Moy AJ, Whited AL, Albano AM, Radigan M, Hoagwood KE. Statewide CBT training for clinicians and supervisors treating youth: The New York State Evidence based treatment dissemination center. Journal of Emotional and Behavioral Disorders. 2011;19(3):182–192. doi: 10.1177/1063426610367793. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Glisson C, Dukes D, Green P. The effects of the ARC organizational intervention on caseworker turnover, climate and culture in children’s service systems. Child Abuse and Neglect. 2006;30:855–880. doi: 10.1016/j.chiabu.2005.12.010. [DOI] [PubMed] [Google Scholar]
  29. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research. 2005;7(4):243–259. doi: 10.1007/s11020-005-7456-1. [DOI] [PubMed] [Google Scholar]
  30. Gray JAM. The shift to personalised and population medicine. The Lancet. 2013;382(9888):200–201. doi: 10.1016/s0140-6736(13)61590-1. [DOI] [PubMed] [Google Scholar]
  31. Green LW. From research to best practices in other settings and populations. American journal of health behavior. 2001;25(3):165–178. doi: 10.5993/ajhb.25.3.2. [DOI] [PubMed] [Google Scholar]
  32. Hoagwood KE, Chaffin M, Chamberlain P, Bickman L, Mittman B. Maximizing methodolgical flexibility in children’s mental health research. Paper presented at the 4th Annual NIH conference on the Science of Dissemination and Implementation: Policy and Practice.2011. [Google Scholar]
  33. Hoagwood KE, Green E, Kelleher K, Schoenwald SK, Rolls-Reutz J, Landsverk J, Mayberg S. Family advocacy, support and education in children’s mental health: Results of a national survey. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1):73–83. doi: 10.1007/s10488-007-0149-4. [DOI] [PubMed] [Google Scholar]
  34. Hoagwood KE, Vogel JM, Levitt JM, D’Amico PJ, Paisner WI, Kaplan SJ. The Child and Adolescent Trauma Treatments and Services Consortium. Implementing an evidence-based trauma treatment in a state system after September 11: The CATS project. Journal of the American Academy of Child & Adolescent Psychiatry. 2007;46(6):773–779. doi: 10.1097/chi.0b013e3180413def. [DOI] [PubMed] [Google Scholar]
  35. Institute for Healthcare Improvement. IHI Innovation Series white paper. Boston, MA: 2003. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. [Google Scholar]
  36. Institute of Medicine, Committee on Quality of Health Care in America. . Crossing the quality chasm: A new health system for the 21st century. National Academies Press; 2001. [Google Scholar]
  37. Institute of Medicine, Committee on Quality of Health Care in America. Improving the Quality of Health Care for Mental and Substance-Use Conditions: Quality Chasm Series. Washington, DC: National Academies Press; 2006. [PubMed] [Google Scholar]
  38. Institute of Medicine, Committee on Science, Engineering, and Public Policy, National Academy of Sciences, National Academy of Engineering. On Being a Scientist: A Guide to Responsible Conduct in Research: Third Edition. 3. Washington, DC: National Academies Press; 2009. [PubMed] [Google Scholar]
  39. Jensen PS, Hoagwood KE. Improving children’s mental health through parent empowerment: A guide to assisting families. Oxford University Press; 2008. [Google Scholar]
  40. Johnson S. Where Good Ideas Come From: The Natural History of Innovation. New York: Penguin Group, Inc; 2010. [Google Scholar]
  41. Kazdin AE, Weisz JR. Evidenced-based Psychotherapy For Children And Adolescents. New York: Guilford Press; 2003. [Google Scholar]
  42. Kelleher K. Organizational Capacity to Deliver Effective Treatments for Children and Adolescents. Administration and Policy in Mental Health and Mental Health Services Research. 2010;37(1–2):89–94. doi: 10.1007/s10488-010-0284-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Kohn LT, Corrigan J, Donaldson MS. To err is human: building a safer health system. Vol. 6. Joseph Henry Press; 2000. [PubMed] [Google Scholar]
  44. Kutash K, Acri MC, Pollock M, Armusewicz K, Olin SS, Hoagwood KE. Administration and Policy in Mental Health and Mental Health Services Research. Quality indicators for multidisciplinary team functioning in community-based children’s mental health services. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Leckman J, King R. A developmental perspective on the controversy surrounding the use of SSRIs to treat pediatric depression. American Journal of Psychiatry. 2007;164(9):1304–1306. doi: 10.1176/appi.ajp.2007.07050880. [DOI] [PubMed] [Google Scholar]
  46. Lewandowski RE, Acri MC, Hoagwood KE, Olfson M, Clarke G, Gardner W, Horwitz SM. An evidence-based care pathway with quality indicators for the management of adolescent depression. Pediatrics. doi: 10.1542/peds.2013-0600. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Lloyd DR. How to maximize service capacity: Nuts and bolts solutions for implementing change in behavioral healthcare organizations below the senior management level. Fayetteville, N.C: National Council for Community Behavioral Healthcare; 1998. [Google Scholar]
  48. Lochman JE, Wells KC. The coping power program for preadolescent aggressive boys and their parents: outcome effects at the 1-year follow-up. Journal of consulting and clinical psychology. 2004;72(4):571. doi: 10.1037/0022-006X.72.4.571. [DOI] [PubMed] [Google Scholar]
  49. McCellan J, Werry J. Evidence-based treatments in child and adolescent psychiatry: an inventory. Journal of the American Academy of Child & Adolescent Psychiatry. 2003;42(12):1388–1400. doi: 10.1097/01.chi.0000092322.84052.88. [DOI] [PubMed] [Google Scholar]
  50. McHugh RK, Barlow DH. The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. American Psychologist. 2010;65(2):73. doi: 10.1037/a0018121. [DOI] [PubMed] [Google Scholar]
  51. McKay MM, Gopalan G, Franco L, Dean-Assael K, Chacko A, Jackson JM, Fuss A. A Collaboratively Designed Child Mental Health Service Model Multiple Family Groups for Urban Children with Conduct Difficulties. Research on social work practice. 2011;21(6):664–674. doi: 10.1177/1049731511406740. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. McKay MM, Nudelman R, McCadam K, Gonzales J. Evaluating a social work engagement approach to involving inner-city children and their families in mental health care. Research on Social Work Practice. 1996;6(4):462–472. [Google Scholar]
  53. McKay MM, Stoewe J, McCadam K, Gonzales J. Increasing access to child mental health services for urban children and their caregivers. Health & Social Work. 1998;23(1):9–15. doi: 10.1093/hsw/23.1.9. [DOI] [PubMed] [Google Scholar]
  54. McLellan AT, Carise D, Kleber HD. Can the national addiction treatment infrastructure support the public’s demand for quality care. Journal of Substance Abuse Treatment. 2003;25(2):117–121. [PubMed] [Google Scholar]
  55. Miller JE. The Changing Landscape of Behavioral Health and Going into the Future. Paper presented at the National Association of State Mental Health Program Directors 2011 Annual Commissioners Meeting.2011. [Google Scholar]
  56. Mittman B. Implementation Salvage Strategies: Maximixing Methodological Flexibility in Children’s Mental Health Research. Paper presented at the NIH Conference on the Science of Dissemination and Implmentation: Policy and Practice; Rockville, MD. 2011. [Google Scholar]
  57. Nadeem E, Gleacher AA, Pimentel S, Hill L, McHugh M, Hoagwood KE. Administration and Policy in Mental Health and Mental Health Services Research. The Role of Consultation Calls for Clinic Supervisors in Supporting Large-Scale Dissemination of Evidence-Based Treatments for Children. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. National Institute for Health and Clinical Excellence (NICE) Depression in children and young people: Identification and management in primary, community and secondary care. Leicester, UK: 2005. [PubMed] [Google Scholar]
  59. New York State Office of Mental Health. Statewide Comprehensive Plan 2009-2013. New York City: New York State Office of Mental Health; 2009. (Vol. Appendix 2) [Google Scholar]
  60. New York State Office of Mental Health. Psychiatric Clinical Knowledge Enhancement System (PSYCKES) New York, NY: New York State Office of Mental Health; 2011. Quality Concerns in Psychotropic Prescribing: Reducing Psychotropic Prescribing Risk for Youth. Reference Guide. [Google Scholar]
  61. North MS, Gleacher AA, Radigan M, Greene L, Levitt JM, Chassman J, Hoagwood KE. Evidence-Based Treatment Dissemination Center (EBTDC): Bridging the research–practice gap in New York State. Report on Emotional & Behavioral Disorders in Youth. 2008;8(1):1–5. [Google Scholar]
  62. Olin SS, Hoagwood KE, Rodriguez J, Radigan M, Burton G, Cavaleri M, Jensen PS. Impact of empowerment training on the professional work of family peer advocates. Children and youth services review. 2010;32(10):1426–1429. doi: 10.1016/j.childyouth.2010.06.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Olin SS, Hoagwood KE, Rodriguez J, Ramos B, Burton G, Penn M, Jensen PS. The application of behavior change theory to family-based services: Improving parent empowerment in children’s mental health. Journal of child and family studies. 2010;19(4):462–470. doi: 10.1007/s10826-009-9317-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Olin SS, Kutash K, Pollock M, Burns BJ, Kuppinger A, Craig N, Hoagwood KE. Administration and Policy in Mental Health and Mental Health Services Research. 2013. Developing quality indicators for family support services in community team-based mental health care; pp. 1–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Olin SS, Williams NJ, Pollock M, Armusewicz K, Kutash K, Glisson C, Hoagwood KE. Administration and Policy in Mental Health and Mental Health Services Research. Quality Indicators for Family Support Services and Their Relationship to Organizaional Social Context. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Patient Protection and Affordable Care Act (ACA), P.L. 111-148 C.F.R. (2010).
  67. Paul Wellstone and Pete Domenici Mental Health Equity and Addictions Parity Act, P.L. 110-343 C.F.R. (2008).
  68. Pimentel S, Hoagwood K, Albano AM, Regan J. Disseminating evidence-based treatments for children: A microanalysis of consultation calls as an ongoing training strategy. In K. Hoagwood (Chair) Beyond Didactics: Emerging evidence on EBP implementation strategies in New York State. Paper presented at the Research and Training Center for Children’s Mental Health Conference; Tampa, Florida. 2009. [Google Scholar]
  69. Pincus HA, Spaeth-Rublee B, Watkins KE. The case for measuring quality in mental health and substance abuse care. Health Affairs. 2011;30(4):730–736. doi: 10.1377/hlthaff.2011.0268. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Preethy G, Blase K, Kanary PJ, Wotring J, Bernstein D, Carter WJ. A Policy Brief for Financing Evidence-Based Programs and Practices. Utah Technical Assistance Center for Children’s Services; 2008. [Google Scholar]
  71. Proctor EK, Landsverk J, Aarons GA, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research. 2009;36(1):24–34. doi: 10.1007/s10488-008-0197-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  72. Rodriguez J, Olin SS, Hoagwood KE, Shen S, Burton G, Radigan M, Jensen PS. The development and evaluation of a parent empowerment program for family peer advocates. Journal of Child and Family Studies. 2011;20(4):397–405. doi: 10.1007/s10826-010-9405-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, Rolls-Reutz J. A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs) Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1–2):84–97. doi: 10.1007/s10488-007-0147-6. [DOI] [PubMed] [Google Scholar]
  74. Schoenwald SK, Hoagwood KE. Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatric Services. 2001;52(9):1190–1197. doi: 10.1176/appi.ps.52.9.1190. [DOI] [PubMed] [Google Scholar]
  75. Schoenwald SK, Hoagwood KE, Atkins MS, Evans ME, Ringeisen H. Workforce development and the organization of work: The science we need. Administration and Policy in Mental Health and Mental Health Services Research. 2010;37(1):71–80. doi: 10.1007/s10488-010-0278-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  76. Silverman WK, Hinshaw SP. The second special issue on evidence-based psychosocial treatments for children and adolescents: A 10-year update. Journal of Clinical Child & Adolescent Psychology. 2008;37(1):1–7. doi: 10.1080/15374410701818293. [DOI] [PubMed] [Google Scholar]
  77. Stark KD, Curry J, Goldman E. Therapist’s Manual for treating depressive disorders in children and adolescents. New York, NY: Columbia University; 2006. [Google Scholar]
  78. Stetler C, McQueen L, Demakis J, Mittman B. An organizational framework and strategic implementation for system-level change to enhance research-based practice: QUERI Series. Implementation Science. 2008;3(1):30. doi: 10.1186/1748-5908-3-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  79. Substance Abuse and Mental Health Services Administration. SAMHSA’s National Registry of Evidence-based Programs and Practices. Retrieved 12 December 2012 http://www.nrepp.samhsa.gov/
  80. Substance Abuse and Mental Health Services Administration. National Consensus Statement on mental health recovery. Rockville, MD: Substance Abuse and Mental Health Services Administration, Center for Mental Health Services; 2006. [Google Scholar]
  81. Substance Abuse and Mental Health Services Administration. Shared decision-making in mental health care: Practice, research, and future directions. Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration; 2010. [Google Scholar]
  82. Torrey WC, Drake RE, Dixon L, Burns BJ, Flynn L, Rush AJ, Klatzker D. Implementing evidence-based practices for persons with severe mental illnesses. Psychiatric services. 2001;52(1):45–50. doi: 10.1176/appi.ps.52.1.45. [DOI] [PubMed] [Google Scholar]
  83. Walker DK, Crocker RW. Measuring family systems outcomes. New York: Aldine de Gruyter; 2008. [Google Scholar]
  84. Weisz JR, Weiss B, Han SS, Granger DA, Morton T. Effects of psychotherapy with children and adolescents revisited: A meta-analysis of treatment outcome studies. Psychological Bulletin. 1995;117(3):450–468. doi: 10.1037/0033-2909.117.3.450. [DOI] [PubMed] [Google Scholar]
  85. Wisdom JP, Chor KHB, Hoagwood KE, Horwitz SM. Administration and Policy in Mental Health and Mental Health Services Research. 2013. Innovation Adoption: A Review of Theories and Constructs; pp. 1–23. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
  86. Wisdom JP, Ford JH, II, Hayes RA, Edmundson E, Hoffman K, McCarty D. Addiction treatment agencies’ use of data: A qualitative assessment. The Journal of Behavioral Health Services and Research. 2006;33(4):394–407. doi: 10.1007/s11414-006-9039-x. [DOI] [PubMed] [Google Scholar]
  87. Wisdom JP, White N, Goldsmith K, Bielavitz S, Rees A, Davis C. Implementation of accessible information technology in Northwest K-12 Schools: The need for a systems perspective. Educational Technology & Society. 2007;10(3):222–232. [Google Scholar]
  88. Zima BT, Murphy JM, Scholle SH, Hoagwood KE, Sachdeva RC, Mangione-Smith R, Jellinek M. National quality measures for child mental health care: background, progress, and next steps. Pediatrics. 2013;131(Supplement 1):S38–S49. doi: 10.1542/peds.2012-1427e. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES