Learning Collaboratives in Mental Health Care: Used but Untested

Abstract

Objective

Policymakers have increasingly turned to Learning Collaboratives (LCs) as a strategy for improving usual care through the dissemination of evidence-based practices. The purpose of this review is to characterize the state of the evidence on LCs in mental health care.

Methods

A systematic search of major academic databases for peer-reviewed articles on LCs in mental health care generated 421 unique articles across a range of disciplines; 28 mental health articles were selected for full-text review, and 20 articles comprising 16 distinct studies met criteria for final inclusion. Articles were coded to identify the LC components reported, the focus of the research, and key findings.

Results

The majority of the articles included baseline to post-collaborative assessments of provider- or patient-level variables; there was only one study with a comparison condition. The LC targets ranged tremendously, from the use of a depression screener to implementation of evidence-based treatments. Fourteen crosscutting LC components (e.g., in-person learning sessions, phone meetings, data reporting, leadership involvement, training in QI methods) were identified from a systematic review of the extant literature on LCs. The LCs in this review reported including, on average, 7 components, most commonly in-person learning sessions, Plan-Do-Study-Act (PDSA) cycles, multidisciplinary QI teams, and data collection for QI.

Conclusions

LCs are being used widely in mental health care with minimal evidence of their effectiveness and unclear reporting on specific components. There is a great need for rigorous observational and controlled research studies on the impact of LCs on targeted provider- and patient-level outcomes.

Recently, a tremendous emphasis has been placed on the integration of evidence-based practices into routine mental health care. Substantial budget cuts to mental health funding at the state and national level have forced policymakers to seek out efficient and effective ways to scale up training in evidence-based practices (1). States, counties, and national organizations have turned to Learning Collaboratives (LCs) as a method for large-scale training with ongoing support. This collaborative approach has clearly become a priority in the field. The Substance Abuse and Mental Health Services Administration (SAMHSA) recently issued a call for applications for State Adolescent Treatment Enhancement and Dissemination grants totaling $30 million over three years to help states develop “learning laboratories” focused on shared provider experiences during the implementation of new evidence-based practices (2). Similarly, through the National Council for Community Behavioral Healthcare (NCCBH), 35 states are now using LCs to change healthcare provider practices (A. Salerno, personal communication, July 1, 2012). LCs represent a significant investment in the field as a potentially viable approach to large-scale implementation and dissemination of new treatment practices. However, there has been little research on the effectiveness of LCs for mental health evidence-based practices.

Learning Collaboratives as they are implemented in mental health are adapted from Quality Improvement Collaborative (QIC) models used in healthcare. One of the most widely cited and adapted QIC models is the Institute for Healthcare Improvement’s Breakthough Series (BTS) Collaboratives (3–9). The QI processes at the core of the IHI and other approaches are rooted in industrial improvement practices and the work of W. Edwards Deming and Joseph Juran, statisticians who advocated for process improvement driven by both ongoing data collection and analysis, and an assumption of workers’ interest in learning and improvement (10–12).

While there is some evidence for the effectiveness of QICs in healthcare, there remains a need for rigorous research in this area. A systematic literature review by Schouten and colleagues (13) identified nine controlled studies of healthcare QICs and concluded that the QICs showed promise in changing provider practices. However, there was less evidence in support of an impact on patient-level outcomes (13). Although the review included two randomized controlled trials, the majority of the studies included use matched-control sites or compared administrative data from similar sites in a larger provider network. Building on these findings, a more recent review included 24 articles, with the goal of updating the original literature review and developing a deeper understanding of the core components of QICs as they are reported in the literature (14). This review included additional RCTs (five distinct studies); however, as with earlier reviews (13), the vast majority of studies used matched-controls. Of the 14 crosscutting components identified as common ingredients in QICs, the collaboratives reported including, on average, 6 to 7 components (most commonly, in-person learning sessions, Plan-Do-Study-Act (PDSA) cycles, multidisciplinary QI teams, and data collection for QI). Similar to the earlier review (13), outcome data suggested that the greatest impact of the collaboratives was on provider-level process-of-care variables; patient-level findings were less robust. Due to the imprecise reporting on specific components of the collaborative, it was not possible to link any specific components of a collaborative with improved care.

Of note, neither of these systematic reviews included collaboratives focused on mental health issues because when they were undertaken there had been no controlled studies targeting mental health care. The application of LCs in mental health has been to a wide range of practices—including process of care (e.g., engagement in services, care integration, use of a screener) (15–18) and implementation of complex evidence-based practice (7, 19). The focus on evidence-based practices is notable given the complexity of the patient outcomes and the substantial skill development required of providers. The current systematic literature review focuses on peer-reviewed studies of mental health LCs that include any patient or provider pre-to-post outcome data. Given differences between mental health and general healthcare settings in terms of their structure, types of interventions and patient issues addressed, and data systems available, there is a critical need for a better understanding of how LCs are implemented in mental health. The primary goal of this review is to identify the components of LCs as reported in mental health studies, and characterize the existing data on collaboratives (e.g., patient-level data, reports of changed provider practices, analyses of feasibility and/or acceptability in real-world care settings).

Methods

This literature search on Learning Collaboratives focused on individual empirical articles published within the date range January 1995- October 2013. The database search included Ovid MEDLINE, ProQuest, PsycInfo, and PubMed. Search terms included “learning collaborative”, “quality improvement collaborative”. “Breakthrough series”, or “NIATx.” These terms were refined after several preliminary searches, and are similar to those used in earlier reviews (13, 14). The term “NIATx” was included in order to capture the NIATX process improvement approach used in the substance abuse literature, which draws on similar conceptual models to the predominant approach to collaboratives, specifically the IHI’s Breakthrough Series (20).

Articles that met inclusion criteria were peer-reviewed, written in English, and included a pre- and post-intervention comparison of the impact of an LC. In order to define LCs in mental health, we searched the theoretical literature on Quality Improvement Collaboratives (3–5, 9, 21–24) and reviewed the definition used by Schouten et al. (13); subsequently, the authors conducted informational interviews with a subset of LC purveyors to elicit more detail. This study defines LCs as organized, structured group learning initiatives in which organizers took the following steps: convened multidisciplinary teams representative of different levels of the organization; focused on improving specific provider practices or patient outcomes; included training from experts in a particular practice and/or the quality improvement methods; included a model for improvement with measurable targets, data collection, and feedback; engaged multidisciplinary teams in active improvement processes wherein they implemented “small tests of change” or engaged in PDSA activities; and employed structured activities and opportunities for learning and cross-site communication (e.g., in-person learning sessions, phone calls, email listservs) (3, 5–7, 9, 25, 26). We asssess the ways in which the 14 components identified by Author and colleagues (14), including in-person learning sessions, phone meetings, data reporting, feedback, training in QI methods, use of process improvement methods, were reported in these studies.

Two of the study’s authors reviewed all abstracts generated by the initial search to select articles that merited a full-text review. The same two independent coders reviewed each individual article retrieved to determine if the article met final inclusion criteria. In the event of a discrepancy, or if inclusion was unclear, coders conferred with members of the research team to make a final determination. Once article selection was finalized, each article was coded using a standardized table to summarize study details (e.g., targets for improvement, study design, setting, study sample, and LC components). A primary coder was assigned to each article, and a secondary coder reviewed the primary coder’s work. Disagreements were resolved by consensus.

The initial search generated 421 unique articles across several disciplines (primarily mental health, education, and healthcare). From a review of those 421 abstracts, 52 were determined to be mental health or substance abuse-related articles, 28 of which met criteria for full-text article review (i.e., they appeared to be focused on learning collaboratives). Articles were excluded after the full-text review if they did not report any pre- to post-collaborative quantitative data. Following a review of those articles and their references, 20 articles were selected for final inclusion (see Figure 1 in online appendix) (15–18, 27–40).

Results

The 20 articles selected for inclusion encompass 16 distinct studies. Table 1 provides a summary of the study type, LC model, and LC components reported in each study. Table 2 provides definitions of the study characteristics and LC components tracked in this review. The LC features were categorized into components, QI processes, and organizational involvement. LC components refer to LC features that comprised the structure of the model. QI processes include available details about PDSAs and other QI activities. The organizational involvement section included indicators of the ways in which the LC penetrated different levels of the organization.

Table 1.

LC Components by Studya^a

AIb	AI	AI	AI	LCC	LCC	LCC	LCC	LCC	LCC	LCC	LCC	QI	QI	QI	OP	OP	OP
Articlec	Target for Improvement	Model	Study Sample	Length	PW: EP	PW: ODC	In-Person Learning Sessions	PDSAs	MD QITs	QIT Phone Calls	Email or Web Support	New QI Data	QI Data Review	External Support with Data Review	Involved Leaders	External Training for Non- QIT Staff	QIT Training for Non- QIT Staff
Cavaleri et al., 2006 (15)d	Mental health service use & evidence-based engagement strategies	BTS	12 mental health agencies	9 months	Un- clear	Yes	Yes	Un- clear	Yes	Yes	No	Un- clear	Un- clear	No	Yes	Unclear	No
Cavaleri et al., 2007 (27)d	Mental health service use & evidence-based engagement strategies	BTS	9 mental health agencies: 9 of 12 mental health agencies from Cavaleri et al., 2006	9 months	Un- clear	Yes	Yes	Un- clear	Yes	Yes	No	Un- clear	Un- clear	No	Yes	Unclear	No
Cavaleri et al., 2010 (16)	Mental health service use & evidence-based engagement strategies	BTS	5 mental health agencies: 4 experimental, 1 did not implement any engagement strategies	9 months	Un- clear	Yes	Yes	Un- clear	Yes	Yes	No	Un- clear	Un- clear	No	Yes	No	Yes
Duffy et al., 2008 (32)	Use of a Depression assessment in psychiatric practices	BTS, CCM	19 psychiatric practices (2 practices dropped out without completing data collection)	12 months	No	Yes	Yes	Yes	Yes	Yes	No	Yes	No	No	Unclear	No	Yes
Ebert et al., 2011 (7)	Use of Trauma- Focused Cognitive Behavioral Therapy (TF-CBT) in community practice settings	BTS	11 mental health agencies (2 agencies did not complete)	18 months	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	No	Yes	No	No
Epstein et al., 2008 (28)e	Adherence to guidelines for evidence-based assessment and treatment of attention deficit hyperactivity disorder (ADHD)	CCM	19 practices 65 pediatricians 19 family physicians	Unclear	Un- clear	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	No
Epstein et al., 2010 (29)e	Adherence to guidelines for evidence-based assessment and treatment of attention deficit hyperactivity disorder (ADHD)	CCM	47 practices 142 pediatricians 11 family physicians	Unclear	Un- clear	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	No
Epstein et al., 2010 (30)e	Adherence to guidelines for evidence-based assessment and treatment of attention deficit hyperactivity disorder (ADHD)	CCM	31 pediatric practices 123 pediatricians *Data from family physicians excluded	Unclear	Un- clear	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	No
Gustafson et al., 2013 (34)	Time to treatment, client retention, and new patient recruitment in addiction centers	NIATx	201 addiction treatment centers	18 months	Un- clear	No	Yes	Yes	Un- clear	Yes	Yes	Yes	Yes	Yes	Yes	No	No
Haine- Schlagel et al., 2013 (37)	Attendance engagement in community-based early childhood intervention programs	BTS	4 developmental services programs within 1 children’s hospital 29 providers (2 providers did not complete, 1 was added after initiation)	9 months	Yes	No	Yes	Yes	Yes	Yes	Yes	No	No	No	Yes	No	No
Katzelnick et al., 2005 (17)	Implementation of the Chronic Care Model for depression treatment in primary healthcare	BTS, CCM	20 healthcare organizations (3 teams did not complete)	13 months	No	No	Yes	Yes	Un- clear	Yes	Yes	Yes	Un- clear	Yes	Yes	No	Yes
McCarty et al., 2007 (35)f	Time to treatment and client retention in outpatient, intensive outpatient, or residential addiction treatment units	NIATx	FIRST COHORT 13 addiction treatment agencies 7 outpatient, 4 Intensive outpatient, 4 residential units	18 months	Yes	Yes	Yes	Yes	Un- clear	Yes	Yes	Yes	Yes	Yes	Yes	No	No
Hoffman et al., 2008 (36)f	Time to treatment and client retention in outpatient, intensive outpatient, or residential addiction treatment units	NIATx	SECOND COHORT 11 addiction treatment agencies 10 outpatient, 4 intensive outpatient units	18 months	Yes	Yes	Yes	Yes	Un- clear	Yes	Yes	Yes	Yes	Yes	Yes	No	No
Meredith et al., 2006 (31)	Depression treatment in primary care	BTS, CCM	17 mental health agencies	13 months	No	No	Yes	Yes	Yes	Yes	No	Un- clear	No	No	No	No	Yes
Roosa et al., 2011 (19)	Client retention in chemical dependency treatment & client access to mental health services	NIATx	CHEMICAL DEPENDENCY COLLABORATIVE 4 treatment agencies (1 did not complete) MENTAL HEALTH COLLABORATIVE 6 treatment agencies	27 months	No	No	Yes	Yes	Yes	No	No	No	No	No	Yes	Unclear	No
Rutkowski et al., 2009 (38)	Time to treatment, no- show rates, admissions, or continuation in treatment for addiction treatment services	NIATx	PHASE I 6 treatment agencies 7 change teams PHASE II 8 treatment agencies 13 change teams	PHASE I 11 months PHASE II 12 months	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	No	No
Stephan et al., 2011 (39)	Mental health service quality and collaborative care in school- based mental health centers	NA SBHC	19 school-based health centers	15 months	Yes	Yes	Yes	No	Yes	Yes	No	Yes	Yes	Yes	Unclear	No	Yes
Strating et al., 2012 (55)	Four distinct collaboratives focused on: (1) Social psychiatric care; (2) Recovery- oriented care; (3) Social participation; and (4) Somatic co- morbidity of psychiatric clients	BTS	94 distinct teams of mental health care providers COLLABORATIVE 1 25 teams COLLABORATIVE 2 25 teams COLLABORATIVE 3 26 teams COLLABORATIVE 4 18 teams	12 months	No	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	No	No	No
Vannoy et al., 2011 (18)	Integration of services between community health centers (CHCs) and community mental health centers (CMHCs), specifically improving treatment of depression and bipolar disorder in CHCs and improving care of patients at risk of metabolic syndrome in CMHCs	BTS	15 CHC & CMHC pairs (1 pair dropped out due to staff turnover)	12 months 3 cohorts	No	Yes	Yes	Yes	Un- clear	Yes	No	Yes	Yes	Yes	No	No	No
Versteeg et al., 2012 (33)	Implementation of multidisciplinary practice guidelines in mental healthcare organizations (specific domains: anxiety disorders, dual diagnosis, and schizophrenia)	BTS	19 mental healthcare organizations 26 distinct LC teams	12 months	Un- clear	No	Yes	No	Yes	No	Yes	Yes	Yes	Yes	Unclear	No	No

^aArticles are organized by author name and grouped by study. Table Abbreviations: BTS: Breakthrough Series; CCM: Chronic Care Model; NIATx: Network for the Improvement of Addiction Treatment; NASBHC: National Assembly on School-Based Health Care

^bRow Abbreviations: AI: Article Information; LCC: Learning Collaborative Components; QI: Quality Improvement; OP: Organizational Penetration

^cRow Abbreviations: PW: Pre-Work; EP: Expert Panel; ODC: Organizations Demonstrated Commitment; PDSAs: Plan-Do-Study-Act Cycles; MD: Multidisciplinary; QIT: Quality Improvement Team; New QI Data: Sites Collected New Data for QI Purposes; QI Data Review: Sites Reviewed QI Data and Used Feedback; Involved Leaders: Involvement and/or Outreach to Organizational Leadership

^{d, e, f}Multiple publications based on the same study data

Table 2.

LC Components Highlighted for Comparison^*

Study Information	LC Components	QI Processes	Organizational Involvement
As this review is focused on the state of the extant literature, this category highlights the basic study details highlighted by the published article	Given the definition of LCs compiled from a review of the literature, which common LC components were explicitly referenced by study authors	Beyond the basic components of the LC, which quality improvement techniques were included?	In theory, LCs enable an organization to enact change at multiple levels within their organizational structure; Did the LC take steps to train or otherwise involve members of the organization who were not directly included in the collaborative?
Target for Improvement What was the focus of the LC?	Length of Collaborative Can a standard collaborative length be established?	Sites Collected New Data for QI During the collaborative, did sites collect new data for quality improvement purposes?	Leadership Involvement/Outreach Did members of the collaborative involve or otherwise reach out to local leadership?
Model(s) Did the LC align with existing collaborative models?	Pre-Work: Convened Expert Panel The BTS model calls for a planning group that identifies targets for improvement change and plans the collaborative	Sites Reviewed Data & Used Feedback Did the collaborative sites review new data and adjust their practices according to findings?	Training for ‘Non-QI Team Staff Members’ by Experts Did LC faculty or other experts provide training for staff members who were not a part of the QI Team?
Study Sample What was the population of focus?	Pre-Work:Organizations Required to Demonstrate Commitment The BTS model recommends requiring formal commitments, application criteria, or “readiness” activities for LC sites.	External Support with Data Synthesis & Feedback Did LC faculty or other experts provide support with data synthesis and feedback?	Training for ‘Non-QI Team Staff Members’ by the QI Team After the collaborative, did newly trained QI team members provide training for staff members who were not a part of the QI Team?
	In-Person Learning Sessions Teams are traditionally trained in clinical approaches and QI approaches during initial in-person sessions
	PDSAs Plan-Do-Study-Act (PDSA) cycles are a key component of the rapid cycle approach to change recommended by the BTS model
	Multidisciplinary QI Team LCs typically involve staff members at various levels of the organization
	QI Team Calls Calls among QI team members or between members in other participating organizations are a common component
	Email or Web Support Email, listservs, or other forms of web support have become a common approach for providing ongoing support

^*Adapted from: Understanding the Components of Quality Improvement Collaboratives: A Systematic Literature Review. Nadeem, E., Olin, S. S., Hill, L. C., Hoagwood, K. E., & Horowitz, S. M. (2013). The Milbank Quarterly, 91, 354–394.

Description of LC components reported in studies

Ten of the studies were explicitly based on the IHI Breakthrough Series (BTS) model, three of which also noted using the Chronic Care Model, a model originally used as part of a joint effort by the IHI and the Robert Wood Johnson Foundation (RWJF) (41). One additional study cited the Chronic Care Model without the BTS model, four studies reported using the NIATx model for process improvement (42), and one study reported using the National Assembly on School-Based Health Care’s (NASBHC) quality collaborative model based on nationally recognized models for qualiy improvement (39). On average, each study reported implementing 7 LC components. The most commonly reported components included: in-person learning sessions (16 out of 16), multidisciplinary QI teams (12 out of 16), PDSAs (12 out of 16), and QI team calls (12 out of 16). In addition, 11 of the 16 reported doing some leadership outreach or engagement. Across articles, there was great variability in the level of detail provided by descriptions of the components of each collaborative.

Overall LC structure

The LCs lasted an average of 14 months (range 9–27 months), with a modal length of 12 months. Collaboratives typically began with an in-person learning session; LC faculty hosted the sessions and multidisciplinary Quality Improvement (QI) teams attended. Follow-up occurred via additional in-person learning sessions, regular phone meetings for the QI teams, and email or web-based support. Sites conducted QI projects between QI team calls and in-person learning sessions. All in-person learning sessions and most phone meetings involved multiple sites.

Content of in-person learning sessions

All studies reported including in-person learning sessions throughout the course of the collaborative. The most common number of sessions was three (range 1–4). In-person sessions were typically 2 days long, ranging from half-day to 3 day-long sessions. One of the studies was a randomized controlled trial; the four conditions compared interest circle calls (group teleconference calls), individual site coaching, in-person learning sessions, and a combination of all three (34). All of the studies appear to have included in their sessions some didactic training in a particular care process or specific practice. One study, focused on ADHD care in primary care clinics, used a combination of shorter in-person sessions (four 90 minute sessions focused on didactic lectures and quality improvement methods) and office visits (28–30).

In the National Child Traumatic Stress Network (NCTSN) model, all of the LC participants had already received standard treatment developer training in trauma-focused cognitive behavioral therapy (TF-CBT) before the collaborative began (7). Participants in a NIATx collaborative took part in a two-day workshop on an evidence-based practice, “Seeking Safety” (19), in addition to LC activities. Similarly, participants in the NASBHC collaborative learned core components from evidence-based treatment elements for Depression, Anxiety, Disruptive Behavior Disorders and Substance Abuse, and selected manualized interventions (39); and participants in an LC on engagement strategies received training for agency staff in addition to the standard learning sessions (37).

All of the studies that included descriptions of the in-person sessions also reported that they provided training in quality improvement techniques, such as engaging in PDSA cycles or improvement projects. Very few details were provided on the techniques that were taught. In some studies, the LC purveyors had already identified potential areas for improvements that sites should consider for their QI projects (e.g., domains in the chronic care model, system improvements, known implementation barriers) (7, 18, 31, 33). In addition to didactic training related to practices and QI methods, four of the studies reported that individual sites presented information to other participating QI teams during the in-person sessions (7, 15, 27, 37). Few specific details were included on the structure of these cross-site collaborative efforts. Some studies reported having individual site presentations, breakout sessions among “affinity groups,” or the use of “storyboards” (7).

PDSAs

Twelve studies reported using PDSAs between in-person sessions during “action periods.” By and large, it was unclear what occurred during the PDSA cycles, how they were used, or how the ongoing data collection informed the QI process. However, there were a few studies that did provide some detail about the use of QI methods. In those LCs, the faculty set forth possible improvement areas from which site could develop their PDSAs or provided hands on coaching and support (7, 17, 18, 29–31, 34, 37, 38). One study, did not include PDSAs, but rather provided teams with a template to develop “work plans” to facilitate the integration of mental health and primary care in school based health centers (39).

QI team calls

Twelve studies reported that there were calls between in-person sessions for the QI teams. The calls were typically held monthly with the goal of allowing sites to share progress and problem-solve together. Little detail was provided on the content or structure of the calls. Two studies reported holding “all collaborative” calls to facilitate sharing and problem solving (7, 37). Other described “affinity group” calls targeted towards clinical supervisors, change leaders, or executive leadership, or calls focused on specific clinical issues and other special topics (7, 38). Studies using the NIATx model also described holding individual site coaching calls focused on the use of process improvement methods (34, 38).

Email or web support

Six studies reported email or web-based support for the LC participants. Articles did not provide information about the extent to which LC participants used email listservs or web-based support to communicate with other LC participants or LC faculty.

Quality improvement processes

Eleven studies reported some type of ongoing data collection for the purposes of the LC (e.g., performance indicators, ongoing reporting on target outcomes), eight of which reported that the LC faculty provided sites with data-based feedback. Nine studies reported external support with data collection and feedback. With a few exceptions (7, 30, 33, 34, 38), most articles provided very little information about the data collected, how it was used, or how it informed quality improvement activities.

Organizational involvement

Ten studies reported that the organization’s leadership was involved in the LC. However, it was unclear if the organizational leadership was included as a part of the QI team, or was engaged through other outreach efforts. We also examined indicators of the LCs’ penetration into the broader organization by tracking the training provided to non-QI team members, either by LC faculty or by local QI team members themselves. No articles reported providing expert training (conducted by LC faculty or treatment developers) for frontline staff members that were not already on the QI team. Five studies reported that QI team members trained additional staff in the organization.

Pre-LC Activities

Finally, we tracked “pre-work” activities, which we defined as planning activities delineated in the original IHI BTS model (8, 9). Only five studies reported that the LC used an “expert panel” during this pre-work phase, a planning group that identifies targets for improvement change and plans the collaborative. Eight studies reported requiring formal commitments, application criteria, or “readiness” activities prior to the start of the LC.

Study Goals and Findings of Articles Included in the Review

Study Goals

The primary intent of 19 of the twenty articles was either to explore general feasibility and acceptability of the LC model, or to examine pre- to post-collaborative changes at the patient and provider level. The only randomized controlled trial was designed to test different components of the LC in order to determine which were most related to change. In this study, sites were randomly assigned to receive interest circle calls (group teleconference), individual site coaching, in-person learning sessions, or a combination of all three components with the intent of examining which components were related to study outcomes (34). The study’s use of individual site coaching is somewhat unique. One-to-one coaching was described in some studies of the NIATx model (35, 36), but most papers did not specify the use of coaching.

Across the studies, ten articles examined provider-level variables; eleven articles examined patient-level variables; nine articles examined acceptability of the LC model to providers; and eight articles examined sustainability of the changes achieved. One study examined the relation between LC components and study outcomes in an RCT (34). Three of the studies examined how elements of the collaborative process may have contributed to the findings from the collaborative by exploring issues such as the relation between reported barriers/facilitators (31), social networks (31), and theoretically- or empirically-derived attitudinal and contextual factors (e.g., team effectiveness) (33, 40) and changes in outcomes. In addition, two articles provided cost estimates for participation in the collaborative (31, 34) (see Table 3).

Table 3.

Study Variables Across Articles^a

Article	Provider- level variables	Patient- level variables	Acceptability of the LC model to providers	Sustainability of changes	Relationship between individual components of the collaborative and collaborative outcomes	Relationship between aspects of implementation (e.g., barriers, facilitators) and collaborative outcomes	Cost estimates
Cavaleri et al., 2006 (15)b		Yes
Cavaleri et al., 2007 (27)b				Yes
Cavaleri et al., 2010 (16)		Yes
Duffy et al., 2008 (32)	Yes		Yes
Ebert et al., 2011 (7)	Yes		Yes	Yes
Epstein et al., 2008 (28)c		Yes
Epstein et al., 2010 (29)c		Yes
Epstein et al., 2010 (30)c	Yes			Yes
Gustafson et al., 2013 (34)		Yes		Yes	Yes		Yes
Haine-Schlagel et al., 2013 (37)	Yes		Yes
Katzelnick et al., 2005 (17)	Yes	Yes	Yes
McCarty et al., 2007 (35)d		Yes
Hoffman et al., 2008 (36)d		Yes		Yes
Meredith et al., 2006 (31)	Yes		Yes	Yes		Yes	Yes
Roosa et al., 2011 (19)	Yes	Yes	Yes	Yes
Rutkowski et al., 2009 (38)		Yes
Stephan et al., 2011 (39)	Yes		Yes	Yes
Strating et al., 2012 (55)	Yes	Yes	Yes			Yes
Vannoy et al., 2011 (18)	Yes		Yes
Versteeg et al., 2012 (33)						Yes

^aArticles are organized by author name and grouped by study

^{b, c, d}Multiple publications based on the same study data

Study Findings

There was wide variability in the study designs and methods, quality of the methodology, and methodological details provided in the articles. Moreover, with the exception of one RCT (34), the strength of the outcomes was difficult to judge across studies due to the lack of control groups and the variability in the reporting of the LC elements. As such, we were unable to draw conclusions about the overall effectiveness of LC within the mental health context.

However, the study by Gustafson and colleagues (34) does suggest that certain LC elements may be more potent in predicting patient outcomes. Specifically, the authors found that waiting times declined for clinics in the individual site coaching, in-person learning sessions, and combination of three LC components groups; the number of new patients served increased for the combination and coaching only groups; and that interest circle group teleconferences had no impact on outcomes. Although individual coaching and the combination intervention were considered to be similarly effective, individual site coaching was more cost effective ($2878 per clinic versus $7930) (34).

Of the 19 remaining articles, most studies did report positive findings with respect to patient, provider, or sustainability variables. Each of the ten articles that reported on provider-level variables reported positive trends from pre- to post-LC, suggesting improvements in areas such as process of care and uptake of new practices (17, 18, 28, 30–33, 37, 39). Similarly, although there were some mixed findings, each of the eleven articles reporting on patient-level variables reported positive pre- to post-LC changes in areas such as symptoms and engagement in services (15–17, 19, 28, 29, 35, 36). Six of the eight papers that reported on sustainability reported sustained use of new practices or procedures after the conclusion of the collaborative (7, 27, 30, 31, 36, 39). Additionally, the LC model was reported to be feasible and acceptable to providers in each of the nine articles that assessed these variables.

Discussion

The application of LCs to the mental health context is an important area for research as policymakers seek to scale-up evidence-based practices and improve the quality of care. LCs are being widely used as an attractive alternative to traditional developer training models because they hold promise for achieving sustained change in a way that typical treatment developer trainings may not (7, 43–45). LCs can help sites build local capacity and address organization-and provider-level implementation barriers (43, 46, 47). They have the potential to foster local ownership of the implementation process, promote transparency and accountability, create a culture of continuous learning, provide an infrastructure for addressing barriers, and cultivate support networks (7, 43).

The major challenge for the mental heath field is the lack of rigorous studies of LCs. In our previous review, we found 20 studies on LCs in other areas of healthcare that used comparison groups (14)—yet only one study in mental health was an RCT (34). In the current review, we identified 20 articles that reported data on LC outcomes. While we can be encouraged by the positive trends reported in these studies with respect to provider, patient, and sustainability outcomes, these findings must be interpreted with caution given the lack of comparison data. In addition, due to the variability in methods and rigor used in these studies, it was not possible to come to any broad conclusions about the effects of LCs on provider- or patient-level outcomes. It is critical that future research on LCs include more studies with comparison conditions, ideally with randomized designs that can examine the impact of different implementation strategies. There are a number of QI approaches to implementation of new practices that could be tested against learning collaboratives. Specifically, audit and feedback methods from healthcare (48), individual site-focused quality improvement initiatives that involve training of local QI teams, leadership support, coaching, and audit and feedback (49, 50), and the Availability, Responsiveness, and Continuity (ARC) model, an organizational-level quality improvement intervention (51) each have evidence for improving the quality of care. Additionally, a recent review of Six Sigma and Lean continuous improvement approaches borrowed from industry and applied in healthcare suggest these are promising strategies that could be further tested (52). Of particular importance are studies like the one conducted by Gustafson and colleagues (34) that can identify which structural and theoretical components of LCs contribute to favorable outcomes.

Recent studies provide insights into active components that could be directly tested. These include: cross-site and local learning activities (e.g., staff education, PDSAs, team effectiveness) (31, 47, 53–55), local leadership support, sites’ ability to address common implementation barriers, expert support, ongoing data collection, and the visibility of local changes achieved through QI methods (3, 33, 47, 56–59). Additionally, there is a great need to continue to examine the costs associated with LCs and the incremental cost-benefit of using this approach, compared to traditional developer trainings and other QI methods. This type of information is critical for decision makers as LCs can be costly. One study of an LC for depression care (31) reported an average cost of participation at over $100,000 per site. Another suggests that the added cost of in-person learning sessions may not bring much incremental cost-benefit with respect to patient outcomes, compared to individual site coaching (34).

With respect to the reporting of LC components, we found similar patterns to those found in previous research. Prior reviews highlight the variation in the implementation of LC model and inconsistent reporting of components (4, 13, 14, 25). Across studies, the LCs in the current review had a similar structure. However, there was insufficient detail provided with respect to presence of LC components and how they were implemented in majority of studies. Moreover, as the original collaborative models in healthcare were based on management theory (10–12), the lack of specificity on how process improvement was conducted, how QI data was collected, and how data was used is striking. It is essential to carefully describe how quality improvement methods are being used in mental health care because previous studies have suggested that LC participants perceive instruction in QI methods to be useful (31, 47, 59), and because the innovations implemented in mental health are often complex evidence-based treatments that may require adaptations from the original QIC models in healthcare. The current review provides one potential template for the reporting of specific LC components, each of which should be reported in sufficient detail that others could replicate the activities and processes (i.e., dosage provided, engagement of participants, details on how QI was taught, how data was use, how teams and leadership was engaged). In addition, it will be important for future research to report on and explore theoretically-driven active ingredients for LC by examining not only structure but also LC processes.

There are limitations that should be considered in interpreting these findings. As with any systematic review, it is possible that relevant studies were omitted. By searching multiple databases, reviewing the reference lists of key articles, and crosschecking with free-text search terms, we minimized the possibility of such omissions. In addition, negative findings are generally not published, potentially biasing our results. Despite these potential limitations, our review does provide an important assessment of the state of the evidence for the use of LCs in mental health care. The uses of LCs that focus on processes of care (e.g., engagement practices, depression guidelines implementation) align more closely with the targets of collaboratives that have been applied in other areas of healthcare. The applicability of LCs for disseminating and implementing more complex mental health evidence-based practices remains unknown; in the mental health field, such efforts often require additional specialized trainings to develop provider skills in implementing these evidence-based practices. The cost-effectiveness or added value of such an approach must thus be carefully assessed.

As LCs continue to grow in popularity among policymakers and national organizations, there is great need for rigorous research that evaluates the utility of these costly endeavors. Moreover, research focused on active components of LCs is vital to the replication of successful LCs, ensuring quality and fidelity to the model, guiding future adaptations, and identifying the types of innovations and improvements for which the model is most appropriate.