Long-Term Clinical Outcomes from a Randomized Controlled Trial of Two Implementation Strategies to Promote Collaborative Care Attendance in Community Practices

Abstract

This randomized controlled implementation study compared the effectiveness of a standard versus Enhanced version of the Replicating Effective Programs (REP) implementation strategy to improve the uptake of the Life Goals-Collaborative Care Model (LG-CC) for bipolar disorder. Seven community-based practices (384 patient participants) were randomized to standard (manual/training) or Enhanced REP (customized manual/training/Facilitation) to promote LG-CC implementation. Participants from Enhanced REP sites had no significant changes in primary outcomes (improved quality of life, reduced functioning or mood symptoms) by 24 months. Further research is needed to determine whether implementation strategies can lead to sustained, improved participant outcomes in addition to program uptake.

Introduction

It can take years if not decades for effective treatments or practices to be implemented into community-based settings (Proctor et al., 2009). This research-to-practice gap profoundly impacts access to effective treatments for persons with mental health conditions (Proctor et al., 2009), who are disproportionately burdened by functional impairment and premature mortality (Druss et al., 2008; Gale et al., 2012; Sharfstein, 2011). Collaborative care models (CCMs) are evidence-based practices shown to improve medical and psychiatric outcomes for persons with chronic mental disorders (Woltmann et al., 2012). CCMs are typically implemented by a clinical nurse or social worker (“care manager”) who provides individual self-management support, panel management, ongoing outcomes assessment, and linkages to specialists in collaboration with the individual's primary care or mental health physician.

However, CCMs have not been widely disseminated in community-based practices (Kilbourne et al., 2004), in part because of their multicomponent nature, limited access to specific, user-friendly tools for providers to implement CCM core components, and lack of ongoing provider support to address barriers to CCM adoption (Coleman, Austin, Brach, & Wagner, 2009; Katon, 2012). Without additional implementation support, providers are unlikely to sustain the CCM in their practice.

A number of implementation frameworks have been proposed to understand the barriers and facilitators to the uptake of multicomponent practices such as the CCM (Chinman, Imm, & Wandersman, 2004; Damschroder et al., 2009; Rubenstein et al., 2002; Solberg et al., 2001; Stetler, Damschroder, Helfrich, & Hagedorn, 2011). These frameworks primarily focus on identifying factors at the organizational level that can facilitate or impede adoption of effective practices. However, few implementation frameworks have been operationalized into specific strategies to help mitigate organizational barriers and help support providers in the implementation of effective practices in their clinical setting. Implementation strategies comprise a set of specific organizational (i.e., provider or clinic-level) interventions meant to help providers use and sustain effective clinical practices in routine care, and can include features such as toolkits or guides for using the effective clinical practice, training, as well as coaching or ongoing feedback on clinical practice use (Gaglio & Glasgow, 2012).

Replicating Effective Programs (REP), based on the Centers for Disease Control and Prevention Division of HIV/AIDS Prevention's Research-to-Practice Framework, is an implementation strategy based on Rogers' Diffusion of Innovations (Rogers, 2003) and Social Learning Theories (Bandura, 1977). REP specifically outlines how effective programs can be implemented, notably through their “packaging” into user-friendly components which are supported by structured training and brief technical assistance for providers. In a randomized controlled implementation trial, REP was shown to improve the uptake of HIV behavioral change interventions (Kelly et al., 2000). However, it is unclear whether REP can result in a public health impact, notably improved patient outcomes over time, for multicomponent treatments such as CCMs.

Based on a recent review of the implementation literature and feedback from community-based practices (Kilbourne, Neumann, Pincus, Bauer, & Stall, 2007; Kilbourne, Neumann, et al., 2012), Enhanced REP was developed as an extension of the original REP framework to promote the use of CCMs for mental health conditions. While the primary “target population” of Enhanced REP is still the principal provider responsible for implementing the CCM, they appeared to need additional guidance in integrating the CCM into their clinical practice. To this end, Enhanced REP added an implementation intervention called Facilitation, which is based on the Promoting Action on Research Implementation in Health Services (PARiHS) framework (Rycroft-Malone, 2004; Stetler et al., 2006). As described in previous research (Kirchner, Ritchie, Dollar, Gundlach, & Smith, 2013), Facilitation involves two distinct roles: an External and Internal Facilitator, both of which support providers in using multicomponent clinical practices (i.e., CCMs). External Facilitators represent a core group of experts typically from the study team who provide technical expertise in CCM implementation. External Facilitators coach the provider in customizing the CCM to meet specific clinical workflow needs in their practice and provide guidance in addressing organizational barriers to implementation. In contrast, Internal Facilitators are employees of the clinical practice (e.g., administrators or clinical managers) who are identified by clinical leadership as the “go-to” person for managing and making changes to clinic practice workflows. Internal Facilitators are not seen as direct advocates of the CCM but rather can help the care manager integrate CCM components into routine care through their internal knowledge of the clinic administration (Kirchner et al., 2014; Kirchner et al., 2013).

We report findings from the Recovery-Oriented Collaborative Care (ROCC) study, a cluster randomized controlled trial comparing the effects of standard versus Enhanced REP (REP+ External and Internal Facilitation) on improved clinical outcome sat the individual participant level when applied to help providers implement a CCM. The CCM used in the study is Life Goals Collaborative Care- LG-CC, an evidence-based practice for persons with bipolar disorder (Waxmonsky et al., 2014). Bipolar disorder was chosen because it is a chronic illness characterized by recurring manic and depressive symptoms and is associated with high suicide rates (Novick, Swartz, & Frank, 2010), functional impairment (Huxley & Baldessarini, 2007), and high health care costs (Dean, Gerner, & Gerner, 2004). LG-CC has been shown in several randomized controlled trials to improve outcomes for individuals with bipolar disorder (Bauer et al., 2006b; Kilbourne, Goodrich, et al., 2012; Kilbourne et al., 2013; Kilbourne et al., 2008; Simon, Ludman, Bauer, Unutzer, & Operskalski, 2006). Yet LG-CC has not been fully translated into community-based practices. The primary hypothesis is that participants from Enhanced versus standard REP sites will experience improved health-related quality of life, decreased functional impairment, or decreased mood symptoms.

Methods

Seven community-based mental health and primary care clinics (sites) were randomized to receive either standard REP or Enhanced REP to facilitate implementation of Life Goals Collaborative Care (LG-CC). Participating sites were initially selected from a total of nine clinics represented by three large organizations from Michigan and Colorado. Eligible sites included mental health or primary care outpatient facilities that cared for at least 200 individuals with bipolar disorder, and had at least two providers with mental health treatment experience available to implement LG-CC.

Clinical Practice to be Implemented (LG-CC)

LG-CC for bipolar disorder (Table 1) was chosen in part because community-based providers expressed a need for EBPs for this illness and because LG-CC has been shown to improve outcomes among individuals with bipolar disorder, particularly among minority groups and individuals with co-occurring conditions (Kilbourne, Neumann, et al., 2012). LG-CC includes four sessions focused on individual self-management, ongoing individual and provider contacts to reinforce self-management skills, provision of care management, guideline support for providers, and a registry tool to track individual participant progress (Table 1).

Table 1. Life Goals Collaborative Care Core Components.

LG-CC Component	LG-CC Core Element
Self-management Sessions	Four group sessions led by care manager that involved active discussions focused on conditions/counseling approaches Discussions with patients to enhance/maintain physical and mental health goals and symptom recognition Discussions with patients on cost-benefit of symptoms management and provider engagement
Care Management	Maintenance of lessons from self-management sessions learned via ongoing contacts with the care manager Regular communication by the care manager with providers including mental health providers and general medical physicians on patient clinical status Outreach/crisis management after critical service encounters Regular documentation of patient clinical status and provider contact, including summary of clinical status and wellness goals
Provider Support	Mental health and general medical providers were given information on relevant bipolar disorder treatment guidelines

Implementation Strategy Randomization

The study analyst randomized community-based sites stratified by state (MI, CO) to receive either the Enhanced or standard Replicating Effective Programs (REP) implementation strategy to support the uptake of LG-CC (Table 1).

Standard and Enhanced REP Implementation Strategies

Timelines for standard and Enhanced REP components are presented in Table 2. Standard REP components, delivered to a designated provider at each site by the study investigator team, included the LG-CC manual (AK, JW, JK), LG-CC training (AK, JK, MB), and brief, as-needed technical assistance (JW, JK). Enhanced REP components, also delivered by the same study personnel, included customization of the LG-CC manual and training program based on input from local site providers (AK, JW, JK), and External Facilitation (AK, JW). Specifically, study investigators obtained input from providers and clinical leadership at each site randomized to receive Enhanced REP in customizing the LG-CC package and training program to fit their local needs. Using previously described methods in obtaining input in customizing CCMs (Kilbourne, Neumann, Pincus, Bauer, & Stall, 2007; Kilbourne, Neumann, et al., 2012; Bauer et al., 2006a; Simon, Ludman, Unutzer, & Bauer, 2002), clinical managers from each Enhanced REP site completed a needs assessment of their current site's priorities and barriers and facilitators to LG-CC implementation and provided input via face-to-face interviews on customizing the standard LG-CC package and training program. For the face-to-face interviews, study investigators used as now ball purposeful sampling method (Palinkas, Horwitz, Green, Widsom, Duan, and Hoagwood, 2013) to select providers from the Enhanced REP sites by job type. The site administrator was initially contacted and interviewed, and a snowball sampling method was used to identify and interview at least two additional master's level clinical social workers, at least two additional bachelor's level case managers, and at least one physician, physician, or nurse practitioner. Providers were interviewed to obtain additional recommendations in customizing the LG-CC package/training program. At Enhanced REP sites, study investigators (AK, JK, JW) also held a focus group of 6-8 consumers recruited from each Enhanced REP sites to obtain additional input on customizing the LG-CC package components.

Table 2. Enhanced and Standard REP Intervention Components.

	Standard REP		Enhanced REP
Phase and Time Period	Standard REP Components	Total Time Spent during Period & Across Standard REP Sites	Enhanced REP Components	Total Time Spent during Period & Across Enhanced REP Sites
Package (6 Month Period)	Non-customized LG-CC manual development	TOTAL: 840 minutes	LG-CC manuals customized to sites using data from organizational needs assessment on population and treatment setting resources Site manager and provider interviews, consumer focus group provided further input into customization	Customized Manual:1470 minutes Interviews/focus groups: 930 minutes TOTAL: 2400 minutes
Training (3 Month Period)	Standard LG-CC training	TOTAL: 480 minutes	Customized training for providers based on input on LG-CC package	Training customization: 1660 minutes Training: 480 hours TOTAL: 2140 minutes
Facilitation (12 Month Period)	As-needed technical assistance (TA) if practice contacts study team for 6 months Monitor LG-CC uptake via monthly reporting sheets: 12 months	TA: 1470 minutes Monitoring: 500 minutes TOTAL: 1970 minutes	External Facilitation (EF) Components: Initiation: EF identifies IF Provider Contact: EF works with IF and LG-CC providers to review potential barriers and set measureable goals in LG-CC uptake: monthly1-hour meetings for 12 months Technical Assistance: EF makes calls to IF and LG-CC providers, giving specific guidance on implementation: 30-minute calls/mon Ongoing marketing and Sustainability: EF supports public recognition of LG-CC success stories and helps IF and LG-CC provider to summarize progress and develop long-term plans for sustainability Internal Facilitation (IF) Components: IF meetings with EF, LG-CC providers to establish measureable goals in LG-CC uptake, identify how LG-CC is aligned with practice priorities, and publicize LG-CC to leadership (monthly 1-hour meetings) Additional brief consultations with LG-CC provider on average 15 minutes/month)	External Facilitator time across all Enhanced REP sites: Components 1-3: 3740 minutes Component 4: 640 minutes Internal Facilitator time across all Enhanced REP sites: 3600 minutes TOTAL: 7980 minutes

Providers delivering LG-CC at Enhanced REP sites were also given ongoing facilitation for 12 months (Table 2). External Facilitators who were part of the study team and were external to the clinical practice (AK, JW, JK) provided guidance to practices in implementing LG-CC, identified potential barriers to implementation based on the previous needs assessment, and helped the providers obtain support for the LG-CC program at the site. In addition, External Facilitators with input from site program leadership identified an Internal Facilitator at the beginning of initiating Facilitation. The Internal Facilitator had to be an employee and in a managerial role at the site with a direct reporting line to leadership, but was not responsible for directly delivering LG-CC. The Internal Facilitator from each site met monthly with the LG-CC care manager and External Facilitator to support the care manager in addressing specific organizational barriers to LG-CC implementation. When available, the Internal Facilitator, External Facilitators and LG-CC provider also met with practice leadership on a regular basis. Enhanced REP Internal and External Facilitators also worked with the LG-CC provider to regularly report on implementation progress, identify opportunities to leverage LG-CC implementation to address other site quality improvement goals, and develop a plan for sustaining LG-CC beyond the study (Table 2).

The total time for each standard and Enhanced REP component was recorded by study research assistants using a standardized form that was previously established for similar LG-CC implementation studies (Kilbourne, Goodrich, et al., 2012; Kilbourne et al., 2013; Kilbourne et al., 2008; Simon, Ludman, Bauer, Unutzer, & Operskalski, 2006). Total time spent on Facilitation activities by each Internal Facilitator and LG-CC care manager was recorded on a monthly basis by general area of activity and tallied across all sites randomized to standard REP and forall sites randomized to Enhanced REP (Table 2)

Procedures

One care manager at each site was jointly hired by research and clinical staff to implement LG-CC components (Table 1). All LG-CC care managers had a clinical social work background (MSW) and one to five years of mental health clinical background and previous experience in addressing suicidal ideation, severe manic episodes, and other emergent issues. LG-CC care managers identified and enrolled individuals with an active diagnosis or treatment plan for bipolar disorder based on medical record reviews. Adult individuals receiving care at the site were eligible if they had a diagnosis of bipolar disorder, received outpatient care from the participating practices, and were community-dwelling; and were excluded if they: 1) had evidence of intoxication or active suicidal ideation at enrollment precluding participation in LG-CC components as indicated by the provider or 2) were not able to provide informed consent.

Eligible individuals were then approached in person and enrolled after providing informed consent to participate. Once consented, participants completed a baseline assessment survey, and were scheduled by the care managers to attend LG-CC group sessions. Participants were asked to attend four weekly group sessions over a 4 week-period, followed by monthly individual care management phone calls over 6 months. Care managers recorded participant attendance to LG-CC sessions and subsequent contacts, health behavior goals, and clinical status using the LG-CC electronic registry. Each participant received $5 in remuneration for each group session attended to cover transportation costs, and $20 for the baseline survey.

Data Collection

The primary aim of the study was to compare Enhanced versus standard REP used to implement LG-CC on changes in individual participant-level outcomes. A prior analysis from the same study focused on the impact of Enhanced versus standard REP on fidelity, as measured by frequency of client attendance at group sessions and number of care manager contacts to LG-CC (Waxmonsky et al., 2014). After study enrollment, participants completed a 30-minute survey in person or by phone at baseline and 6, 12, and 24 months later. The primary outcome for this intervention included mood symptoms, health-related quality of life, and functioning. Mood symptoms were ascertained from the Patient Health Questionnaire 9-item survey (PHQ-9) for depressive symptoms (Kroenke, Spitzer, & Williams, 2001; Spitzer, Kroenke, & Williams, 1999) and the Internal State Scale (ISS) (Bauer, Vojta, Kinosian, Altshuler, & Glick, 2000; Glick, McBride, & Bauer, 2003) for manic symptoms. The PHQ-9 has been validated across diverse populations to both screen for depression and monitor response to treatment. The questionnaire has also demonstrated strong test-retest reliability (r_(ICC) = .81 to .96) in populations with chronic illnesses and responsiveness to change over time, such that a change of 5 points on the 27 scale constitutes clinically meaningful change in depressive symptoms (Lowe, Unutzer, Callahan, Perkins, & Kroenke, 2004; Spitzer et al., 1999). Depression remission was also calculated based on the PHQ-9, defined as having a 50% reduction in PHQ-9 score and PHQ-9 score of less than 5 at 24 months (Kroenke, Spitzer, Williams, & Lowe, 2010).

Manic symptoms were measured using the Internal State Scale (ISS), a 15-item self-report measure designed to provide a mood state self-report and level of symptomatology. The ISS activation subscale has been used to assess level of manic symptoms based populations with chronic illnesses (Bauer et al., 2000; Glick et al., 2003). Health-related quality of life was assessed using the Short Form 12-item survey to assess individual self-reported health, physical and psychological symptoms, and limitations in activities of day living due to physical and mental health impairments experienced during the prior four weeks (SF-12) (Ware, Kosinski, & Keller, 1996). The SF-12 generates a mental and physical health component score (MCS, PCS) that are standardized to the general population (range: 0-100 each scale). Original psychometric evaluation found 2 week test-retest reliability with correlations ranging from 0.76 to 0.89, and criterion validity scores for identifying physical and mental health conditions ranged from 0.43-0.93 (median=0.67) for PCS and 0.60-1.07 (median=0.97) for MCS (Ware et al., 1996). Functional disability was assessed using the World Health Organization assessment disability scale (WHO-DAS-2) (Ustun & Chisholm, 2001) which measures impairment in activities in daily living across the subscales of cognition, mobility, self-care, social functioning, and role functioning. The scale has strong discriminant validity and construct validity in identifying first episodes of depression (Luciano, Ayuso-Mateos, Fernandez, Aguado, et al., 2010) and good internal consistency (Chronbach α=.89) (Luciano, Ayuso-Mateos, Fernandez, Serrano-Blanco, et al., 2010) while demonstrating sensitivity to change in multi-condition CCM interventions (Von Korff et al., 2011).

Analyses

The target participant sample size was 350 (70 participants per site). Assuming an interclass correlation of <.01 this sample size enables adequate power to detect significant effects in changes in our symptom and quality of life outcomes (Cohen's D=.30). Intent-to-treat analyses using multivariable linear mixed models were applied to compare changes in 6, 12, and 24-month outcomes among participants from Enhanced versus standard REP sites. Multivariable logistic regression was also used to compare the odds of participants from Enhanced versus standard REP sites on depression remission. Multivariable models included an indicator for Enhanced REP (versus standard REP), time (6, 12, 24-month), the interaction of Enhanced REP and time.

The models also controlled for participants' baseline demographic and clinical characteristics as well as baseline values of the outcome measures that were found in our previous study to influence changes in outcomes (Waxmonsky et al., 2014), as well as changes in long-term outcomes in other research trials (i.e., 24-month) results (Harpole et al., 2005; Unutzer et al., 2001). Individual covariates included age, gender, race (non-white versus white), college education (vs. no), unemployed (vs. employed, student, or retired), living alone (vs. living with someone), lifetime history of homelessness (vs. no), hazardous drinking, and self-reported number of comorbid illnesses. Hazardous drinking was defined based on the AUDIT-C question on binge drinking (having consumed 6 or more drinks on a single occasion within the past month) (Dawson, Grant, & Stinson, 2005). Number of comorbid illnesses was ascertained from participant self-report and based on the question, “Has the doctor ever told you that you have one or more of the following:” with check boxes for hypertension/high blood pressure, arthritis/chronic pain, angina/coronary heart disease, heart attack/MI, depression, PTSD, diabetes/high blood sugar, or high cholesterol or parents with high cholesterol.

An exploratory analysis was also conducted to determine whether changes in participant outcomes might be explained by differences in attendance to the LG-CC intervention, defined as the total dose, or number of group sessions and care management contacts combined received by each participant. Described previously (Waxmonsky et al., 2014) the specific fidelity construct used was intervention attendance, or total frequency of LG-CC group sessions and number of care manager contacts completed by the LG-CC participant (Bellg et al., 2004). Participant-level total number of sessions and contacts was then added to each regression multivariable model.

Results

Out of 2,019 individuals initially screened for study participation, 384 were eligible for study participation. After participants were enrolled and consented, 320 started LG-CC at the standard REP (n=143) and Enhanced REP (n=177) practices. Participants from standard REP sites were less likely to have a college education or more likely to be unemployed; otherwise, there were no other demographic or clinical differences among those from Enhanced or standard REP sites (Table 3). The mean age among individual participants was 42 years and 66.7% were female.

Table 3. Characteristics of Patient Characteristics and Baseline Outcome Measures: Overall and Stratified by Enhanced versus Standard REP Sites.

	Overall N=384	Enhanced REP (n=221)	Standard REP (n=163)
Demographic Characteristics	Mean (SD)	Mean (SD)	Mean (SD)
Age, years	42.0 (11.3)	42.2 (11.4)	41.8 (11.3)
Number of medical comorbidities	2.6 (1.1)	2.6 (1.0)	2.6 (1.1)
	N (%)	N (%)	N (%)
Female	256 (66.7)	146 (66.1)	110(67.5)
Non-White	108 (29.3)	54 (25.2)	54(34.8)*
College Education or greater (yes, no)	71 (18.8)	59 (27.1)	12(7.5)**
Unemployed (yes, no)	279 (72.7)	149 (67.4)	130(79.8)
Alcohol misuse	40 (10.7)	24 (11.2)	16 (10.0)
Illicit drug use	123 (32.0)	70 (31.7)	53 (32.5)
Clinical Characteristics	Mean(SD)	Mean(SD)	Mean(SD)
Baseline Depressive Symptoms (PHQ-9)(a)	12.9 (6.4)	13.4 (6.7)	12.4 (5.9)
Baseline Manic Symptoms: ISS(b)	20.3 (12.7)	19.3 (12.8)	21.6 (12.4)
Baseline Health-related Quality of Life Scores (SF-12) MCS(c)	31.8 (8.4)	31.2 (8.3)	32.6 (8.4)
Baseline Health-related Quality of Life Scores (SF-12) PCS(c)	36.5 (7.5)	37.2 (7.4)	35.6 (7.5)

^aDepressive symptom scores were based on the Patient Health Questionnaire (PHQ-9); a higher score reflects more symptoms for these three measures.

^bManic symptom scores were based on the Internal State Scale (ISS), possible scores range from 0 to 50 and a higher score indicates worse manic symptoms.

^cHealth-related quality of life (SF-12) includes a mental component scale (MCS) and physical component scale (PCS). Possible scores range from 0 to 100, with higher scores indicating better health. For both summary scores, the population M ± SD is 50 ± 10.

^dWHO-DAS: World Health Organization Disability Adjustment Scale is a scale measuring social and physical dysfunction, with a range of 0-48 and higher scores indicating greater impaired functioning

There were no significant differences inPHQ-9, ISS, WHO-DAS, or SF-12 scores at baseline between participants from Enhanced versus standard REP sites (all P>.05, see Table 3). Reliability (Cronbach Coefficient scores) for the PHQ-9, ISS, and WHO-DAS from the results were respectively .86, .84, and .88.

Overall, study staff spent a total of 3,290 minutes, or 54 hours across the three Standard REP sites, and 7,980 minutes, or 133 hours across the four Enhanced REP sites over the 21-month implementation intervention period. The greater amount of time spent in Enhanced REP activities was primarily attributed to External Facilitation (e.g., coaching the LG-CC care managers) and meetings with Internal Facilitators (Table 2).

After adjusting for participant demographic and clinical factors, there were no significant differences in outcomes among participants from Enhanced versus standard REP sites (Table 4). The addition of LG-CC attendance as a potential explanatory factor also revealed similar results (Table 4).

Table 4. Twenty-Four Month Outcomes Comparing Patients from Enhanced REP versus Standard REP Sites Adjusting for Patient Factors and LG-CC Attendance: Multivariable Results with and without Addition of LG-CC Attendance^ǂ.

	Depressive Symptoms (PHQ-9) (a) Beta (95% CI)		Depression remission (PHQ 9 <5 and 50% reduction) OR (95% CI)		Manic Symptoms: ISS (b) Beta (95% CI)		SF-12 Mental Health (MCS) Score (c) Beta (95% CI)		SF-12 Physical Health (PCS) Score (c) Beta (95% CI)		Functional Disability: WHO-DAS (d) Beta (95% CI)
	Without LG-CC Attendance	Adding LG-CC Attendance	Without LG-CC Attendance	Adding LG-CC Attendance	Without LG-CC Attendance	Adding LG-CC Attendance	Without LG-CC Attendance	Adding LG-CC Attendance	Without LG-CC Attendance	Adding LG-CC Attendance	Without LG-CC Attendance	Adding LG-CC Attendance
Enhanced REP (ref: standard REP)	0.5 (-0.4, 1.4)	0.6 (-0.5, 1.6)	1.4 (0.6, 3.0)	1.5 (0.6, 3.4)	-0.04 (-1.9, 1.8)	0.5 (-1.6, 2.6)	-0.3 (-1.6, 1.1)	-0.2 (-1.7, 1.3)	0.5 (-0.5, 1.6)	0.6 (-0.6, 1.9)	-0.2 (-1.6, 1.1)	-0.1 (-1.6, 1.4)
Time	-0.1 (-0.1, -0.01)*	-0.1 (-0.1, -0.004)*	--	--	-0.1 (-0.2, -0.001)*	-0.1 (-0.2, -0.004)*	0.03 (-0.1, 0.1)	0.03 (-0.1, 0.1)	0.04 (-0.03, 0.1)	0.04 (-0.02, 0.1)	-0.04 (-0.1, 0.03)	-0.1 (-0.1, 0.02)
Enhanced REPX Time	-0.1 (-0.1, 0.01)	-0.1 (-0.1, 0.02)	--	--	-0.03 (-0.2, 0.1)	-0.02 (-0.2, 0.1)	0.1 (-0.03, 0.2)	0.1 (-0.05, 0.2)	-0.1 (-0.1, 0.03)	-0.1 (-0.1, 0.03)	-0.05 (-0.1, 0.05)	-0.03 (-0.1, 0.1)
LG-CC Attendance	--	0.01 (-0.1, 0.2)	--	0.9 (0.8, 1.2)	--	-0.2 (-0.5, 0.1)	--	-0.05 (-0.3, 0.2)	--	-0.005 (-0.2, 0.2)	--	-0.1 (-0.3, 0.1)
Outcome at baseline	0.7 (0.7, 0.8)*	0.7 (0.6, 0.8)*	0.9 (0.8, 0.9)*	0.9 (0.8, 0.9)*	0.6 (0.6, 0.7)*	0.6 (0.5, 0.7)*	0.5 (0.5, 0.6)*	0.5 (0.4, 0.6)*	0.7 (0.7, 0.8)*	0.7 (0.7, 0.8)*	0.8 (0.7, 0.8)*	0.8 (0.7, 0.8)*
Age	-0.01 (-0.04, 0.03)	-0.004 (-0.04, 0.03)	1.0 (0.9, 1.0)	1.0 (0.9, 1.0)	-0.03 (-0.09, 0.04)	-0.02 (-0.1, 0.1)	0.03 (-0.02, 0.1)	0.03 (-0.02, 0.1)	-0.04 (-0.1, -0.01)*	-0.05 (-0.1, -0.04)*	0.1 (0.003, 0.1)*	0.1 (0.01, 0.1)*
Female	0.5 (-0.2, 1.2)	0.5 (-0.3, 1.3)	0.8 (0.4, 1.6)	0.8 (0.3, 1.7)	-0.2 (-1.7, 1.2)	-0.5 (-2.0, 1.0)	-0.6 (-1.6, 0.5)	-0.6 (-1.7, 0.5)	-0.3 (-1.2, 0.5)	-0.3 (-1.2, 0.6)	1.2 (0.2, 2.2)*	1.0 (-0.1, 2.1)
Nonwhite	0.5 (-0.3, 1.2)	0.5 (-0.3, 1.4)	0.5 (0.2, 1.2)	0.5 (0.2, 1.2)	1.3 (-0.3, 2.8)	1.2 (-0.4, 2.8)	0.3 (-0.8, 1.4)	0.3 (-0.8, 1.5)	-0.7 (-1.6, 0.2)	-0.8 (-1.7, 0.2)	0.1 (-0.9, 1.2)	0.1 (-1.0, 1.3)
College education	-1.4 (-2.3, -0.5)*	-1.4 (-2.4, -0.5)*	1.2 (0.5, 2.9)	1.2 (0.5, 2.9)	-2.3 (-4.1, -0.5)*	-2.4 (-4.3, -0.5)*	0.2 (-1.1, 1.5)	0.1 (-1.2, 1.5)	1.2 (0.1, 2.3)*	1.2 (0.1, 2.4)*	-1.2 (-2.5, 0.1)	-1.3 (-2.6, 0.1)
Unemployed	-0.4 (-1.2, 0.4)	-0.4 (-1.2, 0.4)	1.4 (0.6, 3.2)	1.4 (0.6, 3.2)	0.1 (-1.4, 1.7)	0.1 (-1.6, 1.7)	0.7 (-0.4, 1.9)	0.7 (-0.5, 1.9)	-0.4 (-1.3, 0.5)	-0.4 (-1.4, 0.6)	-0.5 (-1.6, 0.7)	-0.6 (-1.8, 0.5)
Living alone	-0.1 (-0.8, 0.7)	-0.1 (-0.9, 0.7)	1.2 (0.6, 2.7)	1.3 (0.6, 2.7)	-0.1 (-1.6, 1.4)	-0.1 (-1.7, 1.5)	0.6 (-0.5, 1.7)	0.7 (-0.5, 1.8)	0.7 (-0.2, 1.6)	0.8 (-0.2, 1.7)	-0.3 (-1.4, 0.8)	-0.4 (-1.5, 0.7)
Homeless History	0.4 (-0.3, 1.1)	0.4 (-0.4, 1.1)	0.7 (0.3, 1.4)	0.7 (0.3, 1.4)	0.7 (-0.7, 2.0)	0.4 (-1.1, 1.9)	-0.5 (-1.5, 0.4)	-0.5 (-1.5, 0.6)	-0.05 (-0.9, 0.8)	-0.1 (-0.9, 0.8)	0.2 (-0.8, 1.1)	-0.1 (-1.1, 0.9)
Hazardous drinking	-0.9 (-2.2, 0.2)	-1.2 (-2.5, 0.04)	3.6 (1.1, 11.7)*	3.5 (1.1, 11.6)*	-0.4 (-2.7, 1.9)	-0.6 (-3.2, 1.9)	0.4 (-1.2, 2.1)	0.7 (-1.1, 2.5)	-0.5 (-1.9, 0.8)	-0.7 (-2.2, 0.8)	-1.2 (-2.8, 0.5)	-1.7 (-3.5, 0.1)
# comorbid illnesses	0.1 (-0.2, 0.4)	0.1 (-0.3, 0.4)	1.2 (0.8, 1.7)	1.2 (0.8, 1.7)	-0.3 (-1.0, 0.4)	-0.3 (-1.1, 0.4)	-0.1 (-0.6, 0.4)	-0.1 (-0.6, 0.4)	-0.1 (-0.5, 0.3)	-0.03 (-0.5, 0.4)	-0.2 (-0.7, 0.3)	-0.2 (-0.8, 0.3)

^ǂIn addition to the variables listed in the table, multivariable models with LG-CC attendance included the total number of self-management sessions and care management contacts (range: 0-19). Depression remission is defined as % with >=50% drop in PHQ-9 score and endpoint score of <5.

^*<.05

Discussion

We describe results from a randomized controlled trial comparing the effects of two different implementation strategies on changes in participant-level outcomes for mental disorders. We found that compared to standard REP, Enhanced REP was not associated with improved mental or physical health-related quality of life, reduced functional impairment, manic symptom severity, or greater likelihood of depression remission at 24 months after adjusting for participant factors. While in a previous study we found that Enhanced REP led to improved LG-CC attendance (Waxmonsky et al., 2014), participant outcomes did not appear to be explained by changes to LG-CC attendance in this current study.

To our knowledge, ROCC is one of the first studies assessing whether a specified implementation strategy designed for a CCM for bipolar disorder impacts participant-level outcomes. Specifically, the Enhanced REP facilitation strategy involved active generation of input from providers and consumers in customizing the program, coaching of frontline providers in program implementation by study team members (External Facilitators), and the identification of Internal Facilitators at each site to identify where and how LG-CC can be leveraged to enhance other required services (Rubenstein et al., 2002; Stetler et al., 2011). These Facilitation components have been employed in previous studies, notably in the VA (Kirchner et al., 2014; Kirchner et al., 2013), and have been referred to in other research as Consultation or Coaching (Michie, Fixsen, Grimshaw, & Eccles, 2009). However, few studies have reported on the impact of Facilitation or similar implementation strategies on changes in participant-level outcomes. While typical implementation endpoints involve provider adoption of the program or participant-level use (Curran et al. 2012), benchmarking implementation strategies on measureable participant outcomes can help determine whether any “voltage drop” occurs when programs are rolled out into community settings beyond the typical research setting.

The observed null findings of our study could be attributed to several factors. It is likely that the implementation of LG-CC, which occurred during the first 12 months of the participant observation, did not have lasting effects on participant outcomes within 24 months. Perhaps attendance to LG-CC components, while higher with Enhanced REP (Waxmonsky et al., 2014), may not have increased enough for clinical outcomes to improve over time. Moreover, while the External Facilitator roles were standardized (same study team members across intervention sites), the experience of the Internal Facilitators varied widely across the sites, which may have led to variations in the ability to better integrate LG-CC components into routine practice to achieve maximum effect on participant-level quality and outcomes.

Reliance on a single care manager at each site to lead the implementation of LG-CC without involving other clinicians art the site may also have contributed to a lack of sustained impact of LG-CC over time. Variations in the organizational capacity of the practices varied, in which some sites may not have had organized practice teams with prior experience in implementing CCMs, nor have the resources to identify ways in which LG-CC could be better integrated, notably through changes in provider workflow, incorporation of measurement-based care, or training of other clinicians to employ self-management practices that aligned with LG-CC components.

In addition, evidence from other studies comparing implementation strategies may point to key lessons that can inform the refinement of Enhanced REP in order to maximize its public health impact and sustainability (Glasgow, 2013; Knox et al., 2011). First, focusing implementation efforts on a single provider, even with customized manuals and training is insufficient to influence outcomes without ongoing efforts to integrate components of the effective practice into routine care (Grimshaw et al., 2001; Nadeem, Gleacher, & Beidas, 2013). Second, while External Facilitators identified organizational barriers and facilitators to LG-CC implementation, interviews provided insufficient detail to map LG-CC clinical components into clinic workflows without a more comprehensive observation of routine practice (Powell et al., 2012; Baskerville, Liddy, & Hogg, 2012). Third, other implementation studies incorporate principals of quality improvement that promote improved clinic workflow, information system integration, and outcomes benchmarking (e.g., Aarons, Hurlburt, & Horwitz, 2011; Knox et al., 2011; Pina et al., 2014; Solberg, 2007; Wandersman et al., 2008; Chambers, Glasgow, & Stange, 2013; Michie, Fixsen, Grimshaw, & Eccles, 2009).

In light of these findings and relevance of prior implementation studies involving multicomponent models, there are limitations to this study that warrant consideration. The limited number of sites precluded a comprehensive assessment of the impact of Enhanced REP components on more comprehensive provider-level LG-CC fidelity measures and subsequent participant outcomes. We were unable to assess further dimensions of LG-CC fidelity beyond participant attendance, including quality of the sessions received or whether consumers understood the content of the sessions (Bellg, Borrelli, Resnick, et al., 2004; Perepletchikova, Treat, & Kazdin, 2007; Schoenwald et al., 2011). Moreover, we only included participant-level measures of attendance in the analyses, in order to assess whether differences in participant-level outcomes were explained by differences in receipt of LG-CC core components at the individual level. In addition, Enhanced REP Internal Facilitators may not have been able to fully integrate LG-CC into routine workflows to enhance its sustainability, in part because of variations in their expertise or leadership capacity. Moreover, LG-CC providers were relatively highly trained (e.g., masters level) who had full-time effort to devote to the program, and thus did not represent typical mental health providers, especially from smaller, lower-resourced practices (e.g., rural settings). Finally, we were unable to assess whether Enhanced REP impacted other care processes, as sites did not have consistent utilization data (e.g., from electronic medical records) that captured ongoing utilization or quality of care over time.

Overall, the Enhanced REP implementation strategy compared to standard REP did not lead to changes in participant-level outcomes over a two-year period. Further research is needed to determine how specific implementation strategies can support the integration of multicomponent care models, and whether they ultimately lead to improved individual-level clinical outcomes over time. More comprehensive studies of implementation strategies are also needed to promote sustained use of effective practices across multiple treatment settings.

Figure 1. CONSORT Diagram: Recovery-Oriented Collaborative Care Study.