Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Sep 1.
Published in final edited form as: Adm Policy Ment Health. 2015 Sep;42(5):642–653. doi: 10.1007/s10488-014-0598-5

Long-Term Clinical Outcomes from a Randomized Controlled Trial of Two Implementation Strategies to Promote Collaborative Care Attendance in Community Practices

Amy M Kilbourne 1, David E Goodrich 1, Kristina M Nord 1, Celeste Van Poppelen 1, Julia Kyle 1, Mark S Bauer 2, Jeanette A Waxmonsky 3, Zongshan Lai 1, Hyungjin M Kim 1, Daniel Eisenberg 4, Marshall R Thomas 3
PMCID: PMC4400210  NIHMSID: NIHMS634206  PMID: 25315181

Abstract

This randomized controlled implementation study compared the effectiveness of a standard versus Enhanced version of the Replicating Effective Programs (REP) implementation strategy to improve the uptake of the Life Goals-Collaborative Care Model (LG-CC) for bipolar disorder. Seven community-based practices (384 patient participants) were randomized to standard (manual/training) or Enhanced REP (customized manual/training/Facilitation) to promote LG-CC implementation. Participants from Enhanced REP sites had no significant changes in primary outcomes (improved quality of life, reduced functioning or mood symptoms) by 24 months. Further research is needed to determine whether implementation strategies can lead to sustained, improved participant outcomes in addition to program uptake.

Keywords: Mood disorders- bipolar, implementation, chronic care model, collaborative care

Introduction

It can take years if not decades for effective treatments or practices to be implemented into community-based settings (Proctor et al., 2009). This research-to-practice gap profoundly impacts access to effective treatments for persons with mental health conditions (Proctor et al., 2009), who are disproportionately burdened by functional impairment and premature mortality (Druss et al., 2008; Gale et al., 2012; Sharfstein, 2011). Collaborative care models (CCMs) are evidence-based practices shown to improve medical and psychiatric outcomes for persons with chronic mental disorders (Woltmann et al., 2012). CCMs are typically implemented by a clinical nurse or social worker (“care manager”) who provides individual self-management support, panel management, ongoing outcomes assessment, and linkages to specialists in collaboration with the individual's primary care or mental health physician.

However, CCMs have not been widely disseminated in community-based practices (Kilbourne et al., 2004), in part because of their multicomponent nature, limited access to specific, user-friendly tools for providers to implement CCM core components, and lack of ongoing provider support to address barriers to CCM adoption (Coleman, Austin, Brach, & Wagner, 2009; Katon, 2012). Without additional implementation support, providers are unlikely to sustain the CCM in their practice.

A number of implementation frameworks have been proposed to understand the barriers and facilitators to the uptake of multicomponent practices such as the CCM (Chinman, Imm, & Wandersman, 2004; Damschroder et al., 2009; Rubenstein et al., 2002; Solberg et al., 2001; Stetler, Damschroder, Helfrich, & Hagedorn, 2011). These frameworks primarily focus on identifying factors at the organizational level that can facilitate or impede adoption of effective practices. However, few implementation frameworks have been operationalized into specific strategies to help mitigate organizational barriers and help support providers in the implementation of effective practices in their clinical setting. Implementation strategies comprise a set of specific organizational (i.e., provider or clinic-level) interventions meant to help providers use and sustain effective clinical practices in routine care, and can include features such as toolkits or guides for using the effective clinical practice, training, as well as coaching or ongoing feedback on clinical practice use (Gaglio & Glasgow, 2012).

Replicating Effective Programs (REP), based on the Centers for Disease Control and Prevention Division of HIV/AIDS Prevention's Research-to-Practice Framework, is an implementation strategy based on Rogers' Diffusion of Innovations (Rogers, 2003) and Social Learning Theories (Bandura, 1977). REP specifically outlines how effective programs can be implemented, notably through their “packaging” into user-friendly components which are supported by structured training and brief technical assistance for providers. In a randomized controlled implementation trial, REP was shown to improve the uptake of HIV behavioral change interventions (Kelly et al., 2000). However, it is unclear whether REP can result in a public health impact, notably improved patient outcomes over time, for multicomponent treatments such as CCMs.

Based on a recent review of the implementation literature and feedback from community-based practices (Kilbourne, Neumann, Pincus, Bauer, & Stall, 2007; Kilbourne, Neumann, et al., 2012), Enhanced REP was developed as an extension of the original REP framework to promote the use of CCMs for mental health conditions. While the primary “target population” of Enhanced REP is still the principal provider responsible for implementing the CCM, they appeared to need additional guidance in integrating the CCM into their clinical practice. To this end, Enhanced REP added an implementation intervention called Facilitation, which is based on the Promoting Action on Research Implementation in Health Services (PARiHS) framework (Rycroft-Malone, 2004; Stetler et al., 2006). As described in previous research (Kirchner, Ritchie, Dollar, Gundlach, & Smith, 2013), Facilitation involves two distinct roles: an External and Internal Facilitator, both of which support providers in using multicomponent clinical practices (i.e., CCMs). External Facilitators represent a core group of experts typically from the study team who provide technical expertise in CCM implementation. External Facilitators coach the provider in customizing the CCM to meet specific clinical workflow needs in their practice and provide guidance in addressing organizational barriers to implementation. In contrast, Internal Facilitators are employees of the clinical practice (e.g., administrators or clinical managers) who are identified by clinical leadership as the “go-to” person for managing and making changes to clinic practice workflows. Internal Facilitators are not seen as direct advocates of the CCM but rather can help the care manager integrate CCM components into routine care through their internal knowledge of the clinic administration (Kirchner et al., 2014; Kirchner et al., 2013).

We report findings from the Recovery-Oriented Collaborative Care (ROCC) study, a cluster randomized controlled trial comparing the effects of standard versus Enhanced REP (REP+ External and Internal Facilitation) on improved clinical outcome sat the individual participant level when applied to help providers implement a CCM. The CCM used in the study is Life Goals Collaborative Care- LG-CC, an evidence-based practice for persons with bipolar disorder (Waxmonsky et al., 2014). Bipolar disorder was chosen because it is a chronic illness characterized by recurring manic and depressive symptoms and is associated with high suicide rates (Novick, Swartz, & Frank, 2010), functional impairment (Huxley & Baldessarini, 2007), and high health care costs (Dean, Gerner, & Gerner, 2004). LG-CC has been shown in several randomized controlled trials to improve outcomes for individuals with bipolar disorder (Bauer et al., 2006b; Kilbourne, Goodrich, et al., 2012; Kilbourne et al., 2013; Kilbourne et al., 2008; Simon, Ludman, Bauer, Unutzer, & Operskalski, 2006). Yet LG-CC has not been fully translated into community-based practices. The primary hypothesis is that participants from Enhanced versus standard REP sites will experience improved health-related quality of life, decreased functional impairment, or decreased mood symptoms.

Methods

Seven community-based mental health and primary care clinics (sites) were randomized to receive either standard REP or Enhanced REP to facilitate implementation of Life Goals Collaborative Care (LG-CC). Participating sites were initially selected from a total of nine clinics represented by three large organizations from Michigan and Colorado. Eligible sites included mental health or primary care outpatient facilities that cared for at least 200 individuals with bipolar disorder, and had at least two providers with mental health treatment experience available to implement LG-CC.

Clinical Practice to be Implemented (LG-CC)

LG-CC for bipolar disorder (Table 1) was chosen in part because community-based providers expressed a need for EBPs for this illness and because LG-CC has been shown to improve outcomes among individuals with bipolar disorder, particularly among minority groups and individuals with co-occurring conditions (Kilbourne, Neumann, et al., 2012). LG-CC includes four sessions focused on individual self-management, ongoing individual and provider contacts to reinforce self-management skills, provision of care management, guideline support for providers, and a registry tool to track individual participant progress (Table 1).

Table 1. Life Goals Collaborative Care Core Components.

LG-CC Component LG-CC Core Element
Self-management Sessions
  • Four group sessions led by care manager that involved active discussions focused on conditions/counseling approaches

  • Discussions with patients to enhance/maintain physical and mental health goals and symptom recognition

  • Discussions with patients on cost-benefit of symptoms management and provider engagement

Care Management
  • Maintenance of lessons from self-management sessions learned via ongoing contacts with the care manager

  • Regular communication by the care manager with providers including mental health providers and general medical physicians on patient clinical status

  • Outreach/crisis management after critical service encounters

  • Regular documentation of patient clinical status and provider contact, including summary of clinical status and wellness goals

Provider Support
  • Mental health and general medical providers were given information on relevant bipolar disorder treatment guidelines

Implementation Strategy Randomization

The study analyst randomized community-based sites stratified by state (MI, CO) to receive either the Enhanced or standard Replicating Effective Programs (REP) implementation strategy to support the uptake of LG-CC (Table 1).

Standard and Enhanced REP Implementation Strategies

Timelines for standard and Enhanced REP components are presented in Table 2. Standard REP components, delivered to a designated provider at each site by the study investigator team, included the LG-CC manual (AK, JW, JK), LG-CC training (AK, JK, MB), and brief, as-needed technical assistance (JW, JK). Enhanced REP components, also delivered by the same study personnel, included customization of the LG-CC manual and training program based on input from local site providers (AK, JW, JK), and External Facilitation (AK, JW). Specifically, study investigators obtained input from providers and clinical leadership at each site randomized to receive Enhanced REP in customizing the LG-CC package and training program to fit their local needs. Using previously described methods in obtaining input in customizing CCMs (Kilbourne, Neumann, Pincus, Bauer, & Stall, 2007; Kilbourne, Neumann, et al., 2012; Bauer et al., 2006a; Simon, Ludman, Unutzer, & Bauer, 2002), clinical managers from each Enhanced REP site completed a needs assessment of their current site's priorities and barriers and facilitators to LG-CC implementation and provided input via face-to-face interviews on customizing the standard LG-CC package and training program. For the face-to-face interviews, study investigators used as now ball purposeful sampling method (Palinkas, Horwitz, Green, Widsom, Duan, and Hoagwood, 2013) to select providers from the Enhanced REP sites by job type. The site administrator was initially contacted and interviewed, and a snowball sampling method was used to identify and interview at least two additional master's level clinical social workers, at least two additional bachelor's level case managers, and at least one physician, physician, or nurse practitioner. Providers were interviewed to obtain additional recommendations in customizing the LG-CC package/training program. At Enhanced REP sites, study investigators (AK, JK, JW) also held a focus group of 6-8 consumers recruited from each Enhanced REP sites to obtain additional input on customizing the LG-CC package components.

Table 2. Enhanced and Standard REP Intervention Components.

Standard REP Enhanced REP
Phase and Time Period Standard REP Components Total Time Spent during Period & Across Standard REP Sites Enhanced REP Components Total Time Spent during Period & Across Enhanced REP Sites
Package (6 Month Period)
  • Non-customized LG-CC manual development

TOTAL: 840 minutes
  • LG-CC manuals customized to sites using data from organizational needs assessment on population and treatment setting resources

  • Site manager and provider interviews, consumer focus group provided further input into customization

  • Customized Manual:1470 minutes

  • Interviews/focus groups: 930 minutes


TOTAL: 2400 minutes
Training (3 Month Period)
  • Standard LG-CC training

TOTAL: 480 minutes
  • Customized training for providers based on input on LG-CC package

  • Training customization: 1660 minutes

  • Training: 480 hours


TOTAL: 2140 minutes
Facilitation (12 Month Period)
  • As-needed technical assistance (TA) if practice contacts study team for 6 months

  • Monitor LG-CC uptake via monthly reporting sheets: 12 months

TA: 1470 minutes

Monitoring: 500 minutes

TOTAL: 1970 minutes
External Facilitation (EF) Components:
  • Initiation: EF identifies IF

  • Provider Contact: EF works with IF and LG-CC providers to review potential barriers and set measureable goals in LG-CC uptake: monthly1-hour meetings for 12 months

  • Technical Assistance: EF makes calls to IF and LG-CC providers, giving specific guidance on implementation: 30-minute calls/mon

  • Ongoing marketing and Sustainability: EF supports public recognition of LG-CC success stories and helps IF and LG-CC provider to summarize progress and develop long-term plans for sustainability


Internal Facilitation (IF) Components:
  • IF meetings with EF, LG-CC providers to establish measureable goals in LG-CC uptake, identify how LG-CC is aligned with practice priorities, and publicize LG-CC to leadership (monthly 1-hour meetings)

  • Additional brief consultations with LG-CC provider on average 15 minutes/month)

External Facilitator time across all Enhanced REP sites:
  • Components 1-3: 3740 minutes

  • Component 4: 640 minutes



Internal Facilitator time across all Enhanced REP sites: 3600 minutes

TOTAL: 7980 minutes

Providers delivering LG-CC at Enhanced REP sites were also given ongoing facilitation for 12 months (Table 2). External Facilitators who were part of the study team and were external to the clinical practice (AK, JW, JK) provided guidance to practices in implementing LG-CC, identified potential barriers to implementation based on the previous needs assessment, and helped the providers obtain support for the LG-CC program at the site. In addition, External Facilitators with input from site program leadership identified an Internal Facilitator at the beginning of initiating Facilitation. The Internal Facilitator had to be an employee and in a managerial role at the site with a direct reporting line to leadership, but was not responsible for directly delivering LG-CC. The Internal Facilitator from each site met monthly with the LG-CC care manager and External Facilitator to support the care manager in addressing specific organizational barriers to LG-CC implementation. When available, the Internal Facilitator, External Facilitators and LG-CC provider also met with practice leadership on a regular basis. Enhanced REP Internal and External Facilitators also worked with the LG-CC provider to regularly report on implementation progress, identify opportunities to leverage LG-CC implementation to address other site quality improvement goals, and develop a plan for sustaining LG-CC beyond the study (Table 2).

The total time for each standard and Enhanced REP component was recorded by study research assistants using a standardized form that was previously established for similar LG-CC implementation studies (Kilbourne, Goodrich, et al., 2012; Kilbourne et al., 2013; Kilbourne et al., 2008; Simon, Ludman, Bauer, Unutzer, & Operskalski, 2006). Total time spent on Facilitation activities by each Internal Facilitator and LG-CC care manager was recorded on a monthly basis by general area of activity and tallied across all sites randomized to standard REP and forall sites randomized to Enhanced REP (Table 2)

Procedures

One care manager at each site was jointly hired by research and clinical staff to implement LG-CC components (Table 1). All LG-CC care managers had a clinical social work background (MSW) and one to five years of mental health clinical background and previous experience in addressing suicidal ideation, severe manic episodes, and other emergent issues. LG-CC care managers identified and enrolled individuals with an active diagnosis or treatment plan for bipolar disorder based on medical record reviews. Adult individuals receiving care at the site were eligible if they had a diagnosis of bipolar disorder, received outpatient care from the participating practices, and were community-dwelling; and were excluded if they: 1) had evidence of intoxication or active suicidal ideation at enrollment precluding participation in LG-CC components as indicated by the provider or 2) were not able to provide informed consent.

Eligible individuals were then approached in person and enrolled after providing informed consent to participate. Once consented, participants completed a baseline assessment survey, and were scheduled by the care managers to attend LG-CC group sessions. Participants were asked to attend four weekly group sessions over a 4 week-period, followed by monthly individual care management phone calls over 6 months. Care managers recorded participant attendance to LG-CC sessions and subsequent contacts, health behavior goals, and clinical status using the LG-CC electronic registry. Each participant received $5 in remuneration for each group session attended to cover transportation costs, and $20 for the baseline survey.

Data Collection

The primary aim of the study was to compare Enhanced versus standard REP used to implement LG-CC on changes in individual participant-level outcomes. A prior analysis from the same study focused on the impact of Enhanced versus standard REP on fidelity, as measured by frequency of client attendance at group sessions and number of care manager contacts to LG-CC (Waxmonsky et al., 2014). After study enrollment, participants completed a 30-minute survey in person or by phone at baseline and 6, 12, and 24 months later. The primary outcome for this intervention included mood symptoms, health-related quality of life, and functioning. Mood symptoms were ascertained from the Patient Health Questionnaire 9-item survey (PHQ-9) for depressive symptoms (Kroenke, Spitzer, & Williams, 2001; Spitzer, Kroenke, & Williams, 1999) and the Internal State Scale (ISS) (Bauer, Vojta, Kinosian, Altshuler, & Glick, 2000; Glick, McBride, & Bauer, 2003) for manic symptoms. The PHQ-9 has been validated across diverse populations to both screen for depression and monitor response to treatment. The questionnaire has also demonstrated strong test-retest reliability (r(ICC) = .81 to .96) in populations with chronic illnesses and responsiveness to change over time, such that a change of 5 points on the 27 scale constitutes clinically meaningful change in depressive symptoms (Lowe, Unutzer, Callahan, Perkins, & Kroenke, 2004; Spitzer et al., 1999). Depression remission was also calculated based on the PHQ-9, defined as having a 50% reduction in PHQ-9 score and PHQ-9 score of less than 5 at 24 months (Kroenke, Spitzer, Williams, & Lowe, 2010).

Manic symptoms were measured using the Internal State Scale (ISS), a 15-item self-report measure designed to provide a mood state self-report and level of symptomatology. The ISS activation subscale has been used to assess level of manic symptoms based populations with chronic illnesses (Bauer et al., 2000; Glick et al., 2003). Health-related quality of life was assessed using the Short Form 12-item survey to assess individual self-reported health, physical and psychological symptoms, and limitations in activities of day living due to physical and mental health impairments experienced during the prior four weeks (SF-12) (Ware, Kosinski, & Keller, 1996). The SF-12 generates a mental and physical health component score (MCS, PCS) that are standardized to the general population (range: 0-100 each scale). Original psychometric evaluation found 2 week test-retest reliability with correlations ranging from 0.76 to 0.89, and criterion validity scores for identifying physical and mental health conditions ranged from 0.43-0.93 (median=0.67) for PCS and 0.60-1.07 (median=0.97) for MCS (Ware et al., 1996). Functional disability was assessed using the World Health Organization assessment disability scale (WHO-DAS-2) (Ustun & Chisholm, 2001) which measures impairment in activities in daily living across the subscales of cognition, mobility, self-care, social functioning, and role functioning. The scale has strong discriminant validity and construct validity in identifying first episodes of depression (Luciano, Ayuso-Mateos, Fernandez, Aguado, et al., 2010) and good internal consistency (Chronbach α=.89) (Luciano, Ayuso-Mateos, Fernandez, Serrano-Blanco, et al., 2010) while demonstrating sensitivity to change in multi-condition CCM interventions (Von Korff et al., 2011).

Analyses

The target participant sample size was 350 (70 participants per site). Assuming an interclass correlation of <.01 this sample size enables adequate power to detect significant effects in changes in our symptom and quality of life outcomes (Cohen's D=.30). Intent-to-treat analyses using multivariable linear mixed models were applied to compare changes in 6, 12, and 24-month outcomes among participants from Enhanced versus standard REP sites. Multivariable logistic regression was also used to compare the odds of participants from Enhanced versus standard REP sites on depression remission. Multivariable models included an indicator for Enhanced REP (versus standard REP), time (6, 12, 24-month), the interaction of Enhanced REP and time.

The models also controlled for participants' baseline demographic and clinical characteristics as well as baseline values of the outcome measures that were found in our previous study to influence changes in outcomes (Waxmonsky et al., 2014), as well as changes in long-term outcomes in other research trials (i.e., 24-month) results (Harpole et al., 2005; Unutzer et al., 2001). Individual covariates included age, gender, race (non-white versus white), college education (vs. no), unemployed (vs. employed, student, or retired), living alone (vs. living with someone), lifetime history of homelessness (vs. no), hazardous drinking, and self-reported number of comorbid illnesses. Hazardous drinking was defined based on the AUDIT-C question on binge drinking (having consumed 6 or more drinks on a single occasion within the past month) (Dawson, Grant, & Stinson, 2005). Number of comorbid illnesses was ascertained from participant self-report and based on the question, “Has the doctor ever told you that you have one or more of the following:” with check boxes for hypertension/high blood pressure, arthritis/chronic pain, angina/coronary heart disease, heart attack/MI, depression, PTSD, diabetes/high blood sugar, or high cholesterol or parents with high cholesterol.

An exploratory analysis was also conducted to determine whether changes in participant outcomes might be explained by differences in attendance to the LG-CC intervention, defined as the total dose, or number of group sessions and care management contacts combined received by each participant. Described previously (Waxmonsky et al., 2014) the specific fidelity construct used was intervention attendance, or total frequency of LG-CC group sessions and number of care manager contacts completed by the LG-CC participant (Bellg et al., 2004). Participant-level total number of sessions and contacts was then added to each regression multivariable model.

Results

Out of 2,019 individuals initially screened for study participation, 384 were eligible for study participation. After participants were enrolled and consented, 320 started LG-CC at the standard REP (n=143) and Enhanced REP (n=177) practices. Participants from standard REP sites were less likely to have a college education or more likely to be unemployed; otherwise, there were no other demographic or clinical differences among those from Enhanced or standard REP sites (Table 3). The mean age among individual participants was 42 years and 66.7% were female.

Table 3. Characteristics of Patient Characteristics and Baseline Outcome Measures: Overall and Stratified by Enhanced versus Standard REP Sites.

Overall
N=384
Enhanced REP
(n=221)
Standard REP
(n=163)
Demographic Characteristics Mean (SD) Mean (SD) Mean (SD)
Age, years 42.0 (11.3) 42.2 (11.4) 41.8 (11.3)
Number of medical comorbidities 2.6 (1.1) 2.6 (1.0) 2.6 (1.1)
N (%) N (%) N (%)
Female 256 (66.7) 146 (66.1) 110(67.5)
Non-White 108 (29.3) 54 (25.2) 54(34.8)*
College Education or greater (yes, no) 71 (18.8) 59 (27.1) 12(7.5)**
Unemployed (yes, no) 279 (72.7) 149 (67.4) 130(79.8)
Alcohol misuse 40 (10.7) 24 (11.2) 16 (10.0)
Illicit drug use 123 (32.0) 70 (31.7) 53 (32.5)
Clinical Characteristics Mean(SD) Mean(SD) Mean(SD)
Baseline Depressive Symptoms (PHQ-9)(a) 12.9 (6.4) 13.4 (6.7) 12.4 (5.9)
Baseline Manic Symptoms: ISS(b) 20.3 (12.7) 19.3 (12.8) 21.6 (12.4)
Baseline Health-related Quality of Life Scores (SF-12) MCS(c) 31.8 (8.4) 31.2 (8.3) 32.6 (8.4)
Baseline Health-related Quality of Life Scores (SF-12) PCS(c) 36.5 (7.5) 37.2 (7.4) 35.6 (7.5)
a

Depressive symptom scores were based on the Patient Health Questionnaire (PHQ-9); a higher score reflects more symptoms for these three measures.

b

Manic symptom scores were based on the Internal State Scale (ISS), possible scores range from 0 to 50 and a higher score indicates worse manic symptoms.

c

Health-related quality of life (SF-12) includes a mental component scale (MCS) and physical component scale (PCS). Possible scores range from 0 to 100, with higher scores indicating better health. For both summary scores, the population M ± SD is 50 ± 10.

d

WHO-DAS: World Health Organization Disability Adjustment Scale is a scale measuring social and physical dysfunction, with a range of 0-48 and higher scores indicating greater impaired functioning

There were no significant differences inPHQ-9, ISS, WHO-DAS, or SF-12 scores at baseline between participants from Enhanced versus standard REP sites (all P>.05, see Table 3). Reliability (Cronbach Coefficient scores) for the PHQ-9, ISS, and WHO-DAS from the results were respectively .86, .84, and .88.

Overall, study staff spent a total of 3,290 minutes, or 54 hours across the three Standard REP sites, and 7,980 minutes, or 133 hours across the four Enhanced REP sites over the 21-month implementation intervention period. The greater amount of time spent in Enhanced REP activities was primarily attributed to External Facilitation (e.g., coaching the LG-CC care managers) and meetings with Internal Facilitators (Table 2).

After adjusting for participant demographic and clinical factors, there were no significant differences in outcomes among participants from Enhanced versus standard REP sites (Table 4). The addition of LG-CC attendance as a potential explanatory factor also revealed similar results (Table 4).

Table 4. Twenty-Four Month Outcomes Comparing Patients from Enhanced REP versus Standard REP Sites Adjusting for Patient Factors and LG-CC Attendance: Multivariable Results with and without Addition of LG-CC Attendanceǂ.

Depressive Symptoms (PHQ-9) (a)
Beta (95% CI)
Depression remission (PHQ 9 <5 and 50% reduction)
OR (95% CI)
Manic Symptoms:
ISS (b)
Beta (95% CI)
SF-12 Mental Health (MCS) Score (c)
Beta (95% CI)
SF-12 Physical Health (PCS) Score (c)
Beta (95% CI)
Functional Disability:
WHO-DAS (d)
Beta (95% CI)
Without
LG-CC Attendance
Adding
LG-CC Attendance
Without
LG-CC Attendance
Adding
LG-CC Attendance
Without
LG-CC Attendance
Adding
LG-CC Attendance
Without
LG-CC Attendance
Adding
LG-CC Attendance
Without
LG-CC Attendance
Adding
LG-CC Attendance
Without
LG-CC Attendance
Adding
LG-CC Attendance
Enhanced REP (ref: standard REP) 0.5
(-0.4, 1.4)
0.6
(-0.5, 1.6)
1.4
(0.6, 3.0)
1.5
(0.6, 3.4)
-0.04
(-1.9, 1.8)
0.5
(-1.6, 2.6)
-0.3
(-1.6, 1.1)
-0.2
(-1.7, 1.3)
0.5
(-0.5, 1.6)
0.6
(-0.6, 1.9)
-0.2
(-1.6, 1.1)
-0.1
(-1.6, 1.4)
Time -0.1
(-0.1, -0.01)*
-0.1
(-0.1, -0.004)*
-- -- -0.1
(-0.2, -0.001)*
-0.1
(-0.2, -0.004)*
0.03
(-0.1, 0.1)
0.03
(-0.1, 0.1)
0.04
(-0.03, 0.1)
0.04
(-0.02, 0.1)
-0.04
(-0.1, 0.03)
-0.1
(-0.1, 0.02)
Enhanced REPX Time -0.1
(-0.1, 0.01)
-0.1
(-0.1, 0.02)
-- -- -0.03
(-0.2, 0.1)
-0.02
(-0.2, 0.1)
0.1
(-0.03, 0.2)
0.1
(-0.05, 0.2)
-0.1
(-0.1, 0.03)
-0.1
(-0.1, 0.03)
-0.05
(-0.1, 0.05)
-0.03
(-0.1, 0.1)
LG-CC Attendance -- 0.01
(-0.1, 0.2)
-- 0.9
(0.8, 1.2)
-- -0.2
(-0.5, 0.1)
-- -0.05
(-0.3, 0.2)
-- -0.005
(-0.2, 0.2)
-- -0.1
(-0.3, 0.1)
Outcome at baseline 0.7
(0.7, 0.8)*
0.7
(0.6, 0.8)*
0.9
(0.8, 0.9)*
0.9
(0.8, 0.9)*
0.6
(0.6, 0.7)*
0.6
(0.5, 0.7)*
0.5
(0.5, 0.6)*
0.5
(0.4, 0.6)*
0.7
(0.7, 0.8)*
0.7
(0.7, 0.8)*
0.8
(0.7, 0.8)*
0.8
(0.7, 0.8)*
Age -0.01
(-0.04, 0.03)
-0.004
(-0.04, 0.03)
1.0
(0.9, 1.0)
1.0
(0.9, 1.0)
-0.03
(-0.09, 0.04)
-0.02
(-0.1, 0.1)
0.03
(-0.02, 0.1)
0.03
(-0.02, 0.1)
-0.04
(-0.1, -0.01)*
-0.05
(-0.1, -0.04)*
0.1
(0.003, 0.1)*
0.1
(0.01, 0.1)*
Female 0.5
(-0.2, 1.2)
0.5
(-0.3, 1.3)
0.8
(0.4, 1.6)
0.8
(0.3, 1.7)
-0.2
(-1.7, 1.2)
-0.5
(-2.0, 1.0)
-0.6
(-1.6, 0.5)
-0.6
(-1.7, 0.5)
-0.3
(-1.2, 0.5)
-0.3
(-1.2, 0.6)
1.2
(0.2, 2.2)*
1.0
(-0.1, 2.1)
Nonwhite 0.5
(-0.3, 1.2)
0.5
(-0.3, 1.4)
0.5
(0.2, 1.2)
0.5
(0.2, 1.2)
1.3
(-0.3, 2.8)
1.2
(-0.4, 2.8)
0.3
(-0.8, 1.4)
0.3
(-0.8, 1.5)
-0.7
(-1.6, 0.2)
-0.8
(-1.7, 0.2)
0.1
(-0.9, 1.2)
0.1
(-1.0, 1.3)
College education -1.4
(-2.3, -0.5)*
-1.4
(-2.4, -0.5)*
1.2
(0.5, 2.9)
1.2
(0.5, 2.9)
-2.3
(-4.1, -0.5)*
-2.4
(-4.3, -0.5)*
0.2
(-1.1, 1.5)
0.1
(-1.2, 1.5)
1.2
(0.1, 2.3)*
1.2
(0.1, 2.4)*
-1.2
(-2.5, 0.1)
-1.3
(-2.6, 0.1)
Unemployed -0.4
(-1.2, 0.4)
-0.4
(-1.2, 0.4)
1.4
(0.6, 3.2)
1.4
(0.6, 3.2)
0.1
(-1.4, 1.7)
0.1
(-1.6, 1.7)
0.7
(-0.4, 1.9)
0.7
(-0.5, 1.9)
-0.4
(-1.3, 0.5)
-0.4
(-1.4, 0.6)
-0.5
(-1.6, 0.7)
-0.6
(-1.8, 0.5)
Living alone -0.1
(-0.8, 0.7)
-0.1
(-0.9, 0.7)
1.2
(0.6, 2.7)
1.3
(0.6, 2.7)
-0.1
(-1.6, 1.4)
-0.1
(-1.7, 1.5)
0.6
(-0.5, 1.7)
0.7
(-0.5, 1.8)
0.7
(-0.2, 1.6)
0.8
(-0.2, 1.7)
-0.3
(-1.4, 0.8)
-0.4
(-1.5, 0.7)
Homeless History 0.4
(-0.3, 1.1)
0.4
(-0.4, 1.1)
0.7
(0.3, 1.4)
0.7
(0.3, 1.4)
0.7
(-0.7, 2.0)
0.4
(-1.1, 1.9)
-0.5
(-1.5, 0.4)
-0.5
(-1.5, 0.6)
-0.05
(-0.9, 0.8)
-0.1
(-0.9, 0.8)
0.2
(-0.8, 1.1)
-0.1
(-1.1, 0.9)
Hazardous drinking -0.9
(-2.2, 0.2)
-1.2
(-2.5, 0.04)
3.6
(1.1, 11.7)*
3.5
(1.1, 11.6)*
-0.4
(-2.7, 1.9)
-0.6
(-3.2, 1.9)
0.4
(-1.2, 2.1)
0.7
(-1.1, 2.5)
-0.5
(-1.9, 0.8)
-0.7
(-2.2, 0.8)
-1.2
(-2.8, 0.5)
-1.7
(-3.5, 0.1)
# comorbid illnesses 0.1
(-0.2, 0.4)
0.1
(-0.3, 0.4)
1.2
(0.8, 1.7)
1.2
(0.8, 1.7)
-0.3
(-1.0, 0.4)
-0.3
(-1.1, 0.4)
-0.1
(-0.6, 0.4)
-0.1
(-0.6, 0.4)
-0.1
(-0.5, 0.3)
-0.03
(-0.5, 0.4)
-0.2
(-0.7, 0.3)
-0.2
(-0.8, 0.3)
ǂ

In addition to the variables listed in the table, multivariable models with LG-CC attendance included the total number of self-management sessions and care management contacts (range: 0-19). Depression remission is defined as % with >=50% drop in PHQ-9 score and endpoint score of <5.

*

<.05

Discussion

We describe results from a randomized controlled trial comparing the effects of two different implementation strategies on changes in participant-level outcomes for mental disorders. We found that compared to standard REP, Enhanced REP was not associated with improved mental or physical health-related quality of life, reduced functional impairment, manic symptom severity, or greater likelihood of depression remission at 24 months after adjusting for participant factors. While in a previous study we found that Enhanced REP led to improved LG-CC attendance (Waxmonsky et al., 2014), participant outcomes did not appear to be explained by changes to LG-CC attendance in this current study.

To our knowledge, ROCC is one of the first studies assessing whether a specified implementation strategy designed for a CCM for bipolar disorder impacts participant-level outcomes. Specifically, the Enhanced REP facilitation strategy involved active generation of input from providers and consumers in customizing the program, coaching of frontline providers in program implementation by study team members (External Facilitators), and the identification of Internal Facilitators at each site to identify where and how LG-CC can be leveraged to enhance other required services (Rubenstein et al., 2002; Stetler et al., 2011). These Facilitation components have been employed in previous studies, notably in the VA (Kirchner et al., 2014; Kirchner et al., 2013), and have been referred to in other research as Consultation or Coaching (Michie, Fixsen, Grimshaw, & Eccles, 2009). However, few studies have reported on the impact of Facilitation or similar implementation strategies on changes in participant-level outcomes. While typical implementation endpoints involve provider adoption of the program or participant-level use (Curran et al. 2012), benchmarking implementation strategies on measureable participant outcomes can help determine whether any “voltage drop” occurs when programs are rolled out into community settings beyond the typical research setting.

The observed null findings of our study could be attributed to several factors. It is likely that the implementation of LG-CC, which occurred during the first 12 months of the participant observation, did not have lasting effects on participant outcomes within 24 months. Perhaps attendance to LG-CC components, while higher with Enhanced REP (Waxmonsky et al., 2014), may not have increased enough for clinical outcomes to improve over time. Moreover, while the External Facilitator roles were standardized (same study team members across intervention sites), the experience of the Internal Facilitators varied widely across the sites, which may have led to variations in the ability to better integrate LG-CC components into routine practice to achieve maximum effect on participant-level quality and outcomes.

Reliance on a single care manager at each site to lead the implementation of LG-CC without involving other clinicians art the site may also have contributed to a lack of sustained impact of LG-CC over time. Variations in the organizational capacity of the practices varied, in which some sites may not have had organized practice teams with prior experience in implementing CCMs, nor have the resources to identify ways in which LG-CC could be better integrated, notably through changes in provider workflow, incorporation of measurement-based care, or training of other clinicians to employ self-management practices that aligned with LG-CC components.

In addition, evidence from other studies comparing implementation strategies may point to key lessons that can inform the refinement of Enhanced REP in order to maximize its public health impact and sustainability (Glasgow, 2013; Knox et al., 2011). First, focusing implementation efforts on a single provider, even with customized manuals and training is insufficient to influence outcomes without ongoing efforts to integrate components of the effective practice into routine care (Grimshaw et al., 2001; Nadeem, Gleacher, & Beidas, 2013). Second, while External Facilitators identified organizational barriers and facilitators to LG-CC implementation, interviews provided insufficient detail to map LG-CC clinical components into clinic workflows without a more comprehensive observation of routine practice (Powell et al., 2012; Baskerville, Liddy, & Hogg, 2012). Third, other implementation studies incorporate principals of quality improvement that promote improved clinic workflow, information system integration, and outcomes benchmarking (e.g., Aarons, Hurlburt, & Horwitz, 2011; Knox et al., 2011; Pina et al., 2014; Solberg, 2007; Wandersman et al., 2008; Chambers, Glasgow, & Stange, 2013; Michie, Fixsen, Grimshaw, & Eccles, 2009).

In light of these findings and relevance of prior implementation studies involving multicomponent models, there are limitations to this study that warrant consideration. The limited number of sites precluded a comprehensive assessment of the impact of Enhanced REP components on more comprehensive provider-level LG-CC fidelity measures and subsequent participant outcomes. We were unable to assess further dimensions of LG-CC fidelity beyond participant attendance, including quality of the sessions received or whether consumers understood the content of the sessions (Bellg, Borrelli, Resnick, et al., 2004; Perepletchikova, Treat, & Kazdin, 2007; Schoenwald et al., 2011). Moreover, we only included participant-level measures of attendance in the analyses, in order to assess whether differences in participant-level outcomes were explained by differences in receipt of LG-CC core components at the individual level. In addition, Enhanced REP Internal Facilitators may not have been able to fully integrate LG-CC into routine workflows to enhance its sustainability, in part because of variations in their expertise or leadership capacity. Moreover, LG-CC providers were relatively highly trained (e.g., masters level) who had full-time effort to devote to the program, and thus did not represent typical mental health providers, especially from smaller, lower-resourced practices (e.g., rural settings). Finally, we were unable to assess whether Enhanced REP impacted other care processes, as sites did not have consistent utilization data (e.g., from electronic medical records) that captured ongoing utilization or quality of care over time.

Overall, the Enhanced REP implementation strategy compared to standard REP did not lead to changes in participant-level outcomes over a two-year period. Further research is needed to determine how specific implementation strategies can support the integration of multicomponent care models, and whether they ultimately lead to improved individual-level clinical outcomes over time. More comprehensive studies of implementation strategies are also needed to promote sustained use of effective practices across multiple treatment settings.

Figure 1. CONSORT Diagram: Recovery-Oriented Collaborative Care Study.

Figure 1

Acknowledgments

Support for this study was provided by the National Institute of Mental Health (R01 MH 79994) and the University of Michigan Comprehensive Depression Center (Director's Innovation Fund). We would like to express appreciation for the contributions of partners from the following agencies: (from Colorado) Behavioral Healthcare, Inc.; Colorado Access; Community Reach Center; Mental Health Centers of Denver; and University of Colorado School of Medicine Depression Center, Departments of Psychiatry and Family Medicine; (from Michigan) Community Support and Treatment Services; Genesee Health System; Packard Health; University of Michigan Departments of Psychiatry, School of Public Health, and Depression Center; and Washtenaw Community Health Organization; (Others) Centers for Disease Control and Prevention; Harvard Medical School; National Institute of Mental Health; and Veterans Health Administration. The views expressed in this article are those of the authors and do not necessarily represent the views of the Veterans Administration.

References

  1. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bandura A. Social Learning Theory. Englewood Cliffs, NJ: Prentice Hall; 1977. [Google Scholar]
  3. Baskerville NB, Liddy C, Hogg W. Systematic review and meta-analysis of practice facilitation within primary care settings. Annals of Family Medicine. 2012;10(1):63–74. doi: 10.1370/afm.1312. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bauer MS, McBride L, Williford WO, Glick H, Kinosian B, Altshuler L, et al. Cooperative Studies Program 430 Study, T Collaborative care for bipolar disorder: Part I. Intervention and implementation in a randomized effectiveness trial. Psychiatric Services. 2006a;57(7):927–936. doi: 10.1176/ps.2006.57.7.927. [DOI] [PubMed] [Google Scholar]
  5. Bauer MS, McBride L, Williford WO, Glick H, Kinosian B, Altshuler L, et al. Cooperative Studies Program 430 Study, T Collaborative care for bipolar disorder: Part II. Impact on clinical outcome, function, and costs. Psychiatric Services. 2006b;57(7):937–945. doi: 10.1176/ps.2006.57.7.937. [DOI] [PubMed] [Google Scholar]
  6. Bauer MS, Vojta C, Kinosian B, Altshuler L, Glick H. The Internal State Scale: Replication of its discriminating abilities in a multisite, public sector sample. Bipolar Disorders. 2000;2(4):340–346. doi: 10.1034/j.1399-5618.2000.020409.x. [DOI] [PubMed] [Google Scholar]
  7. Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, et al. Treatment Fidelity Workgroup of the NIH Behavior Change Consortium Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology. 2004;23:443–451. doi: 10.1037/0278-6133.23.5.443. [DOI] [PubMed] [Google Scholar]
  8. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science. 2013;8:117. doi: 10.1186/1748-5908-8-117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Chinman M, Imm P, Wandersman A. Getting to Outcomestm 2004: Promoting accountability through methods and tools for planning, implementation, and evaluation. Santa Monica, CA: RAND Corporation; 2004. [Google Scholar]
  10. Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the chronic care model in the new millennium. Health Affairs. 2009;28(1):75–85. doi: 10.1377/hlthaff.28.1.75. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. 2009;4:50. doi: 10.1186/1748-5908-4-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dawson DA, Grant BF, Stinson FS. The AUDIT-C: Screening for alcohol use disorders and risk drinking in the presence of other psychiatric disorders. Comprehensive Psychiatry. 2005;46(6):405–416. doi: 10.1016/j.comppsych.2005.01.006. [DOI] [PubMed] [Google Scholar]
  13. Dean BB, Gerner D, Gerner RH. A systematic review evaluating health-related quality of life, work impairment, and healthcare costs and utilization in bipolar disorder. Current Medical Research and Opinion. 2004;20(2):139–154. doi: 10.1185/030079903125002801. [DOI] [PubMed] [Google Scholar]
  14. Druss BG, Hwang I, Petukhova M, Sampson NA, Wang PS, Kessler RC. Impairment in role functioning in mental and chronic medical disorders in the United States: Results from the national comorbidity survey replication. Molecular Psychiatry. 2008;14(7):728–737. doi: 10.1038/mp.2008.13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Gaglio B, Glasgow RE. Evaluation approaches for dissemination and implementation research. In: Brownson R, Colditz G, Proctor E, editors. Dissemination and implementation research in health: Translating science to practice. New York, NY: Oxford University Press; 2012. pp. 327–356. [Google Scholar]
  16. Gale CR, Batty GD, Osborn DP, Tynelius P, Whitley E, Rasmussen F. Association of mental disorders in early adulthood and later psychiatric hospital admissions and mortality in a cohort study of more than 1 million men. Archives of General Psychiatry. 2012;69(8):823–831. doi: 10.1001/archgenpsychiatry.2011.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Education and Behavior. 2013;40(3):257–265. doi: 10.1177/1090198113486805. [DOI] [PubMed] [Google Scholar]
  18. Glick HA, McBride L, Bauer MS. A manic-depressive symptom self-report in optical scanable format. Bipolar Disorders. 2003;5(5):366–369. doi: 10.1034/j.1399-5618.2003.00043.x. [DOI] [PubMed] [Google Scholar]
  19. Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, Vale L. Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. Journal of General Internal Medicine. 2006;21(Suppl 2):S14–20. doi: 10.1111/j.1525-1497.2006.00357.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. O'Brien MA. Changing provider behavior: An overview of systematic reviews of interventions. Medical Care. 2001;39(8 Suppl 2):II2–45. [PubMed] [Google Scholar]
  21. Harpole LH, Williams JW, Jr, Olsen MK, Stechuchak KM, Oddone E, Callahan CM, et al. Unutzer J. Improving depression outcomes in older adults with comorbid medical illness. General Hospital Psychiatry. 2005;27:4–12. doi: 10.1016/j.genhosppsych.2004.09.004. [DOI] [PubMed] [Google Scholar]
  22. Huxley N, Baldessarini RJ. Disability and its treatment in bipolar disorder patients. Bipolar Disorders. 2007;9:183–196. doi: 10.1111/j.1399-5618.2007.00430.x. [DOI] [PubMed] [Google Scholar]
  23. Katon W. Collaborative depression care models: From development to dissemination. American Journal of Preventive Medicine. 2012;42(5):550–552. doi: 10.1016/j.amepre.2012.01.017. [DOI] [PubMed] [Google Scholar]
  24. Kelly JA, Somlai AM, DiFranceisco WJ, Otto-Salaj LL, McAuliffe TL, Hackl KL, et al. Rompa D. Bridging the gap between the science and service of HIV prevention: Transferring effective research-based HIV prevention interventions to community aids service providers. American Journal of Public Health. 2000;90(7):1082–1088. doi: 10.2105/ajph.90.7.1082. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kilbourne AM, Goodrich DE, Lai Z, Clogston J, Waxmonsky J, Bauer MS. Life Goals Collaborative Care for patients with bipolar disorder and cardiovascular disease risk. Psychiatric Services. 2012;63(12):1234–1238. doi: 10.1176/appi.ps.201100528. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kilbourne AM, Goodrich DE, Lai Z, Post EP, Schumacher K, Nord KM, Bauer MS. Randomized controlled trial to assess reduction of cardiovascular disease risk in patients with bipolar disorder: The Self-Management Addressing Heart Risk Trial (SMAHRT) Journal of Clinical Psychiatry. 2013;74(7):e655–662. doi: 10.4088/JCP.12m08082. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care: Application of the replicating effective programs framework. Implementation Science. 2007;2:42. doi: 10.1186/1748-5908-2-42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kilbourne AM, Neumann MS, Waxmonsky J, Bauer MS, Kim HM, Pincus HA, Thomas M. Public-academic partnerships: Evidence-based implementation: The role of sustained community-based practice and research partnerships. Psychiatric Services. 2012;63(3):205–207. doi: 10.1176/appi.ps.201200032. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Kilbourne AM, Post EP, Nossek A, Drill L, Cooley S, Bauer MS. Improving medical and psychiatric outcomes among individuals with bipolar disorder: A randomized controlled trial. Psychiatric Services. 2008;59(7):760–768. doi: 10.1176/ps.2008.59.7.760. [DOI] [PubMed] [Google Scholar]
  30. Kilbourne AM, Schulberg HC, Post EP, Rollman BL, Belnap BH, Pincus HA. Translating evidence-based depression management services to community-based primary care practices. Milbank Quarterly. 2004;82:631–659. doi: 10.1111/j.0887-378X.2004.00326.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Kirchner JE, Kearney LK, Ritchie MJ, Dollar KM, Swensen AB, Schohn M. Research & services partnerships: Lessons learned through a national partnership between clinical leaders and researchers. Psychiatric Services. 2014;65(5):577–579. doi: 10.1176/appi.ps.201400054. [DOI] [PubMed] [Google Scholar]
  32. Kirchner JE, Ritchie MJ, Dollar KM, Gundlach P, Smith J. Implementation facilitation training manual: Using external and internal faciliation to improve care in the Veterans Health Administration. 2013 Retrieved February 15, 2014, from www.queri.research.va.gov/tools/implementation/Facilitation-Manual.pdf.
  33. Knox L, Taylor EF, Geonnotti K, Machta R, Kim J, Nysenbaum J, Parchman M. Developing and running a primary care practice facilitation program: A how-to-guide. Rockville, MD: Agency for Healthcare Research and Quality; Dec, 2011. (Prepared by Mathematica Policy Research under contract No. HHSA 2902009000191 TO5.). (AHRQ Publication No. 12-011) [Google Scholar]
  34. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine. 2001;16(9):606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Kroenke K, Spitzer RL, Williams JB, Lowe B. The Patient Health Questionnaire somatic, anxiety, and depressive symptom scales: A systematic review. General Hospital Psychiatry. 2010;32(4):345–359. doi: 10.1016/j.genhosppsych.2010.03.006. [DOI] [PubMed] [Google Scholar]
  36. Lowe B, Unutzer J, Callahan CM, Perkins AJ, Kroenke K. Monitoring depression treatment outcomes with the Patient Health Questionnaire-9. Medical Care. 2004;42:1194–1201. doi: 10.1097/00005650-200412000-00006. [DOI] [PubMed] [Google Scholar]
  37. Luciano JV, Ayuso-Mateos JL, Fernandez A, Aguado J, Serrano-Blanco A, Roca M, Haro JM. Utility of the twelve-item World Health Organization Disability Assessment Schedule II (WHO-DAS II) for discriminating depression “caseness” and severity in Spanish primary care patients. Quality of Life Research. 2010;19(1):97–101. doi: 10.1007/s11136-009-9566-z. [DOI] [PubMed] [Google Scholar]
  38. Luciano JV, Ayuso-Mateos JL, Fernandez A, Serrano-Blanco A, Roca M, Haro JM. Psychometric properties of the twelve item World Health Organization Disability Assessment Schedule II (WHO-DAS) in Spanish primary care patients with a first major depressive episode. Journal of Affective Disorders. 2010;121(1-2):52–58. doi: 10.1016/j.jad.2009.05.008. [DOI] [PubMed] [Google Scholar]
  39. Meropol SB, Schiltz NK, Sattar A, Stange KC, Nevar AH, Davey C, et al. Cuttler L. Practice-tailored facilitation to improve pediatric preventive care delivery: A randomized trial. Pediatrics. 2014;133(6):e1664–1675. doi: 10.1542/peds.2013-1578. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: The need for a scientific method. Implementation Science. 2009;4:40. doi: 10.1186/1748-5908-4-40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: Unpacking the black box. Administration and Policy in Mental Health and Mental Health Services Research. 2013;40(6):439–450. doi: 10.1007/s10488-013-0502-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Novick DM, Swartz HA, Frank E. Suicide attempts in bipolar I and bipolar II disorder: A review and meta-analysis of the evidence. Bipolar Disorders. 2010;12(1):1–9. doi: 10.1111/j.1399-5618.2009.00786.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Nutting PA, Crabtree BF, Stewart EE, Miller WL, Palmer RF, Stange KC, Jaen CR. Effect of facilitation on practice outcomes in the National Demonstration Project model of the patient-centered medical home. Annals of Family Medicine. 2010;8(Suppl 1):S33–44. S92. doi: 10.1370/afm.1119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Administration and Policy in Mental Health and Mental Health Services Research. 2013 doi: 10.1007/s10488-013-0528-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Perepletchikova F, Treat TA, Kazdin AE. Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology. 2007;75(6):829–841. doi: 10.1037/0022-006X.75.6.829. [DOI] [PubMed] [Google Scholar]
  46. Pina IL, Cohen PD, Larson DB, Marion LN, Sills MR, Solberg LI, Zerzan J. A framework for describing health care delivery organizations and systems. American Journal of Public Health. 2014:e1–e10. doi: 10.2105/AJPH.2014.301926. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. York JL. A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review. 2012;69(2):123–157. doi: 10.1177/1077558711430690. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research. 2009;36(1):24–34. doi: 10.1007/s10488-008-0197-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Rogers E. Diffusion of Innovations. 5th. New York, NY: Free Press; 2003. [Google Scholar]
  50. Rubenstein LV, Parker LE, Meredith LS, Altschuler A, dePillis E, Hernandez J, Gordon NP. Understanding team-based quality improvement for depression in primary care. Health Services Research. 2002;37(4):1009–1029. doi: 10.1034/j.1600-0560.2002.63.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Rycroft-Malone J. The PARiHS Framework--a framework for guiding the implementation of evidence-based practice. Journal of Nursing Care Quality. 2004;19(4):297–304. doi: 10.1097/00001786-200410000-00002. [DOI] [PubMed] [Google Scholar]
  52. Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):32–43. doi: 10.1007/s10488-010-0321-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Sharfstein SS. Integrated care. American Journal of Psychiatry. 2011;168(11):1134–1135. doi: 10.1176/appi.ajp.2011.11050766. [DOI] [PubMed] [Google Scholar]
  54. Simon GE, Ludman E, Unutzer J, Bauer MS. Design and implementation of a randomized trial evaluating systematic care for bipolar disorder. Bipolar Disorders. 2002;4(4):226–236. doi: 10.1034/j.1399-5618.2002.01190.x. [DOI] [PubMed] [Google Scholar]
  55. Simon GE, Ludman EJ, Bauer MS, Unutzer J, Operskalski B. Long-term effectiveness and cost of a systematic care program for bipolar disorder. Archives of General Psychiatry. 2006;63(5):500–508. doi: 10.1001/archpsyc.63.5.500. [DOI] [PubMed] [Google Scholar]
  56. Solberg LI. Improving medical practice: A conceptual framework. Annals of Family Medicine. 2007;5(3):251–256. doi: 10.1370/afm.666. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Solberg LI, Asche SE, Margolis KL, Whitebird RR. Measuring an organization's ability to manage change: The change process capability questionnaire and its use for improving depression care. American Journal of Medical Quality. 2008;23(3):193–200. doi: 10.1177/1062860608314942. [DOI] [PubMed] [Google Scholar]
  58. Solberg LI, Fischer LR, Wei F, Rush WA, Conboy KS, Davis TF, Heinrich RL. A CQI intervention to change the care of depression: A controlled study. Effective Clinical Practice. 2001;4(6):239–249. [PubMed] [Google Scholar]
  59. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: The PHQ Primary Care Study. Primary care evaluation of mental disorders. Patient Health Questionnaire. Journal of the American Medical Association. 1999;282(18):1737–1744. doi: 10.1001/jama.282.18.1737. [DOI] [PubMed] [Google Scholar]
  60. Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A guide for applying a revised version of the PARiHS Framework for implementation. Implementation Science. 2011;6:99. doi: 10.1186/1748-5908-6-99. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, et al. Wallace CM. Role of “external facilitation” in implementation of research findings: A qualitative evaluation of facilitation experiences in the Veterans Health Administration. Implementation Science. 2006;1:23. doi: 10.1186/1748-5908-1-23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Unutzer J, Rubenstein L, Katon WJ, Tang L, Duan N, Lagomasino IT, Wells KB. Two-year effects of quality improvement programs on medication management for depression. Archives of General Psychiatry. 2001;58(10):935–942. doi: 10.1001/archpsyc.58.10.935. [DOI] [PubMed] [Google Scholar]
  63. Ustun TB, Chisholm D. Global “burden of disease”-study for psychiatric disorders. Psychiatrische Praxis. 2001;28(Suppl 1):S7–11. doi: 10.1055/s-2001-15381. [DOI] [PubMed] [Google Scholar]
  64. Von Korff M, Katon WJ, Lin EH, Ciechanowski P, Peterson D, Ludman EJ, et al. Rutter CM. Functional outcomes of multi-condition collaborative care and successful ageing: Results of randomised trial. BMJ. 2011;343:d6612. doi: 10.1136/bmj.d6612. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Saul J. Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology. 2008;41(3-4):171–181. doi: 10.1007/s10464-008-9174-z. [DOI] [PubMed] [Google Scholar]
  66. Ware J, Jr, Kosinski M, Keller SD. A 12-item Short-Form Health Survey: Construction of scales and preliminary tests of reliability and validity. Medical Care. 1996;34(3):220–233. doi: 10.1097/00005650-199603000-00003. [DOI] [PubMed] [Google Scholar]
  67. Waxmonsky J, Kilbourne AM, Goodrich DE, Nord KM, Lai Z, Laird C, et al. Bauer MS. Enhanced fidelity to treatment for bipolar disorder: Results from a randomized controlled implementation trial. Psychiatric Services. 2014;65(1):81–90. doi: 10.1176/appi.ps.201300039. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. Woltmann E, Grogan-Kaylor A, Perron B, Georges H, Kilbourne AM, Bauer MS. Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: Systematic review and meta-analysis. American Journal of Psychiatry. 2012;169(8):790–804. doi: 10.1176/appi.ajp.2012.11111616. [DOI] [PubMed] [Google Scholar]

RESOURCES