Organizational Learning of Collaborative Care in Washington State’s Mental Health Integration Program (MHIP)

Abstract

Background:

Collaborative Care (CoCM) is an evidence-based model for treating depression in primary care. Little is known about its implementation “learning curve” in non-research settings.

Methods:

The authors used eight years of observational data from Washington State’s Mental Health Integration Program (MHIP) to describe organization-level changes in clinical and process-of-care outcomes over time.

Results:

The authors analyzed data from 13,362 unique patients within eight Community Health Centers (CHCs) in Washington State. On average, clinical outcomes improved for the first two years before improvement slowed, peaking at year five. Significant organization-level variation was noted in outcomes. Improvements in clinical outcomes tended to track process-of-care measures.

Conclusions:

Findings suggest that it may take two years post-implementation of CoCM to fully observe organization-level clinical outcome improvement. Substantial between-organization variation in outcomes over time suggests that sustained attention to processes-of-care and quality may be necessary to maintain initially achieved gains.

Introduction

Collaborative care (CoCM) is an evidence-based model for the management of depression in primary care. In CoCM, behavioral health care managers in primary care practices provide assessments, brief psychosocial interventions and medication management support with backup and consultation from a psychiatric consultant. A CoCM registry is used to track patients’ progress toward evidence-based treatment goals. Recommendations from the CoCM team are delivered to the primary care provider (PCP), who prescribes psychiatric medications when appropriate. Core principles of CoCM include patient-centered team care, population-based care, measurement-based treatment-to-target, the use of evidence-based treatments and accountable care¹. To date, the efficacy of CoCM has been corroborated by more than eighty RCTs^2,3 and it has been used to treat several common mental health disorders in primary care⁴. The implementation of CoCM is typically a significant organizational shift for practices and necessitates practice changes at multiple administrative and clinical levels. Little is known, however, about the “learning curve” associated with implementing this care model across diverse provider organizations. In this investigation, the authors used observational data from more than thirteen thousand patients with clinically significant depression treated at eight community health centers (CHCs) in Washington State that implemented and sustained CoCM over an eight-year period. This research aims to examine organizational variability in the process-of-care and depression treatment outcomes over time.

Methods

The Institutional Review Boards at the University of Washington and Weill Cornell Medical College approved this study, with a waiver of informed consent for individual patients. Data were from Washington State’s Mental Health Integration Program (MHIP), a systematic implementation of collaborative care in nine CHCs in two Western Washington counties (King and Pierce) from January 2008 to March 2015. The authors excluded one of the nine organizations because of insufficient patient enrollment during the study period. All patients with a baseline Patient Health Questionnaire-9 (PHQ-9) score of ten or higher and at least one follow-up PHQ-9 score were included in the analyses.

Depression improvement was defined as achieving a PHQ-9 score of less than ten or a reduction in PHQ-9 score from baseline by 50% or greater (referred to as the “depression” outcome measure) at any point within six months of treatment initiation. The authors also examined the outcome of having at least one follow-up contact with the care manager and PHQ-9 assessment in the month following enrollment (referred to as the “process-of-care” outcome measure). This particular process-of-care metric was selected for two reasons: it could be readily measured with available registry data for every enrolled patient and had strong support from previous literature⁵.

The authors estimated a logistic model of depression improvement at the patient level as a function of the year in which a patient was enrolled (1st, 2nd, …, up to 8th year), a dichotomous indicator for each CHC (to capture any between-CHC differences in patient outcomes that did not change over time) and interactions between years since CHC start in MHIP and CHC indicators (to capture potentially varying changes in depression improvement over time across CHCs). The model controlled for patient demographics (age and gender), behavioral health conditions co-morbid with depression at baseline (including anxiety disorders, bipolar disorder, neurocognitive disorders, psychosis, posttraumatic stress disorder (PTSD), substance use disorders, and assessed suicidal ideation), baseline PHQ-9 score, reasons for MHIP program eligibility and a dichotomous indicator for the presence of a value-based payment program that went into effect in January 2009^6,7. The model yielded yearly CHC-level predicted rates of depression improvement (and confidence intervals) and yearly rates of improvement across all CHCs. A similar logistic model was estimated at the patient level for the process-of-care measure.

Results

Aggregated data from all 13,362 patients within the eight CHCs included in the analyses demonstrated the most rapid improvement in depression outcomes over the first two years of MHIP implementation. Based on the adjusted analysis, overall rate of improvement increased from 39.5% (95% CI=33.0–40.5) to 43.1% (95% CI=40.5–45.8) over that time period. After the first two years, depression improvement continued, albeit at a slower rate, until a peak improvement rate of 45.6 % (95% CI=43.6–47.7) at five years since start of implementation (Figure 1). Between years five and eight, the percentage of patients with depression improvement decreased from the peak of 45.6% to 41.8% (95% CI=35.8–47.7). It is noteworthy that changes in the MHIP-wide mean outcome estimates did not achieve statistical significance, although this investigation was not testing any hypothesis. Tabular data of depression outcomes by CHC are available online.

Figure 1.

Adjusted Percentage of Patients Achieving Depression Improvement by Week 24

CHC-level analyses demonstrated substantial variability in depression outcome improvement over time. Two organizations (G and H) performed above the overall rate for most of the study period, while one (D) remained largely below. The remaining five organizations (A, B, C, E and F) alternated between outcomes above and below the average. Of these five organizations, two (B and E) started considerably below average, but improved between years two and three and remained high-performers for the duration of the study. One organization (A) displayed the opposite trend – starting as a high-performer, but dipping below the average rate between years two and three. The remaining two organizations (C and F) were inconsistent, moving above and below the average rate several times throughout the study period.

Patient-level process-of-care data also demonstrated, on average, an improvement over the first three years of implementation. The adjusted rate of having at least one follow-up contact and PHQ-9 assessment in the month following enrollment increased from 67.4% (95% CI=61.573.3) to a peak of 77.0% (95% CI=75.4–78.5) over that time period (figure available online). After the first three years, however, improvement leveled off considerably. In years seven and eight of the study period, the overall rate declined, ending at 73.0% (95% CI=67.7–78.4) in year eight. CHC-level data on the process-of-care outcome are available online.

Substantial variability across CHCs was also observed with the process-of-care outcomes. One organization (H) was consistently above average throughout the study period, while two remained consistently below (E and F). The remaining five organizations (A, B, C, D and G) demonstrated substantial variability.

The results suggest that MHIP-wide improvements in treatment outcomes generally track improvements in program quality and process-of-care. MHIP-wide curves of improvement for both outcomes demonstrated a similar pattern – initial improvement, subsequent plateau and eventual regression. Time to peak performance notably differed between the two metrics, requiring three years for the process-of-care outcome and five years for the clinical outcome. Both outcome curves showed setbacks in years seven and eight of the study period.

At the individual organization level, there was less observed concordance between the processof-care and depression outcome curves, although a number of similarities were identified. Organization H, for example, was a consistent high-performer in both outcomes for the duration of the study. Additionally, organizations A, B and C demonstrated comparably inconsistent results for both outcomes. One example of discordance was Organization G, which was a consistent high-performer in the treatment outcome, but was variable with regard to the process-of-care outcome.

Discussion

Data from this investigation principally suggest that successful implementation of CoCM may take as many as two to three years, with additional (smaller) gains achieved from years three to five. This is not surprising given the complexity of practice change required to implement CoCM. Additionally, the results demonstrate that it took organizations longer to reach peak performance in the clinical outcome measure than in the process-of-care measure, suggesting that organizations achieve process-of-care gains before realizing maximum clinical outcome improvement. Finally, after initial improvements in program outcomes, all organizations encountered setbacks⁸ when working to sustain CoCM and its related clinical practices in years seven and eight. While the authors do not have definitive data on the challenges to sustaining initial gains, possible explanations include a loss of clinical champions or key program staff over time, competing organizational priorities for quality improvement and a lack of resource investment in program sustainment^9–11. Regardless of the reasons, the data suggest that organizations should carefully evaluate their progress roughly two years after implementing CoCM and consider single-loop (improving efficiency of current practices) or higher order (reevaluation of initial workflow and process goals) organizational learning strategies to optimize performance^12,13.

Of note, the overall depression improvement observed during this study was modest, with less than half of the patients showing substantial clinical improvement. It was encouraging, however, that the aggregate depression outcome improvement rate increased by 6.1 percentage points (or 15.4%) from baseline to peak over the course of the observation period. Although different from randomized controlled trials of CoCM, these findings are consistent with the delivery of CoCM in a group of vulnerable, low-income, disabled or unstably housed individuals such as those served by the eight CHCs in this sample. Similar results have been reported from a recent CoCM implementation in a safety-net health system serving a comparable population in New York City¹⁴.

The results also demonstrate that organizational learning varies considerably for CoCM, although the study’s observational design limited the authors’ capacity to identify specific explanations for this deviation. However, variations in process-of-care outcomes were generally similar to variations in depression outcomes, suggesting that sustained attention to core practices, such as timely follow-up after an initial visit^5–7, is important for program success. This is consistent with previously published literature, which suggests that efficacious dissemination of health services interventions is often impeded by sub-optimal adherence to evidence-based processes-of-care¹⁵. It also supports the use of pay-for-performance incentives for CoCM based on operational quality indicators^6,7.

Limitations of this study include the selection of only one process-of-care variable (rate of timely follow-up), as well as the relatively small number and limited geographic distribution of organizations included in the analysis. These factors attenuate the findings’ generalizability, despite the comparatively large patient-level sample size (greater than 13,000). The small organizational sample size also limits the ability to draw conclusions about organization-level associations between process-of-care and depression outcome variables. Finally, the exclusion of one of the nine participating organizations (due to inadequate patient enrollment) could have introduced bias, as organizations with limited enrollment may have had more difficulty implementing or sustaining CoCM.

Quantitative studies with a large sample of organizations may further elucidate associations between process-of-care and depression outcomes in CoCM. Additionally, qualitative studies could help explain potential factors associated with trends in clinical and process-of-care outcomes, as well as variability across organizations.

Conclusions

Participating CHCs experienced the most rapid improvements in depression treatment outcomes in the initial two years following CoCM implementation. This improvement rate then attenuated until peak performance was reached around year five, after which all observed organizations encountered setbacks. On average, depression outcomes tended to trace process-of-care outcomes. Additionally, there was considerable between-organization variability in depression outcomes over time, suggesting that successful implementation of CoCM requires significant up-front practice change and that an ongoing focus on key processof-care measures (such as patient engagement and follow-up) may be important for organizations to achieve sustained outcome improvement over time.