Implementation of Evidence Based Practices for Complex Mood Disorders in Primary Care Safety Net Clinics

Abstract

Objective

Use quality improvement methods to implement evidence-based practices for Bipolar Depression and Treatment Resistant Depression in six Federally Qualified Health Centers.

Methods

Following qualitative needs assessments, implementation teams comprised of front-line providers, patients, and content experts identified, adapted, and adopted evidence-based practices. With external facilitation, onsite clinical champions lead the deployment of the evidence-based practices. Evaluation data were collected from 104 patients with probable Bipolar Disorder or Treatment Resistant Depression via chart review and an Interactive Voice Response telephone system.

Results

Five practices were implemented: 1) screening for Bipolar Disorder, 2) telepsychiatric consultation, 3) prescribing guidelines, 4) on-line Cognitive Behavioral Therapy, and 5) on-line peer support. Implementation Outcomes: 1) 15% of eligible patients were screened for Bipolar Disorder (inter-clinic range: 3%–70%), 2) few engaged in on-line psychotherapy or peer support, 3) 38% received telepsychiatric consultation (inter-clinic range: 0%–83%), and 4) 64% of patients with a consult were prescribed the recommended medication. Clinical Outcomes: Of those screening at high risk or very high risk, 67% and 69% respectively were diagnosed with Bipolar Disorder. A third (32%) of patients were prescribed a new mood stabilizer and 28% were prescribed a new antidepressant. Clinical response (50% reduction in depression symptoms), was observed in 21% of patients at three-month follow-up.

Discussion

Quality improvement processes resulted in the implementation and evaluation of five detection and treatment processes. Though varying by site, screening improved detection and a substantial number of patients received consultations and medication adjustments, however symptom improvement was modest.

INTRODUCTION

Most patients with Bipolar Disorder (BD) are not engaged in specialty mental health care despite the clinical complexity and risk of self-harm. (Wang, Lane, et al., 2005) The prevalence of BD is relatively high in primary care settings, (Cerimele, Chwastiak, Dodson, & Katon, 2014) where a quarter of those patients who screen positive for depression or are prescribed antidepressants also screen positive for BD. (Hirschfeld, Cass, Holt, & Carlson, 2005; Olfson et al., 2005) Accurate recognition of BD occurs infrequently in this setting, (Das et al., 2005) and even among those eventually diagnosed, an average of 6–8 years elapses between onset and diagnosis. (Lish, Dime-Meenan, Whybrow, Price, & Hirschfeld, 1994; Wang, Berglund, et al., 2005) Detection is complicated by the relative infrequency of manic episodes compared to depressive episodes, and because primary care patients typically present during a depressive or mixed episode. (Hirschfeld, 2001; Strakowski, 2007). Because the most common misdiagnosis for BD is unipolar depression, (Hirschfeld, Lewis, & Vornik, 2003) there is substantial risk that patients with unrecognized BD will receive antidepressant monotherapy, (Das et al., 2005) which practice guidelines advise against because of the risk of inducing a manic episode. (Pacchiarotti et al., 2013) Yet, antidepressant monotherapy is common in this setting, (Ghaemi et al., 2006) with 71%–73% of patients with BD receiving inappropriate medication; another 18% receiving no psychotropic medication. (Merikangas et al., 2007; Olfson et al.) Given the negative consequences of misdiagnosing BD as depression, the importance of detecting and treating BD in primary care is increasingly being recognized. (Cerimele, Chwastiak, Chan, Harrison, & Unutzer, 2013; Kilbourne et al., 2010; Kilbourne, Goodrich, O’Donnell, & Miller, 2012; Lewis, 2004; Manning, Ahmed, McGuire, & Hay, 2002; Swann et al., 2005)

Federally Qualified Health Centers (FQHCs) provide services to 26 million patients living in rural and inner city areas, (HRSA, 2017) where they represent the “de facto” mental health system for disadvantaged populations. (Fox, Merwin, & Blank, 1995; Geller & Muus, 1997) Mental health prevalence rates are higher in FQHCs than private practice. (Burke et al., 2013) Evidence-based practices (EBPs) for BD are typically designed and tested in large specialty/multi-specialty clinics and are not feasible to implement in FQHCs that typically lack on-site psychiatrists or psychologists. Moreover, the findings from randomized trials that constitute the evidence base for BD treatments are not necessarily generalizable to FQHCs serving disadvantaged populations. (Bowden et al., 2012) Local practice-based evidence about the effectiveness of adapted EBPs is expected to foster sustainability of adoption. (Hohmann & Shear, 2002)

We report here on the results of an Academic Community Implementation Partnership that focused on the identification, adaptation, implementation and evaluation of EBPs for BD. (Hunt et al., 2012) The partnership was between the University of Arkansas for Medical Sciences, the Community Health Centers of Arkansas, and six Arkansas FQHCs (Table 1). The partnership Steering Committee guided implementation and evaluation efforts. BD was chosen by health care providers from participating FQHCs based on their difficulties managing patients with this disorder. The six FQHCs exhibited considerable variation in organizational culture and climate. (Glisson et al., 2008; Kramer, 2017)

Table 1.

Characteristics of Participating Federally Qualified Health Center (FQHC) Organizations.

FQHC Organization	# Clinic Locations*	# of Providers	# of Patients	% >= Age 18	% Caucasian	% African American	% Hispanic Ethnicity	% Racial and/or Ethnically Minority	% Patients at or below 100% of Poverty level1	% Uninsured	% Medicaid	% Medicare
A	5	9.95	9,664	89.9%	42.7%	56.5%	2.6%	58.5%	65.7%	49.3%	17.8%	11.9%
B	7	14.51	14,967	75.8%	98.5%	0.5%	4.9%	6.1%	46.3%	24.2%	22.1%	20.6%
C	4	4.6	3,701	90.8%	20.4%	78.5%	1.6%	79.9%	72.6%	52.5%	14.8%	5.5%
D	29	34.75	39,835	75.1%	78.3%	20.2%	3.2%	24.0%	71.1%	41.7%	24.9%	14.3%
E	6	7.65	7,529	83.0%	41.3%	55.8%	3.9%	60.7%	81.8%	45.9%	22.1%	13.4%
F	7	11.33	9,351	85.2%	26.8%	71.6%	3.2%	74.2%	74.1%	53.6%	21.7%	8.2%

^*Only one site from each organization participated in this work.

Source – U.S. Health Resources & Services Administration (HRSA) Uniform Data System.

Quality improvement efforts within individual or small numbers of practices are often used to bring about changes in care delivery, but with varying success. Implementing significant practice changes (i.e., practice transformation) requires sustained investment in time and resources. (Crabtree et al., 2010) A wide range of implementation strategies designed to promote the adoption of EBPs can be used. (Powell et al., 2015) These implementation efforts generally involve a team of individuals representing the distinct areas affected by the change (e.g., nursing, information technology) working with a clinical “champion” who serves to direct and support the work of the team. Implementation facilitators external to the organization can also work with practices on problem identification and resolution, and to provide support and encouragement. (Nutting et al., 2010; Stetler et al., 2006) Another critical component is the identification of metrics to assess progress and guide course corrections. (Berwick, 1998; Young, Roberts, & Holden, 2017) The iterative nature of the quality improvement processes allows for tailoring the EBP to the needs and capacities of particular settings.

METHODS

Before initiating the implementation of EBPs for BD each of the participating FQHCs implemented the evidence based telepsychiatry collaborative care model for depression. (Fortney et al., 2013) University researchers provided technical assistance, including care manager training and access to a web-based decision support system. (Fortney et al., 2010) The telepsychiatry collaborative care team included on-site primary care providers (PCPs) and care managers, and an off-site consulting telepsychiatrist. The care manager monitored symptoms and adherence, and encouraged patients to engage in self-management activities. The telepsychiatrist and care managers conducted case reviews of patients not engaging or responding to treatment. The telepsychiatrist provided treatment recommendations to the PCPs who prescribed psychotropic medications. FQHCs received funding for a half-time depression care manager. This depression collaborative care program was the foundation for the BD EBP implementation described below. Many FQHCs have successfully implemented collaborative care for depression. (Bauer et al., 2011; Meredith et al., 2006; Unutzer et al., 2012)

Implementation Strategies

We used an external facilitation method known as Evidence-Based Quality Improvement (EBQI). (Rubenstein, Mittman, Yano, & Mulrow, 2000; Rubenstein et al., 2002) In EBQI, both researchers and clinic staff participate fully in the quality improvement process, with the researchers facilitating rather than dictating implementation efforts. (Parker, dePillis, Altschuler, Rubenstein, & Meredith, 2007; Rubenstein et al., 2002) While emphasizing the involvement of outside experts and empirical evidence, EBQI stresses that a healthcare organization’s own staff are best positioned to improve care. Local staff contributed the knowledge needed to tailor the EBP for their own particular needs and organizational capabilities. Researchers contributed knowledge about the EBPs and implementation best practices.

The EBQI process (see Figure 1) began with a needs assessment that involved administering a brief anonymous survey to PCPs asking them about their experience and comfort diagnosing and prescribing for depression and BD. Qualitative interviews were also conducted with providers and patients. Initial discussions indicated that PCPs had difficulty distinguishing between BD and Treatment Resistant unipolar Depression (TRD), so both types of disorders were targeted by the EBQI process. Interviews focused on screening, diagnosing, and managing BD and TRD. Barriers and facilitators to managing these disorders were assessed along with clinical processes that were perceived to need changing. Interviews were audio-recorded and notes taken. Relying primarily on the notes (using audio recordings for clarification), rapid content analysis was conducted to make results available quickly to the EBQI team. (Sobo, 2005; Sobo, Simmes, Landsverk, & Kurtin, 2003)

Figure 1.

Evidence Based Quality Improvement Process

The EBQI team included a clinician from each FQHC, two FQHC patients with BD, two researchers, and two clinical content experts. The EBQI team had seven conference calls in three months. Topics discussed included: needs assessment findings, screeners for BD, screener workflow, defining and identifying TRD, telepsychiatric consultation, medication provider guidelines and patient handouts, internet-delivered psychotherapy, on-line peer support, and implementation strategies. The role of the content experts and researchers was to ensure that chosen practices were evidence-based and that adaptations did not eliminate core treatment elements. FQHC clinicians were responsible for tailoring the EBP to meet the needs, preferences and resources of their clinic, and minimize staff burden. Patients were responsible for ensuring that adapted EBPs were acceptable. A qualitative researcher monitored the calls for participation, and provided feedback about equity to the researchers.

Once the EBPs were chosen and adapted, an implementation template was developed by the EBQI Team. For each EBP, this template specified: 1) the clinical objective, 2) action steps, 3) staff responsibilities, 4) how success is evaluated, 5) resources available and needed), 6) communications plan, and 7) potential barriers and solutions. Each FQHC organization chose one clinic for implementation and an implementation champion, who was a senior clinician in a leadership position. Researchers provided external facilitation.

Evaluation

The evaluation focused on the RE-AIM constructs of reach, adoption, implementation fidelity and effectiveness. (Glasgow, Vogt, & Boles, 1999) To develop internal capacity among FQHCs to conduct their own evaluations, we mentored FQHC staff to collect their own data. The goal of the evaluation was to provide FQHCs with local evidence that the adapted EBPs were effective. Because we did not want to burden FQHC staff with data collection activities, we used a telephone-based Interactive Voice Response System (IVR) to call patients directly and collect patient-reported outcomes. IVR-administered psychiatric assessments are reliable and valid. (Mundt et al., 2006) The system included a web-portal for FQHC staff to add contact information for consenting patients and download survey data in spreadsheet format. The IVR system required patients to call a toll free number and enter an ID to verify authentication. Three months after completing the baseline assessment, patients received up to three texts or audio messages reminding them to call back into the IVR system and complete the follow-up assessment.

At baseline, the IVR system asked about demographics, depression symptoms using the PHQ-9, (Kroenke, Spitzer, & Williams, 2001) and manic symptoms using the 5-item Altman Mania Rating Scale modified for telephone administration. (Altman, Hedeker, Peterson, & Davis, 1997) The three-month follow-up assessed clinical outcomes (medication adherence, depression and mania symptoms) and implementation success (use of EBPs). Care managers conducted a brief chart review to measure clinical outcomes (BD screening results, diagnosis), and implementation success (receipt of care management, telepsychiatric consultation, PCP prescribed recommended medications).

The FQHCs submitted a Federal Wide Assurance to the Office of Human Research Protection to have the University of Arkansas for Medical Sciences Institutional Review Board provide oversight, which approved the research.

RESULTS

Needs Assessment

The needs assessment survey was administered to 18 PCPs (75% response rate). For depression, 100% of PCPs reported diagnosing and prescribing, and felt “very comfortable” doing so. For BD, 55% of PCPs reported diagnosing and 61% reported prescribing, and those PCPs felt “moderately comfortable” doing so. Qualitative interviews with eight PCPs and six Nursing Directors indicated consistent annual depression screening using the PHQ-9, but inconsistent methods for detecting or differentiating between TRD and BD. Providers believed that both BD and TRD frequently went undetected and were supportive of systematic screening, though views varied about how to screen for BD and the definition of TRD. Many PCPs felt uncomfortable prescribing medications for BD and most patients with a suspected diagnosis of BD or TRD were referred to a local community mental health center. However, referrals were often unsuccessful and PCPs felt obligated but unprepared to treat these patients. Many PCPs were dissatisfied with the communication from their local community mental health center providers. Despite never co-managing their patients with a psychiatrist, PCPs expressed a preference for co-management and were open to telepsychiatric consultation.

Qualitative interviews with seven patients identified misdiagnosis as a common experience, with BD often being first diagnosed during a psychiatric hospitalization. Patients reported being referred to community mental health centers, but lack of insurance, long wait times and stigma prevented them from engaging in care. Most patients reported receiving all their mental health treatment in the FQHC, and despite support from care managers, were not satisfied with their care. Patients reported wishing their PCPs had more mental health training and spent more time listening to them. Strong preferences for counseling and peer support were common themes. Receiving mental health services in the FQHC setting via interactive video was reported as being acceptable.

Evidence Based Practices and Adaptations

The EBQI team chose five EBPs: 1) screening for BD, 2) telepsychiatric consultation, 3) mood stabilizer prescribing guidelines, 4) internet-delivered Cognitive Behavioral Therapy and 5) on-line peer support. Two hours of Continuing Medical Education credit was provided to FQHC staff via webinar. The first educational session focused on diagnosing and prescribing for BD and TRD. The second session focused on telepsychiatric consultation, Cognitive Behavioral Therapy, and peer support.

The EBQI team chose to screen for BD using the Composite International Diagnostic Interview because it is a staged screener which reduces administration burden. (Kessler et al., 2006) The EBQI team considered whether to screen all patients, or increasing smaller subgroups of patients with higher prevalence rates of BD (e.g., patients screening positive for depression, patients diagnosed with depression (Gill, Chen, Grimes, & Klinkman, 2012), patients prescribed antidepressants, patients failing two antidepressant trials). Screening just those at high risk for BD lowers screening burden and minimizes false positives, but may delay or prevent recognition for some patients. To balance these competing concerns, the EBQI team decided to screen patients for BD after a positive annual depression screen. We contracted with the electronic health record vendor to develop a screening template that automatically launched after a positive annual PHQ-9 screen. TRD was defined as two failed adequate antidepressant trials in which medication adherence and treatment non-response (<50% decrease in PHQ-9 scores) was verified by the care manager over an 8-week period. Patients meeting BD screening and TRD criteria were recruited for the evaluation.

A Telepsychiatric Consultation Referral and Medication Algorithm were developed (see Figure 2). The university installed interactive video equipment in the FQHCs and offered free telepsychiatric consultations. A telepsychiatric consultation protocol was developed that provided instructions for eligibility, scheduling, consenting, information sharing, rooming, and establishing the audio-video connection. One FQHC moved into a new building at the beginning of the implementation period and their interactive video equipment was not functional during the duration of the evaluation.

Figure 2. Acute Bipolar Depression.

Referral and Medication Algorithm for Primary Care Providers

The EBQI team chose five mood stabilizers for the BD prescribing guidelines based on efficacy, side effects, cost, PCP acceptability, and monitoring burden (see Figure 3). The guidelines specified dosing instructions, advantages/disadvantages, side effects, drug interactions and recommended lab tests. In addition, two-page patient educational handouts were developed for each medication.

Figure 3.

Medications Guidelines for Acute Bipolar Depression

Based on patients’ requests for counseling and peer support, two technology-assisted interventions were chosen. The first was Beating the Blues®, a self-administered internet-delivered Cognitive Behavioral Therapy program. Care managers helped the patients set up email accounts (required for login ID), manage passwords, and access the program. In addition, care managers encouraged patients to engage in an on-line peer support program operated by the Depression and Bipolar Support Alliance. Because most patients lived in rural areas with high stigma levels and perceived lack of treatment anonymity, being able to access these services via the internet was considered an advantage.

Screening

Implementation fidelity

Outcomes are reported for five of the six FQHCs because one site screened for BD after every positive PHQ-9 rather than per-protocol (i.e., only if the annual PHQ-9 was positive). During the six-month implementation period, there were 2,234 positive annual PHQ-9 screens, and 15% (n=324) were administered the BD screen. Two of the five FQHCs implemented the CIDI screen with good fidelity (70% and 64% completed after a positive annual PHQ-9), one FHQC implemented with poor fidelity (25%) and two FQHCs implemented with very poor fidelity (3% and 11%). Of all the completed BD screens, 40% (131/324) were classified as high or very high risk.

Effectiveness

The BD screener performed reasonably well. Of those screened (n=80), patients with higher risk were significantly (p<0.01) more likely to be chart diagnosed with BD either before or after screening: 17% of very low risk, 33% of low risk, 50% of moderate risk, 67% of high risk and 69% of very high risk. Because most patients did not have an encounter with the consulting telepsychiatrist for formal diagnosis, the diagnoses cannot be used to accurately assess screening sensitivity/specificity.

Patient Characteristics

Table 2 reports the characteristics of the patients (n=104) with BD and TRD who consented to participate in the evaluation. According to chart review, only 14% of the sample had a baseline (pre-screening) diagnosis of BD. About half (47%, n=49) were considered high or very high risk according to the BD screener. About a quarter (23%, n=24) were not screened for BD, and another 29% (n=30) were screened, but screened negative. Among those not screened or screening negative, 10 (10/54=19%) were either diagnosed with BD at baseline or received a new diagnosis during their three-month follow up. Summing the screen positives and the diagnosed, 57% (n=59) of the sample had probable BD. The remaining 43% (n=45) had TRD. Two thirds (62%, n=64) of the sample completed the three-month follow-up IVR survey. The vast majority (88%) reported they were prescribed psychotropic medications and 73% of those reported they “never missed a dose” or “missed only a couple of times” in the past two weeks.

Table 2.

Demographic and Clinical Characteristics of Patient Sample

Patient Characteristic (n=104)	Mean (SD)/Percent (Num/Denom)
Age	38.4 (11.7)
Female	62.5% (65/104)
Race/Ethnicity
Caucasian	55.8% (58/104)
African American	36.5% (38/104)
Other	7.7% (8/104)
High School Graduate	63.5% (66/104)
Employed	19.2% (20/104)
Baseline PHQ-9	16.8 (5.8)
CIDI Risk (n=80)
Very High	16.3% (13/80)
High	46.3% (37/80)
Moderate	5.0% (4/80)
Low	3.8% (3/80)
Very Low*	28.8% (23/80)
CIDI Missing	23.1% (24/104)
Baseline Bipolar Diagnosis	13.5% (14/104)
Baseline Mood Stabilizer	10.6% (11/104)

^*Includes those screening out in the first two stages.

Diagnosis and Treatment

Reach and Adoption

Seventy five percent of the sample had at least one care manager encounter. About a third (38%) of the sample had a telepsychiatric consultation. Two clinics successfully completed telepsychiatric consultations for the majority of their patients (73% and 83%), while half the clinics had no consultations. Overall, 28% received a new diagnosis of BD. Among patients with a telepsychiatric consultation (n=39), the PCP usually (64%) prescribed the recommended medication. Overall, 24% (n=25) of the sample were prescribed a medication recommended by the consulting telepsychiatrist; 32% were prescribed a new mood stabilizer, and 28% were prescribed a new antidepressant. Only 13% of those completing the survey reported using Beating the Blues® during the previous three months, with users completing 1.3 (s.d.=0.9) sessions on average (Table 3). Even fewer (6%) attended an on-line peer support group, with users attending 2.5 (s.d.=1.7) groups on average.

Table 3.

Implementation and Clinical Outcomes

Chart review N=104	Overall	Clinic A	Clinic B	Clinic C	Clinic D	Clinic E	Clinic F
	Percent
New Bipolar Diagnosis	27.9%	32.0%	36.4%	30.4%	36.8%	8.7%	33.3%
New Mood Stabilizer	31.7%	16.0%	36.4%	34.8%	52.6%	30.4%	0%
New Antidepressant	27.9%	16.0%	18.2%	39.1%	42.1%	21.7%	33.3%
Care Manager Encounter	75.0%	100.0%	81.8%	100.0%	100.0%	0.0%	66.7%
Telepsychiatric Consult	37.5%	0.0%	0.0%	73.9%	15.8%	82.6%	0.0%
Recommended Medication Prescribed	24.0%	0.0%	0.0%	60.9%	10.5%	39.1%	0.0%
IVR N=64	Overall	Clinic A	Clinic B	Clinic C	Clinic D	Clinic E	Clinic F
	Percent
Beating The Blues®	12.5%	0.0%	0.0%	17.7%	14.3%	25.0%	0.0%
Online Support Group	6.3%	7.1%	0.0%	5.9%	0.0%	16.7%	0.0%
≥50% ↓ PHQ-9	21.2%	31.3%	16.7%	17.7%	14.3%	25.0%	0.0%
	Mean(SD)
Δ PHQ9	−3.34 (6.48)	−3.50 (8.85)	−4.33 (3.83)	−3.11 (6.45)	−3.57 (4.48)	−3.17 (6.85)	2.00 (NA)
Δ AMRS	0.68 (3.50)	0.56 (4.51)	0.50 (2.88)	1.76 (3.73)	0.36 (2.76)	−0.42 (2.81)	3.00 (NA)

NA – Not Applicable

PHQ9 – Patient Health Questionnaire (depression severity)

AMRS – Altman Mania Rating Scale (mania severity)

IVR – Interactive Voice Response

Clinical Effectiveness

Baseline PHQ-9 depression scores were moderately severe (μ=17.7, s.d.=5.8) and there was a modest (μ=−3.3, s.d.= 6.5) decrease by 12 weeks. A clinical response (≥50% decrease in PHQ-9) was observed in 21% of patients (Table 3). The baseline mania scores were low (μ=4.2, s.d.=2.7) indicating that on average patients experienced four mania symptoms “occasionally” in the past week. At 12 weeks, there was little change (μ=0.7, s.d.=3.5) in mania symptoms.

Discussion

This manuscript reports the implementation and clinical outcomes of an EBQI effort conducted by six FQHCs, their primary care association and a state medical school. Five EBPs for detecting and treating BD/TRD were identified and adapted for FQHCs. To evaluate implementation success, this effort also included developing an internal capacity for FQHCs to assess clinical outcomes.

BD screening was implemented with variable success across sites, with one third of FQHCs adopting the screening protocol with good fidelity. The screener performed well, with two thirds of patients with positive screens being diagnosed with BD. Telepsychiatric consultation was also implemented with variable success across sites, with one third of FQHCs successfully completing a consultation for the majority of their patients. On-line psychotherapy and peer support reached few patients. Alternative approaches to delivering evidence-based psychotherapy and peer support are likely needed for FQHC populations.

Even with substantial resources and sustained support many efforts to implement EBPs fail. (Crabtree et al., 2010) In the current study, a number of common implementation strategies were used including a needs assessment, the use of a multiple-stakeholder team to identify and adapt EBPs, local clinical champions, implementation checklist, and external facilitation. Collectively, these implementation strategies represent a pragmatic approach to practice transformation in primary care settings. Wide variation in implementation outcomes is commonly observed following real-life quality improvement and is consistent with the widely shared experience that practice change is difficult in safety net settings. (Meredith et al., 2006)

Based on our inconsistent implementation outcomes, we recommend several potential modifications to our approach. First, we could have provided more coaching to the local champion at FQHCs, especially at sites with lower organizational culture and climate scores. There is good evidence that external facilitators can coach internal facilitators to promote the adoption of complex interventions at healthcare systems facing implementation barriers. (Kirchner JE, 2014) Second, we could have used pay-for-performance to encourage adoption. This implementation strategy has been shown to be effective at improving the fidelity of collaborative care in FQHC settings. (Unutzer et al., 2012) It also may be that some FHQCs were just not ready to implement these EBPs. One FQHC fully met (and two more almost met) Glisson’s criteria for “worst organizational culture,” (Kramer, 2017) and two of these FQHCs did not successfully adopt any of the EBPs. One highly resource intensive approach would have been to conduct a pre-implementation intervention to improve organizational culture at these FQHCs. (Aarons, Ehrhart, Farahnak, & Hurlburt, 2015; Glisson, Dukes, & Green, 2006)

There are few benchmarks with which to compare the clinical effectiveness observed in these safety net primary care clinics. While 32% of patients were prescribed a new mood stabilizer and another 28% were prescribed a new antidepressant, only 21% experienced a 50% reduction in depression symptoms by 12 weeks. In the STEP-BD medication trial, 35% of patients with BD experienced a 50% reduction by 16 weeks. (Sachs et al., 2007) However, the STEP-BD trial enrolled a much higher income and less diverse population, and patients were prescribed medications by psychiatrists in specialty mood clinics at academic medical centers and had access to evidence based psychotherapy. There has only been one other BD collaborative care study conducted in the FQHC setting. (Cerimele, Chan, et al., 2014) This observational study found that 33% of patients experienced a 50% reduction in depressive symptoms. However, those patients were enrolled in collaborative care for 30 weeks on average, which is substantially longer than our 12-week follow-up. Thus, while the response rate observed in this study was relatively low, this could be due to differences in study characteristics.

Being practice-based and pragmatic, this research had both strengths and limitations. The EBPs were selected by and adapted for the FQHCs, and thus were more likely to meet their needs. Where successfully implemented, screening and telepsychiatric consultation were sustained after the evaluation. The EBQI process also generated practice-based evidence which helped the FQHCs identify which EBPs were and were not effective. Limitations included lack of a control group, short follow-up period, limited casemix adjustment, and small sample size. However, from the perspective of the FQHCs, the lack of internal validity is outweighed by the high generalizability of this practice-based evidence to their own patients.