Enhanced Fidelity to a Psychosocial Treatment for Bipolar Disorder: Results from a Randomized Controlled Implementation Trial

Abstract

Background

We determined whether application of a novel implementation intervention (Enhanced Replicating Effective Programs-REP) versus its standard, dissemination-focused version (REP) improved fidelity to bipolar disorder treatment (Life Goals Collaborative Care- LGCC).

Methods

Five community practices from Michigan and Colorado were randomized to receive LGCC using Enhanced or standard REP. One provider at each practice implemented LGCC which included patient self-management support (4 group sessions focused on symptoms and behavior goals), guideline dissemination to providers, and ongoing phone care management focused on maintaining behavior goals and provider engagement. Standard REP included intervention packaging (i.e., translation of LGCC core components into user-friendly language), training, and as-needed technical assistance. Enhanced REP added customization of LGCC and ongoing, proactive technical assistance through an internal and external facilitator that focused on enhancing provider buy-in and uptake. Multiple and logistic regression analyses determined the impact on patient-level LGCC fidelity between Enhanced versus standard REP.

Results

Participants (N=384; mean age = 42 years, 67% women, 29% nonwhite) averaged 3.0 out of 4 LGCC group sessions and had 4.0 care management contacts. Enhanced REP implementation was associated with 2.6 (p<.001) greater total number of sessions/contacts than standard REP, which was driven by 2.5 (p<.01) more care management contacts, after adjusting for patient factors. Women and those with a history of homelessness received fewer sessions.

Conclusions

Enhanced REP implementation was associated with improved LGCC fidelity, primarily for care management contacts. Additional customization of interventions such as LGCC may be needed to ensure adequate treatment fidelity for key vulnerable populations.

INTRODUCTION

Translating effective behavioral treatment models from research to routine practice is a public health priority articulated by the National Institutes of Health. A number of psychosocial interventions have been shown to improve treatment adherence, quality of care, and health outcomes for persons with mental illnesses (1, 2). However, few of these evidence-based practices have been successfully translated into community-based practices. Even when delivered, treatment fidelity is suboptimal, especially in community-based practices with more limited resources than traditional academic settings. Treatment fidelity is a process to ensure that the core treatment components of an intervention are delivered as intended (3, 4). Fidelity can falter over time due to a lack of tools to support frontline providers in overcoming barriers to implementation as well as lack of provider buy-in (5–7).

Hence, there is a growing body of literature focused on developing implementation interventions that can assist frontline providers in maintaining treatment fidelity. The Centers for Disease Control and Prevention’s (CDC’s) Replicating Effective Programs (REP) is a previously established implementation strategy used to translate effective HIV prevention strategies into community-based organizations (8, 9). REP focuses on strategies that maximize fidelity to treatments through the development of treatment manual “packages” that are supported by provider training and limited technical assistance (10, 11). REP focuses on the steps required to initially implement new programs (package, training), but, it may not fully support ongoing fidelity or sustainability of these practices, notably by encouraging provider buy-in and organizational support.

There are few studies in the literature that compare implementation strategies designed to enhance treatment fidelity. Based on a community-academic partnership described elsewhere (12–14) study investigators enhanced REP by adding Facilitation, a collaborative process between expert consultants and local stakeholders to help implement and sustain new programs (15, 16). Facilitation is based on the Promoting Action on Research Implementation in Health Services (PARiHS) framework and Facilitators help providers address organizational barriers (e.g. clinical flow issues, staff turnover), and foster ongoing relationships with organizational leadership to promote program buy-in (14, 15, 17).

This paper examines 6-month treatment fidelity outcomes from the Recovery-Oriented Collaborative Care study, a randomized controlled trial comparing the effectiveness of Enhanced REP versus standard REP delivered at community-based practices implementing the Life Goals Collaborative Care (LGCC) treatment. The current study employs a hybrid effectiveness-implementation model that focuses on testing two implementation strategies while observing and gathering data on clinical outcomes (i.e., comparing standard REP to Enhanced REP) and allowing for the examination of both clinical and fidelity measures (18). LGCC is an evidence-based, manualized psychosocial intervention for individuals with bipolar disorder (19–23) based on the Chronic Care Model (24, 25). Specifically, we hypothesize that Enhanced REP increases treatment fidelity to LGCC implementation. We also examine patient factors associated with LGCC treatment fidelity.

METHODS

The Recovery-Oriented Collaborative Care study is described elsewhere (15, 26). In brief, community-based practices caring for patients diagnosed with bipolar disorder from Michigan or Colorado were enrolled between March 2010 and December 2011 and randomized to receive Enhanced versus standard REP to help implement LGCC. This study was reviewed and approved by local institutional review boards.

Study Population and Treatment

Five community-based clinical practices agreed to participate; two from Colorado and three from Michigan. These practices served the majority of individuals with mental disorders in their respective regions and they represented a diverse patient population. Eligible practices cared for at least 200 unique patients with bipolar disorder and had at least one mental health provider (e.g., social worker, nurse, or psychologist) available to implement LGCC.

Randomization was stratified by state, and practices in each state were randomized to one of the implementation interventions. Three practices received Enhanced REP implementation intervention and two received standard REP implementation intervention, to support the uptake of LGCC at the practices. All practices were asked to implement LGCC, an evidence-based psychosocial treatment program which has been found in five randomized controlled trials to improve outcomes for patients with bipolar disorder including physical and mental health-related quality of life and symptom reduction (19–23). Core components of LGCC were derived from the Collaborative Care Model (24, 25) and described in Table 1.

Table 1.

Core Components of Life Goals Collaborative Care and Fidelity Measures

	Components	Fidelity Measures
Self-management support	Four group sessions lasting 90–120 minutes Session 1 focus points: Self-management & Collaborative Care Understanding Bipolar Disorder, Mood Monitoring, and Stigma Impact of Core Values and Self-Efficacy on Health Habits Bipolar Disorder and its Impact on Physical Health Session 2 focus points: Working Through Mania Identifying Personal Triggers of (Hypo) Mania Responding to Manic Symptoms- Cost Benefit Analysis Develop a Personal Action Plan for (Hypo) Mania Session 3 focus points Recognizing the Symptoms of Depression Identifying Personal Triggers of Depression Responding to Depressive Symptoms- Cost Benefit Analysis Develop a Personal Action Plan for Depression Session 4 focus points Personal Wellness Change Plan Building and Strengthening a Collaborative Relationship Relapse Prevention- Community Resources Each session focused on having patients identify a health behavior goal from one of the following areas: Nutrition Physical activity Stress reduction Medications Sleep	Study Design -Theory-based model previously tested -A prior plan for number, length, and frequency of contacts -Plan to monitor consistency of dose and fidelity to protocol Training -Hired mental health clinicians from local regions -Standardized training curriculum -Booster trainings and supervision based on direct observations of providers Treatment Delivery -Used treatment manual as well as session scripts and outlines -Used LGCC Session Fidelity checklista to directly observe degree focus points were covered and occurrence of nonspecific factors Treatment receipt: -Number of self-management sessions completed Treatment Enactment -Progress on self-management goals measured in care management phase -Number of care management contacts made by health specialist to patient and providers -Monitor length of calls, missed calls, and completion of goals in registry -Fidelity monitoring not available
Care management	Phone contacts lasting 15–20 minutes covering the following: Health behavior goal progress using motivational enhancement techniques Symptom self-management Review of upcoming medical and psychiatric appointments and reminding patients to attend Relaying urgent medical or psychiatric concerns to providers
Provider guideline dissemination	Frontline providers at practices received an educational in-service focused on bipolar disorder, including impact on physical health, followed by dissemination of materials related to the linkage of the Life Goals self-management program goals to bipolar disorder treatment guidelines as outlined by the American Psychiatric Association (26)

^aThe LGCC Session Fidelity Checklist (29, 33) is used by an independent rater who performs direct observations of all four self-management sessions randomly selected from each health specialist’s cohorts of participants. The first section allow raters to record session length, number of participants, location, potential sources of delivery problems, and the overall degree to which the LGCC session’s educational objectives were met on a five-point scale (0=not met, 1=somewhat met, 2=partly met, 3=mostly met, and 4=fully met). Section II of the checklist rates nonspecific session characteristics about Interventionist Characteristics (e.g., came prepared and organized, elicited clarification of participant understanding of didactic content) and Process Goals (e.g., health specialist facilitated participant discussion and interaction, avoided judgmental feedback, displayed empathy) on a three-point scale of whether an objective occurred (0=disagree; 1=somewhat agree; 2=fully agree). Section III of the checklist rates Content Fidelity - the degree in which focus points were covered for a specific LGCC self-management section based scripted session outlines, using the same three-point scale from Section II.

Implementation Intervention (Enhanced and Standard REP)

Providers at practices receiving standard REP (26) were given a package of LGCC that included an outline, treatment manual and implementation guide that were not customized to the practice, a standard training program, and as-needed technical assistance (Table 2). All practices had administrative buy in to participate in the study prior to randomization to implementation strategy.

Table 2.

Enhanced and Standard Replication Effective Programs Implementation Intervention Components Received by Practices Randomized to Enhanced or Standard REP

	Standard REP	Enhanced REP
Package (Manual)	Non-customized LGCC manual	LGCC manuals customized to practices using data from organizational needs assessment on population and treatment setting resources
Training	Standard LGCC training	Customized training for providers
Facilitation	As-needed (passive) technical assistance if practice contacts study team Monitor LGCC uptake via monthly reporting sheets	External Facilitation Steps: Initiation: External facilitator (EF) identifies an internal facilitator (IF) at each practice Provider Contact: EF works with IF and LGCC providers to set measureable goals in LGCC uptake Active Technical Assistance: EF makes structured calls to IF and LGCC practice providers, giving specific guidance on implementation LGCC components Ongoing marketing and Sustainability: EF supports public recognition of LGCC success stories and helps IF and LGCC provider to summarize progress and develop long-term plans for sustainability Internal Facilitation Steps: Initiation: IF meets with EF, LGCC providers, and practice leadership to establish measureable goals in LGCC uptake Relationship-Building: IF identifies practice priorities per leadership input and identifies other LGCC program champions Benchmarking and Ongoing Rapport: IF measures progress and continues to develop rapport with practice leadership Cultural Adaptation: IF uses knowledge of local practice culture to facilitate LGCC, addressing potential barriers, and aligning LGCC goals with practice priorities Marketing and Sustainability: IF works with EF and LG providers to summarize progress to leadership and develop a business and training to ensure LGCC continuity.

Enhanced REP included steps to further adapt LGCC to the local practices based on provider and consumer input, as well as Facilitation provided by a study staff member for 6 months to enhance provider buy-in and uptake of LGCC over time (Table 2). We chose to provide facilitation for six months, balancing the potential cost of the implementation intervention with the desire for a more intensive intervention than REP (9). The first step was a needs assessment conducted at practices randomized to receive Enhanced REP to identify the community-based practice’s current priorities, and initial organizational barriers and facilitators to implementing LGCC. Step 2 included customization of the LGCC package and a training program for each practice based on feedback from the needs assessment. For example, consumer workbooks were tailored to reflect local community resources and group meeting frequency was limited to four sessions to minimize travel burden. The next three steps involved Facilitation support (Table 2). Two study staff members based in Michigan and Colorado served as External Facilitators who provided technical assistance to practices in implementing LGCC. Each Enhanced REP practice also identified an Internal Facilitator, who was an employee at the practice with direct reporting line to leadership. Internal Facilitators did not provide direct LGCC services but instead, assisted providers to implement LGCC by addressing organizational barriers unique to each practice and met with practice leadership on a regular basis (Table 2). Enhanced REP Internal and External Facilitators worked together to identify opportunities to enhance LGCC implementation.

Procedures

Health specialists from each practice were responsible for implementing LGCC. They were jointly hired by research and clinical staff and had a mental health clinical background and previous experience in addressing suicidal ideation, severe manic episodes, and other emergent issues. Health specialists identified and enrolled eligible patients if they had a diagnosis of bipolar spectrum disorder (including bipolar I, II, NOS, or schizoaffective-bipolar subtype) based on medical record review, received outpatient care from the participating practices, and were community-dwelling (i.e., not living in a nursing home or other institution). Patients were excluded if they were not able to provide informed consent due to serious illness or evidence of intoxication at enrollment.

Eligible patients were then approached via mail, phone, or in person and enrolled after providing informed consent and completed a baseline assessment survey. Participants were asked to attend four weekly group sessions over a 4 week-period, followed by monthly individual care management phone calls over 6 months. Health specialists recorded patient attendance, clinical status and length of time with care management contacts in an electronic registry. Participants in both implementation groups received $5 in remuneration for each LGCC group session attended to cover transportation costs, and $20 for each survey assessment.

LGCC Training, Supervision, and Fidelity

Health specialist training, supervision, and fidelity monitoring were similar across Enhanced and standard REP practices. One exception was that health specialists at the standard REP practices received a standard LGCC manual and training, while health specialists at the Enhanced REP practices received additional education and training in practice-specific organizational factors and barriers in anticipation of implementing LGCC from study investigators based on the needs assessments (Table 2). Standard REP and Enhanced REP LGCC training consisted of two days that covered the four self-management session content, registry use, and LGCC care management. All health specialists participated in biweekly research team conference calls lasting one hour in duration, where recruitment progress and other study issues were discussed. Health specialists at Enhanced REP practices also participated in 6 monthly calls with an external facilitator who employed a problem-solving approach to discuss implementation barriers and offer support to overcome these barriers. Fidelity findings to these implementation strategy processes will be published separately.

Measures

Fidelity Outcomes

We used a practical approach for assessing treatment fidelity to LGCC based on the National Institutes of Health’s Behavioral Change Consortium fidelity framework for psychosocial treatments (3, 4) and previous LGCC intervention studies (19, 27). Fidelity measures were developed a priori based on the core LGCC theoretical components (Table 1). Our two primary outcomes for fidelity included number of group self-management sessions completed and number of care manager contacts completed. In addition, study staff observed a random sample of 25% of group self-management sessions using a LGCC checklist of focus points to be covered in sessions, including the total amount of time for each session (Table 1) (28, 29).

Cutpoints for adequate treatment fidelity were based on previously established minimum necessary standards (19, 27). For optimal fidelity, a cutpoint of attending all four self-management sessions and having at least six care management contacts was used.

Covariates

Independent variables thought to influence fidelity and also differ between Enhanced and standard REP practices were ascertained from the patient baseline survey. Variables included demographics (age, gender, race/ethnicity, education, living alone) and lifetime history of homelessness, defined using a standard question ascertaining whether the patient ever spent at least one night in a shelter, park, abandoned building, or street. Other covariates included clinical indicators at baseline including depressive symptoms, alcohol use, and medical comorbidities. Depressive symptoms were ascertained from the Patient Health Questionnaire 9-item survey (PHQ-9) (30). Alcohol use, particularly hazardous drinking was defined based on the AUDIT-C question on binge drinking (31, 32). Number of psychiatric and medical comorbidities was ascertained from patient self-report and based on the question, “Has the doctor ever told you have one or more of the following:” with check boxes for hypertension/high blood pressure, arthritis/chronic pain, angina/coronary heart disease, heart attack/MI, depression, PTSD, diabetes/high blood sugar, or high cholesterol or parents with high cholesterol.

Data Analysis

Multiple regression models were used to compare treatment receipt fidelity (number of LGCC group sessions, number of completed LGCC care management contacts, and total number of LGCC sessions or contacts) between standard REP and Enhanced REP treatment arms. Logistic regression models were used to compare adequate fidelity and similarly to compare optimal fidelity between patients provided Enhanced REP versus standard REP. All models were adjusted for age, gender, race (white/nonwhite), college education (yes/no), living alone, homeless status (lifetime history- yes/no), baseline depression symptoms (PHQ-9 score), hazardous drinking (yes/no), and number of medical comorbidities. For all fidelity models, alternative analysis was conducted where individual sites are included as fixed effects. All analyses were conducted using SAS 9.2 (Cary, NC, USA).

RESULTS

A total of 1060 potentially eligible subjects were identified as having a bipolar disorder diagnosis from medical record reviews, of which 256 were found to be ineligible based on consultation with providers, leaving a total of 804, of which 416 declined or could not be contacted, leaving a total of 388 patients enrolled and consented. A total of 352 patients attended one or more LGCC sessions at the standard REP (n=155) and Enhanced REP (n=197) practices and were included in the analyses. The patient mean age was 42 years, 67% were female, and 29% nonwhite.

For self-management group sessions, patients attended on average 3.0 ±1.2 out of 4 possible sessions (range = 1 to 4), with 49% attending all four group sessions. Group session length averaged 114 minutes (range =112 to 120 minutes). For group session treatment delivery, number of focus points covered exceeded 80% for all Health Specialists.

Participants averaged 3.95 care management contacts, and 57% had at least four contacts. The mean length of time for each care management contact was 22.6 ± 12.1 minutes (range = 3 to 80 min).

Patients from practices receiving Enhanced REP to implement LGCC had a higher overall fidelity (treatment receipt), as measured by total number of LGCC contacts, than patients from practices receiving standard REP (8.14 vs. 5.53, P<.001;Table 3). Enhanced REP patients had more care manager contacts than those from standard REP practices, but there was no significant difference in the number of group self-management sessions completed (Table 3).

Table 3.

LGCC Treatment Fidelity of Patients from Practices Randomized to Enhanced REP or Standard REP

	Total (N = 352)			Enhanced REP (N = 197)			Standard REP (reference) (N = 155)			Fidelity comparing Enhanced and Standard REP
	N	%	Range	N	%	Range	N	%	Range	Test statisticc	df	P
Mean ± SD number of group sessions	3.02±1.22		0–4	3.08±1.12		0–4	2.94±1.33		0–4	1.02	1	.30
0 Group sessions	22	7		7	4		15	11		6.09	4	.19
1 Group sessions	16	5		10	6		6	4
2 Group sessions	51	16		31	17		20	14
3 Group sessions	74	23		43	24		31	22
4 Group sessions	154	49		86	49		68	49
Mean ± SD number care management sessions	3.95 ±2.40		0–15	5.03±2.35		0–15	2.57±1.64		0–7	10.82	1	<.001
0 Care Management Sessions	22	7		6	3		16	12		97.55	7	<.001
1	30	10		8	4		22	16
2	43	14		12	7		31	23
3	40	13		15	9		25	18
4	45	14		19	11		26	19
5	46	15		36	21		10	7
6	49	16		44	25		5	4
7 or more	36	11		35	20		1	1
Mean± SD total number of sessions (self and care management)	7.00 ±2.91		0–19	8.14±2.94		0–19	5.53±2.12		0–10	9.08	1	<.001
Group sessions ≥ 3 AND Care management sessions ≥ 4a	140	45		111	64		29	22		55.71	1	<.001
Group sessions = 4 AND Care management sessions ≥ 6b	48	16		46	27		2	2		36.34	1	<.001

^aThe minimum fidelity standard.

^bThe optimal fidelity standard.

^cTest statistics were t-value for comparisons of means and chi-square value for comparisons of distributions across categories.

Logistic regression analyses revealed that patients from practices receiving Enhanced REP were 7.2 times more likely to achieve minimal fidelity- that is receiving at least three self-management sessions and at least four care management contacts (95% CI = 3.96 to 7.89;Table 4). Patients from the Enhanced REP practices were 22.3 times more likely to achieve optimal fidelity, i.e., receiving all four self-management sessions and at least 6 care manager contacts (95% CI = 5.09 to 97.37;Table 4).

Table 4.

Adjusted Estimates of the Relationships between Patient Characteristics and LGCC Treatment Fidelity of Patients from Practices Randomized to Receive Enhanced REP or Standard REP

Multivariable Regression Resultsa	Total Number of Sessions (group and care management)				Number of Group Sessions				Number of Care Management (CM) Sessions				Met Minimumb Fidelity Standard (yes/no)				Met Optimalc Fidelity Standard (yes/no)
Covariate	Beta	95% CI	T	df	Beta	95% CI	T	df	Beta	95% CI	T	df	OR	95% CI	χ2	df	OR	95% CI	χ2	df
Patient at Practice w/ Enhanced REPd	2.64	2.01 to 3.27**	8.3	1	.11	−.18 to .41	.7	1	2.51	2.00 to 3.01**	9.8	1	7.15	3.96 to 7.89*	42.7	1	22.27	5.09 to 97.37**	16.9	1
Age	.03	.01 to .06*	2.0	1	.01	−.01 to .03	1.8	1	.02	−.01 to .04	1.5	1	1.01	.99 to 1.04	.8	1	1.02	.98 to 1.06	1.3	1
Female	−.73	−1.36 to –.11*	−2.3	1	−.30	−.59 to .01	−2.0	1	−.41	−.91 to .09	−1.6	1	.61	.34 to 1.08	2.9	1	1.16	.55 to 2.46	.1	1
Nonwhite	−.30	−.97 to .36	−.9	1	−.05	−.36 to .26	−.3	1	−.27	−.80 to .26	−.9	1	.81	.44 to 1.49	.5	1	.43	.16 to 1.15	2.8	1
College education	.28	−.50 to 1.06	.7	1	.07	−.29 to .44	.4	1	.17	−.46 to .79	.5	1	1.39	.69 to 2.79	.8	1	1.30	.59 to 2.89	.4	1
Living alone	.72	.07 to 1.38*	2.2	1	.16	−.15 to .47	1.0	1	.48	−.05 to 1.00	1.8	1	1.80	.99 to 3.29	3.7	1	1.39	.65 to 2.99	.7	1
Life time homelessness history	−.66	−1.26 to –.07*	−2.2	1	.03	−.25 to .32	.2	1	−.72	−1.20 to –.24*	−2.9	1	.82	.47 to 1.43	.5	1	.93	.44 to 1.93	.04	1
PHQ-9 score	−.02	−.07 to .03	−.9	1	−.01	−.03 to .01	−1.1	1	−.01	−.04 to .03	−.3	1	.99	.95 to 1.03	.3	1	.99	.94 to 1.04	.2	1
Hazardous drinking	−.94	−1.99 to .12	−1.7	1	−.33	−.84 to .17	−1.3	1	−.64	−1.48 to .19	−1.5	1	.53	.20 to 1.39	1.7	1	.23	.03 to 1.86	1.9	1
Number of medical comorbidities	.04	−.27 to .35	.2	1	−.03	−.15 to .14	−.04	1	.09	−.16 to .34	.7	1	1.07	.82 to 1.42	.3	1	1.05	.72 to 1.53	.1	1

OR = Odds Ratio

^*p-value<.05,

^**P<.001

^aCovariates that were adjusted in the models included age, gender, race, college education, living alone, homeless status, PHQ-9 score, Alcohol abuse, and number of medical comorbidities.

^bGroup sessions ≥ 3 andCare management sessions ≥ 4

^cGroup sessions = 4 andCare management sessions ≥ 6

^dStandard REP is the reference group

Regression analyses also revealed that women were less likely to complete LGCC, particularly group sessions (Beta=−.30, p< .05) and participants with a history of homelessness were less likely to complete care management contacts (Beta=−.72, p<.05). Younger age (Beta= .03, p< .05) and living alone (Beta= .72, p< .05) were associated with fewer total contacts. Additional analyses in which practices were added as fixed effects produced similar results (Table 5).

Table 5.

Adjusted Estimates of the Relationships between Patient Characteristics and LGCC Treatment Fidelity of Patients from Practices Randomized to Receive Enhanced or Standard REP, based on the Model where site is included as Fixed Effects

Multivariable Regression Resultsa	Total Number of Sessions (group and care management)				Number of Group Sessions				Number of Care Management (CM) Sessions				Met Minimumb Fidelity Standard (yes/no)				Met Optimalc Fidelity Standard (yes/no)
Covariate	Beta	95% CI	T	Df	Beta	95% CI	T	df	Beta	95% CI	T	df	OR	95% CI	χ2	df	OR	95% CI	χ2	df
Sites
Site 1 (Enhanced REP)	2.75	1.87 to 3.63**	6.1	1	53	.11 to .94*	2.5	1	2.25	1.55 to 2.96**	6.3	1	6.13	2.68 to 14.04**	18.4	1	12.18	2.52 to 58.73*	9.7	1
Site 2 (Enhanced REP)	.91	−.13 to 1.95	1.7	1	−.16	−.65 to .34	−.6	1	1.09	.26 to 1.93*	2.6	1	2.11	.83 to 5.37	2.5	1	5.35	.93 to 30.82	3.5	1
Site 3 (Enhanced REP)	3.03	2.08 to 3.98**	6.3	1	.61	.16 to 1.07*	2.6	1	2.38	1.62 to 3.14**	6.2	1	6.35	2.61 to 15.42**	16.7	1	20.54	4.25 to 99.43**	14.1	1
Site 4 (Standard REP)	−.45	−1.36 to .46	−.9	1	.54	.11 to .97*	2.5	1	−.95	−1.68 to − .22*	−2.6	1	.42	.16 to 1.09	3.2	1	.001	<.001 to >999	.003	1
Age	.03	.0004 to .06*	1.9	1	.01	−.002 to .03	1.7	1	0.02	−.01 to .04	1.5	1	1.01	.99 to 1.04	.8	1	1.02	.98 to 1.06	.9	1
Female	−.71	−1.34 to –.08*	−2.2	1	−.17	−.47 to .13	−1.1	1	−.50	−1.01 to .004	−1.9	1	.55	.29 to 1.03	3.5	1	1.16	.54 to 2.53	.1	1
Nonwhite	−.26	−.92 to .40	−.8	1	−.14	−.46 to .17	−.9	1	−.14	−.68 to .39	−.5	1	.88	.46 to 1.67	.2	1	.44	.16 to 1.22	2.5	1
College education	.24	−.52 to 1.0	.6	1	.06	−.30 to .42	.3	1	.13	−.48 to .74	.4	1	1.33	.65 to 2.72	.6	1	1.33	.59 to 2.99	.5	1
Living alone	.69	.06 to 1.34*	2.1	1	.16	−.14 to .46	1.0	1	.46	−.06 to .97	1.7	1	1.79	.96 to 3.31	3.8	1	1.46	.66 to 3.21	.9	1
Life time homelessness history	−.51	−1.09 to .08	−1.7	1	.07	−.21 to .35	.5	1	−.60	−1.07 to − .13*	−2.5	1	.91	.51 to 1.61	.1	1	1.08	.51 to 2.29	.04	1
PHQ-9 score	−.02	−.07 to .02	−1.0	1	−.01	−.04 to .01	−1.2	1	−.01	−.04 to .03	−.4	1	.99	.94 to 1.03	.4	1	.99	.93 to 1.04	.03	1
Hazardous drinking	−.64	−1.67 to .39	−1.2	1	−.27	−.77 to .23	−1.1	1	−.44	−1.26 to .38	−1.1	1	.63	.23 to 1.73	.8	1	.28	.03 to 2.33	1.4	1
Number of medical comorbidities	.09	−.21 to .39	.6	1	.02	−.12 to .16	.3	1	.12	−.12 to .36	.9	1	1.09	.83 to 1.46	.4	1	1.11	.76 to 1.63	.3	1

OR = Odds Ratio

^*p-value<.05,

^**P<.001

^aCovariates that were adjusted in the models included age, gender, race, college education, living alone, homeless status, PHQ-9 score, Alcohol abuse, and number of medical comorbidities.

^bGroup sessions ≥ 3 and Care management sessions ≥ 4

^cGroup sessions = 4 and Care management sessions ≥ 6

^dSite 5 (standard REP) was the reference site

DISCUSSION

As one of the first comparative effectiveness trials of two different implementation strategies, one of the goals of the Recovery-Oriented Collaborative Care study was to determine whether an enhanced versus standard version of a well-established implementation strategy (REP) improved fidelity to a psychosocial treatment for bipolar disorder. There is a growing demand for specific implementation interventions that are theory-based and that are proven to enhance and maintain the fidelity of evidence-based practices. However, few implementation frameworks have been rigorously tested in health services trials as implementation strategies. We found that compared to standard REP implementation, Enhanced REP improved overall fidelity to Life Goals Collaborative Care independent of patient factors, and that differences in fidelity were most pronounced for the ongoing care management component of LGCC.

The level of fidelity to LGCC among patients from the Enhanced REP practices was similar to estimates reported in previous randomized controlled trials in more tightly controlled clinical settings (19, 27). Akin to these previous LGCC studies, we used practical fidelity assessments that were easy to deliver, such as the fidelity rater checklist, and we were able to demonstrate good treatment delivery and treatment receipt fidelity of the LGCC intervention.

Overall, high patient-level treatment fidelity to group sessions attended and the number of care management contacts was achieved across all practices. Health specialists across all of the practices demonstrated good fidelity to LGCC focus points and process goals in the group sessions. These findings could be due to the small incentives given to patients to cover transportation costs to group sessions. LGCC sessions were also limited to four which may have encouraged participation.

Compared to standard REP, Enhanced REP was associated with improved fidelity to LGCC; notably, participants were significantly more likely to attend more overall sessions and contacts as well as complete all 4 group sessions and at least 6 care management contacts. Enhanced REP was associated with optimal fidelity mainly due to increased receipt of care management contacts. Often the hardest component to maintain over time, care management is a crucial component of the LGCC and similar Chronic Care Models because of the need to assess clinical status over time and encourage positive health behavior changes especially within a chronic illness such as bipolar disorder. Enhanced REP might have improved sustainability of care management because the External and Internal Facilitators were able to assist the health specialists in garnering the required resources for group sessions and provider follow-up. Enhanced REP likely led to improved fidelity to LGCC because added Facilitation components such as leadership engagement and customization led to a greater overall acceptance of LGCC in the practices. Greater use of care management in the Enhanced REP practices also suggests that participants were more engaged with the health specialist and focused on their wellness goals.

Additionally, we examined patient factors that influenced treatment delivery fidelity. Younger participants and those who were living alone participated in fewer group sessions. We also found that women participated in fewer group sessions and those study participants who had a history of homelessness completed fewer care management contacts. Younger individuals and those living alone may have difficulty with attending groups due to job constraints or transportation issues. Women may have had difficulty attending groups due to child or other family caretaking responsibilities and may need different options for group times or provision of childcare. Similarly, those with housing instability might benefit from more flexibility in delivering care management contacts such as face-to-face encounters or use of other technologies that enable more cost-efficient communication.

The study has limitations that may impact generalizability to other community settings. We were unable to assess contacts with other providers, either by the health specialists or patients, or assess whether enthusiasm for the LGCC intervention at the sites varied across individual providers, or over time. We were also unable to describe the specific components of Enhanced REP, including the Facilitator’s actions that led to improved fidelity. We were unable to monitor fidelity to provider guideline support, including use of guidelines by other frontline providers in the practice, as well as comprehensively assess potential patient factors influencing fidelity including co-occurring psychiatric conditions. Also, the health specialists who delivered LGCC were all masters’ level clinicians with prior clinical experience, who may not represent the availability of providers especially in smaller community-based clinics.

Nonetheless, implementation interventions such as Enhanced REP that include ongoing technical assistance to help providers implement evidence-based practices may improve treatment fidelity. Future research should consider whether the added costs of Facilitation leads to improved patient outcomes and value from the health care organization’s perspective in terms of cost-efficient delivery of effective psychosocial treatments in routine practice. Finally, measures that assess fidelity to the implementation intervention should be developed further in order to better assess the uptake of implementation interventions as potential tools to improve the translation of research into practice.