Are we being equitable enough? Lessons learned from sites lost in an implementation trial

Abstract

Background

There is a growing interest in practice-based implementation research, yet too often research prioritizes and is most successful in academic settings. During a national implementation trial to evaluate the effectiveness of Collaborative Care for co-occurring opioid use and mental health disorders, we lost three of our 11 participating implementation sites, all representing community sites.

Method

To better understand needed supports for implementation trial participation, we conducted exit interviews (n = 5) with key staff at these community sites. Interview transcripts were double-coded and analyzed using Rapid Assessment Process. Qualitative themes were iteratively reviewed by the study team.

Results

Three themes emerged characterizing challenges for community sites, including that: (1) research threatens sites’ most precious resource—staff; (2) staff lack comfort with and skills for research; and (3) research participation in its current form does not offer a clear return on investment.

Conclusions

Learnings from this work illuminate some of the barriers community sites face when trying to participate in multisite implementation research. An undercurrent of participant perspectives was the belief that community sites like theirs are just not set up to successfully participate in clinical trial research, including population-based implementation trials. Future implementation trials should consider strategies that disrupt traditional approaches, increasing the equitable inclusion of diverse practice settings in implementation research.

Plain Language Summary

There is a growing interest in research that reflects community settings. Yet too often, research is most successful in academic settings. During a national implementation trial to evaluate the effectiveness of Collaborative Care for co-occurring opioid use and mental health disorders, we lost three of our 11 participating implementation sites, all representing community sites. To better understand their perspectives, we conducted exit interviews (n = 5) with staff at these community sites. Interview transcripts were double-coded and analyzed using thematic analysis. Analysis identified three themes: (1) research threatens sites’ most precious resource—staff; (2) staff lack comfort with and skills for research, and (3) research participation in its current form does not offer a clear return on investment. Community sites face many barriers to participating in implementation research trials. Future trials should consider ways to disrupt traditional approaches and increase the equitable inclusion of community settings in implementation research.

Introduction

Recent years have been marked by a paradigm shift in research, including a growing interest in practice-based research that not only demonstrates what works, but what works for diverse settings and stakeholders (Forsythe et al., 2015; Westfall et al., 2007). Central to this is diversifying where research is conducted and expanding practice-based research that draws on the varied real-world conditions of care delivery (Westfall et al., 2019). Too often though, implementation research still prioritizes or is most successful in academic settings, leaving out pivotal stakeholder perspectives like community-based settings. These are also the settings which, arguably, need evidence the most to maximize effective care delivery with often very limited resources.

In 2019, we launched a national multisite hybrid type 2 effectiveness-implementation trial to evaluate the effectiveness of Collaborative Care for patients with co-occurring opioid use and mental health disorders in primary care settings and the effect of tailoring implementation supports for implementation based on clinic-level baseline fidelity to Collaborative Care principles (the Collaborating to Heal Addiction and Mental Health in Primary Care (CHAMP) study). We initially recruited multiple healthcare organizations across the United States that reflected diverse geographic and structural characteristics. Each healthcare organization provided at least two primary care clinics (some provided four) to participate in the trial and we collaborated with local implementation teams for 18 months to support their training and preparation for launch of both practice change (delivery of Collaborative Care for co-occurring disorders) and research activities (enrollment of eligible patients). However, close to the time of study launch, three healthcare organizations (representing six participating clinics) exited the trial. Importantly, all three organizations were nonacademic community sites, including two federally qualified health centers located in underserved communities. To better understand needed supports for implementation trial participation, we conducted exit interviews with clinical and administrative staff at the three exited healthcare organizations to characterize their perspectives on barriers to trial participation.

Materials and Method

Primary care clinics participating in this cluster randomized trial had existing Collaborative Care programs for mental health disorders (e.g., depression and anxiety), and were asked to expand Collaborative Care services to patients with opioid use disorder at clinics randomized to the intervention group. Implementation and training activities began in July 2020, hence all clinic experiences were impacted by the COVID-19 pandemic. Training and practice facilitation were provided by the Advancing Integrated Mental Health Solutions (AIMS) Center at the University of Washington, similar to what is offered to healthcare systems that purchase these services from the AIMS Center outside the context of research. Clinics were offered training and practice facilitation prior to the launch of care delivery for co-occurring disorders and ongoing implementation support via monthly to bimonthly calls with an external facilitator throughout the implementation period.

Clinics were also asked to support patient recruitment, eligibility assessment, consenting, limited data collection, and adverse event reporting, and completed protection of human subjects and other trainings to aid their facilitation of these research tasks. All clinics received payments as they completed specific implementation milestones or recruited patients ($900 per patient), up to $150,000 total if all study milestones were met and had flexibility in how they used those funds to support research participation. Table 1 provides an overview of the activities and supports clinics received.

Table 1.

Description of Study Participation Requirements and Supports Provided to Implementation Sites.

Type of activity	Study participation requirements	Details/examples	Supports provided by study team
Intervention deployment	Identify a local implementation team to lead study activities	Identify clinical champions (≥1 medical,  ≥ 1 behavioral health) Identify ≥1 administrative champions Participate in 1–2 1-h meetings per month with external facilitator over 18-month preparation and implementation period to support clinical implementation of Collaborative Care Participate in scheduled and self-directed trainings (10–30 h, depending on role)	Multiple clinic payments to support coverage of time spent in implementation meetings and trainings (e.g., $15,000 for completion of training) Role-specific training resources for clinical activities (asynchronous online learning modules, live training sessions, access to mentors) A dedicated practice coach to lead all meetings (scheduling, agenda planning, meeting documentation)
Intervention deployment	Implement Collaborative Care for patients with co-occurring opioid use and mental health disorders	Complete necessary training to deliver medications for opioid use disorder (10–30 h, depending on role) Expand workflows for Collaborative Care referral and practice to accommodate co-occurring disorders Monitor and refine implementation processes via monthly meetings with external facilitator Track specific patient care activities (e.g., screening, medication management, measurement-based care) in registry for all patients receiving Collaborative Care for co-occurring disorders	Role-specific training resources for Collaborative Care for co-occurring disorder Role-specific “office hours” and mentorship opportunities (offered monthly) Example workflows Example scripts for staff when working with patients with co-occurring disorders A study-specific clinical registry to document clinical care activities for Collaborative Care
Quality improvement	Implement routine screening for opioid use disorder	Develop workflows for population-based opioid use disorder screening implementation Support local practice change and training of other frontline staff via formal and informal presentations to staff (≥1 presentation to all staff prior to launch of screening) Monitor and refine screening implementation processes	Training resources for the screening tool Example workflows Example scripts for staff to use when using the screener in practice Ongoing coaching support via monthly meetings with external facilitator and monthly role-specific office hours
Research	Assess eligibility and recruit eligible study participants	Assess eligibility of potential study participants Approach and consent eligible study participants Document inclusion and exclusion criteria in study database	Access to Collaborative Institutional Training Initiative (CITI) training for human subjects research Dedicated research support staff to assist with initial training and ongoing technical assistance A study-specific research database (REDCap) A site-specific research dashboard to track enrollments and needed follow-up Financial payment of $900 per patient enrolled, provided to clinics to support coverage of staff time for research activities
Research	Support data collection and adverse event reporting for enrolled study participants	Document specific patient care activities in registry for all patients enrolled in study Provide reminders to enrolled participants about survey due dates Report hospitalizations, emergency room visits, and other adverse events to coordinating center	Dedicated research support staff to assist with initial training and ongoing technical assistance A study-specific research database (REDCap) and clinical registry tool A site-specific research dashboard to track enrollments and needed follow-up Regular meetings with research staff

To understand barriers to research participation among exited sites, we identified and invited key informants (n = 5) from the three organizations to participate in an exit interview based on their involvement in study implementation activities. To enhance credibility, key informants needed to be directly involved in organizational decision-making about exiting the trial (Korstjens & Moser, 2018). Interviews lasted 30–45 min and were conducted via teleconference by a member of the evaluation team with qualitative expertise (Elizabeth J. Austin). Interview questions were informed by prior formative evaluation activities including participant observation of all implementation meetings with individual sites. Interviews asked participants to reflect on their experience with study implementation, decision to exit the study, and recommendations for supports that would enhance future engagement of community sites. Interviews were audio recorded and professionally transcribed. Transcripts were double-coded by two trained qualitative researchers (Elizabeth J. Austin and Jessica Chen) using Rapid Assessment Process, a team-based process where data are coded to structured templates and then iteratively reviewed to generate themes (Beebe, 2001). Coders were chosen specifically to increase reflexivity during the analysis process, as one coder (Elizabeth J. Austin) had conducted prolonged fieldwork (∼18 months) with these implementation sites and the other coder (Jessica Chen) had not (Korstjens & Moser, 2018; Whittemore et al., 2001). Throughout analysis, coders utilized thick description and iterative qualitative memoing to guide identification of themes (Charmaz, 2006). Presentations of themes included direct quotes from multiple participants to increase confirmability and were shared with the qualitative lead (Emily C. Williams) and the full study team iteratively for further refinement and verification (Cope, 2014; Korstjens & Moser, 2018). All activities were approved by the Advarra Institutional Review Board.

Results

Table 2 provides structural characteristics of the 11 implementation sites, including comparisons of sites that exited and sites that remained in the study. Sites that exited the study were more likely to be located in the South, have a higher patient census, serve patients with limited English proficiency, and serve patients on Medicaid. All five key informants held leadership positions within their organizations, and one had direct patient care responsibilities. Analyses identified three themes that characterize the unique barriers community sites face to research participation.

Table 2.

Structural Characteristics of Exited and Continuing Implementation Sites.

Clinic characteristicsa	Sites that continued study (n = 8)	Sites that exited study (n = 3)
Geographic region
Midwest	4 (50%)	0
Northeast	1 (12.5%)	0
South	1 (12.5%)	3 (100%)
West	2 (25%)	0
Clinic ownership
Local, county, or community government	1 (12.5%)	0
Private, for-profit hospital/hospital system	0 (0%)	1 (33%)
Not-for-profit organization/foundation	7 (88%)	2 (66%)
Special designation or accreditation b
Federally qualified health center	2 (25%)	2 (66%)
Patient-centered medical home	1 (12.5%)	3 (100%)
Academic-medical center affiliated	2 (25%)	0
Residency program at clinic	2 (25%)	0
Annual patient census
Average patient census (range)	11,744 (2,010–50,182)	11,582 (7,945–14,800)
LEP needs of patients
Average % of patient census with LEP needs	6.5% c	18.6%
Payer mix of patients served
Direct cash or self-pay	9.4%	24.3%
Medicaid	12.8%	30.6%
Medicare	30%	16.1%
State-financed insurance other than Medicaid	6.7%	3.5%
Federal military insurance (Tricare)	0.1%	3.7%
Private	40%	21.5%
Workman's compensation/L&I	0.2%	0.3%
Other	1%	0%

Note. LEP = limited English proficiency.

^a Percentages are rounded and may not summate to 100.

^b Responses are not mutually exclusive.

^c Two sites reporting unknown.

Theme 1: The work of research threatens community sites’ most precious (and at risk) resource–staff

Participants described feeling overwhelmed by the amount of time staff needed to spend in research-related activities during the study.

I think it was definitely overwhelming, even for me when I was sitting in that project manager lead kind of role of every task we had to get done before the next meeting, or whatever. When you’re trying to incorporate that when you’re already full time doing other things, it's tricky in trying to get everybody to have the time freed up to do that. (Site B1)

Another described the burden of study-related meetings, saying:

There were so many forms that needed to be signed by patients which was totally understandable, but there were the consents that needed [to be done] and so much data that had to be kept, so much information that had to be kept. It was extremely time consuming. There were so many meetings, you know […] our resources were pretty limited. (Site C1)

Participants expressed that they did not understand expectations or have time for the time commitment required, especially staff with clinical responsibilities. As one said:

There's trainings, there's the conference calls, there's those are blocks of time that if I pulled out the care manager […] I have to get somebody else to sub for her […] so, I think it's the amount of time it's going to take, we weren’t realistic about or it just didn’t occur to us that it was going to require that much time. (Site C1)

Participants felt research activities added stress to their clinical staff, who often were already overburdened giving perpetual staff shortages. Even further, participants found hiring and retaining staff challenging in community settings where they felt open jobs may be less competitive:

One of the challenges is, to me, and this is an assessment after two years of working at a federally qualified health center, there's a lot of great upper-level staff that have chosen this as their mission and career. Then, there's lower-level staff that it's a job, which is great, but they can get paid more elsewhere. (Site A3)

As a result, when frontline staff expressed challenges or discouragement from research responsibilities, sites felt compelled to prioritize staff preferences over research participation. One participant shared an example of a provider who felt isolated as they were the only one involved in research activities:

[It] really just kinda boiled down to it just didn’t work out. I think that provider got deflated in, ‘I’m gonna be a lone ranger here, and I don’t wanna do that.’ (Site B1)

Theme 2: Staff lack familiarity with and capacity for research-related tasks

In addition to the time research required, community site staff lacked familiarity with research activities. One participant described the challenge of confidently describing aspects of randomization and study design, saying:

For us, it was pretty hard to explain in the control and intervention clinic what the difference was to people, so that may be something. […] I think that just added so many different factors that it made it kind of confusing. (Site A1)

Another described that research concepts, including trainings related to informed consent and research ethics, felt out of scope for clinical staff:

And things like the CITI [Collaborative Institutional Training Initiative] and anything to do with the research component of it is really far outside the normal training and experience of most mental health clinicians. (Site A1)

Participants also expressed a sense of inadequacy with research. One participant shared a concern about “damaging the study” (Site C1) by lacking the skills to conduct research activities well. Another participant shared their perspective that “the people who are at an academic center might be more capable than the people in a community center” (Site A3).

Participants also unanimously identified the need for dedicated research staff. As one participant described:

We’ve had conversations subsequently at [our organization] that probably we would not participate in research of this type […] unless there was a fulltime research assistant person completely dedicated to oversight of the project and the data and the consenting and all of that process. (Site A1)

Overall, participants expressed a sentiment that in terms of conducting research activities, “we’re not setup that way, we’re just not built that way” (Site A2).

Theme 3: Research participation did not offer a clear return on investment for community sites

Participants felt that participation in research presented time and resource conflicts and “takes away from patient care” (Site B1). As one participant said:

I think that the challenging part is just that patient care and that clinical time always takes priority over any data collection. (Site A2)

Another participant shared:

The [study-related] training for the PCPs is kind of, while I know it's necessary, I think it's cumbersome for the time. Especially when you’re asking them to carve out time and they’re already patients that they book out, three, six months out at a time and they’re full, and they really can’t. They have productivity metrics they have to meet. (Site B1)

Some participants felt that research was just too hard for community settings to do:

I’ve worked in federally qualified health centers for years, and patient-based research, which involves consent and different interventions, is really hard to do in these settings. Usually for trials or things like that, people go somewhere like NIH or NIMH and they enroll in a trial, and it's very hard to do in a context of day-to-day providing care to patients because day-to-day you’re gonna just provide the care you provide for the most part. (Site A1)

Other participants highlighted that the compensation amount and structure of payment provided for study participation did not cover the actual impacts from their investment:

[Study payments] did [help] in the sense that it allowed me a justification to block off some psychiatric time for psychiatric consulting and support to the project. I think it allowed us to reduce somewhat the productivity expectations for the psychiatrists and the two mental health clinicians, but not a whole lot. […] The way the financing was structured from this point forward, the majority of it was really based on numbers of patients to enroll and our anticipated number of patients to enroll at this time is, like, we get almost no revenue from that at the rate we’re going. So, the fact that there was some financial compensation sort of allowed us to get into the project, but then in terms of sustaining it or really supporting staff, it wasn’t really notable. (Site A1)

Lastly, while participants shared that the decision to exit the study was difficult, they also expressed relief at the opportunity to shift focus back to clinical care.

Discussion

In this work, we explored perspectives of community sites that felt unable to continue their participation in a multisite implementation trial. Key learnings emphasized that community sites often feel added tension when asking clinical staff to take on research responsibilities, feel inadequate in their research skills, and find research participation to be a zero-sum game with limited return on investment.

Learnings from this work highlighted that the current approach to site participation in multisite implementation research often shares resources in ways that are equal, but not equitable. For example, research teams often offer sites equal amounts of monetary compensation for their participation. However, this fails to acknowledge that sites may have different financial burdens or impacts from their research participation. Community sites, for example, often receive lower reimbursement rates and have historically and systematically had less access to financial resources beyond direct patient care revenue. This was mirrored in our exited community sites which, compared to sites that were able to continue the study, served more patients on Medicaid and patients requiring language assistance. Instead, future implementation research teams must address inequities by reimagining and redesigning what have been the default settings of research (Benjamin, 2022). This should involve tailoring the types, amounts, and formats of research support to each site's individual capacity and need (Tambor et al., 2021). It may also mean allowing community sites to participate in hybrid ways that are less burdensome, such as allowing sites to implement some but not all research requirements or working with sites to identify differing levels of resources to support their research engagement (Goodlett et al., 2020). While this may challenge the balance between internal and external validity, it may also hasten the pipeline from evidence to translation.

Additionally, research teams may need to consider new ways to incentivize and create value in research participation at the local level (Luger et al., 2020). Research participation, like any implementation, can be a new skillset—in and of itself a form of practice change. A challenge shared by our participants was that while monetary compensation may have supported the organization overall, it did little to encourage individual frontline staff to participate in research activities. The effort needed to generate local buy-in for research participation should not be underestimated. Future research teams should increase stakeholder engagement before and throughout research trials to ensure community site perspectives at all levels of the organization are prioritized (Goodlett et al., 2020). Further expansion of practice-based research networks (PBRNs) may offer another pathway to increase community site participation in clinical research, yet more support is needed to ensure PBRNs have the capacity to match the expanding volume of implementation research (Dania et al., 2022; Davis et al., 2012; Peterson et al., 2012).

This work is limited by its small sample size; however, it does reflect the experiences of three diverse healthcare organizations located in different U.S. states and provides a foundation for future research. Additionally, the implementation of this study took place during the early stages of the COVID-19 pandemic, which has had well-documented impacts on primary care and likely exacerbated the burden on all participating clinics, especially those represented here who exited the study (Khalil-Khan & Khan, 2023).

Conclusions

In this work, we leveraged the loss of sites in our multisite trial to explore community site perspectives on research participation. Learnings emphasize the need to move away from one-size-fits-all approaches to site engagement, and for more collaborative investment with community sites to help identify and strengthen their gains from implementation research.