Accounting for quality improvement during the conduct of embedded pragmatic clinical trials within healthcare systems: NIH Collaboratory case studies

Abstract

Embedded pragmatic clinical trials (ePCTs) and quality improvement (QI) activities often occur simultaneously within healthcare systems (HCSs). Embedded PCTs within HCSs are conducted to test interventions and provide evidence that may impact public health, health system operations, and quality of care. They are larger and more broadly generalizable than QI initiatives, and may generate what is considered high-quality evidence for potential use in care and clinical practice guidelines. QI initiatives often co-occur with ePCTs and address the same high-impact health questions, and this co-occurrence may dilute or confound the ability to detect change as a result of the ePCT intervention.

During the design, pilot, and conduct phases of the large-scale NIH Collaboratory Demonstration ePCTs, many QI initiatives occurred at the same time within the HCSs. Although the challenges varied across the projects, some common, generalizable strategies and solutions emerged, and we share these as case studies.

Key lessons:

Study teams often need to monitor, adapt, and respond to QI during design and the course of the trial. Routine collaboration between ePCT researchers and health systems stakeholders throughout the trial can help ensure research and QI are optimally aligned to support high-quality patient-centered care.

1. Background

Many decisions made in healthcare are based on low-quality evidence from small or observational studies.^1,2 Large-scale embedded pragmatic clinical trials (ePCTs) are typically proposed when there is sufficient evidence from these studies and enough uncertainty about the effects and value of implementing an intervention in everyday clinical settings. Embedding PCTs within healthcare systems (HCSs) can maximize efficiencies of conducting trials, support the potential adoption of promising results, help generate high-quality evidence about important public health questions, integrate best practices within health systems, and improve quality of care. Simultaneously within health systems, ongoing quality improvement (QI) activities that implement smaller-scale interventions also regularly emerge to address urgent public health issues in real time. Both ePCTs and QI initiatives have the potential to improve health outcomes and promote high-quality, cost--effective healthcare. The primary difference is that QI activities are designed to change local processes and practice to achieve accepted standards of care, and ePCTs are designed to help determine the standards of care.³ The development and implementation of QI activities within health systems during the course of ePCTs is a major challenge to their design, methods, and assigned treatments. Therefore, such activities may threaten the ability to glean reliable, broadly generalizable evidence from the ePCT.

2. Organizational context

Since 2013, the National Institute of Health’s (NIH) Health Care Systems Research Collaboratory (Collaboratory) has supported over 15 large-scale, multi-site ePCTs that are conducted in healthcare settings. Collaboratory ePCTs are typically conducted over four years and use system infrastructure, such as staff, space, and data from electronic health records (EHR), to implement trials and ascertain endpoints.⁴ Important healthcare and public health questions addressed by Collaboratory trials and described in this Case Report include hospital-based infections, colorectal cancer screening, dialysis outcomes, alternatives to opioid treatment for chronic pain, and multiple co-morbid condition management, among others (Table 1). When developing the trials, study teams made adjustments to their trial design to accommodate QI activities co-occurring in the health system. During the conduct of these trials, study teams noted changes to both usual care control arms and intervention arms as a result of temporal changes in practice, particularly those due to QI initiatives within the HCS. This article uses case examples from the Collaboratory to illustrate challenges and provide strategies for the pilot phase, design, recruitment, site selection, conduct, and analysis phases of ePCTs.

Table 1.

Collaboratory ePCTs described in this Case Report*.

Trial	Goal	Healthcare Systems (HCSs)/Patients	Trial Design	Phase and brief summary of issue
Active Bathing to Eliminate (ABATE) Infection study (NCT02063867); Status: complete	To reduce multidrug-resistant organisms and bloodstream infections compared to usual care.5	53 HCA Healthcare hospitals (194 non-critical care units; ∼340,000 patients in the intervention period)	Cluster-randomized trial of daily antiseptic bathing for all patients and nasal antibiotic ointment for patients harboring methicillin-resistant Staphylococcus aureus (MRSA) compared to routine care	Conduct phase: potential for competing interventions
Guiding Good Choices for Health (GGC4H) (NCT04040153); Status: ongoing	To demonstrate the feasibility of implementing Guiding Good Choices in pediatric primary care settings and evaluate its effectiveness in reducing adolescent substance use initiation and improving behavioral health.	Three HCSs (Henry Ford Health System, Kaiser Permanente Colorado, Kaiser Permanente Northern California; ∼3600 families)	Cluster-randomized trial with randomization at the pediatrician level; parents whose adolescents are empaneled with intervention arm pediatricians will be offered Guiding Good Choices, a 5-session evidence-based preventive anticipatory guidance intervention for parents of young adolescents.	Conduct phase: potential for exposure to similar interventions
Improving Chronic Disease Management with Pieces (ICD-Pieces) (NCT02587936); Status: ongoing	To test the hypothesis that patients with chronic kidney disease (CKD), diabetes, and hypertension who receive care with a collaborative model of primary care-subspecialty care enhanced by novel information technology (Pieces) and practice facilitators will have fewer hospitalizations, readmissions, cardiovascular events and deaths than patients receiving standard medical care.	Four HCSs (Parkland Health and Hospital System, VA North Texas, Texas Health Resources and ProHealth Physicians; ∼11,000 patients)	Group randomized to receive ICD-Pieces, which is facilitated by clinical decision support and practice facilitators	Conduct phase: Many different overlapping interventions
Lumbar Imaging with Reporting of Epidemiology (LIRE) (NCT02015455); Status: complete	To demonstrate that a simple and inexpensive intervention, providing what are essentially normal values for diagnostic imaging, would decrease health care interventions such as diagnostic testing, injections, opioid prescriptions and surgeries.6	Four HCSs (Kaiser Permanente Northern California, Kaiser Permanente Washington, Mayo Clinic Health System, Henry Ford Health System; 98 clinics; ∼250,000 patients)	A stepped-wedge randomized trial of inserting epidemiological benchmark data in routine spine imaging reports	Pilot phase: Launch of similar intervention
Pain Program for Active Coping and Training (PPACT) trial (NCT02113592); Status: complete	To assess the potential benefit of helping patients adopt self-management skills for chronic pain, limit use of opioid medications, and identify factors amenable to treatment in the primary care setting in three health systems.7	Three Kaiser Permanente HCSs (Northwest, Georgia, and Hawaii; ∼800 patients)	Cluster randomized by primary care provider to receive non-pharmacological interventions, including physical therapy and psychological interventions	Recruitment phase: simultaneous QI efforts caused confusion for clinicians and potential participants
Personalized Patient Data and Behavioral Nudges To Improve Adherence to Chronic Cardiovascular Medications (Nudge) (NCT03973931); Status: ongoing	To improve medication refill adherence among patients with cardiovascular diseases (coronary artery disease, diabetes, hypertension, hyperlipidemia and atrial fibrillation) vs usual care	Three HCSs (UCHealth, VA Eastern Colorado Health Care System, Denver Health; ∼5000 patients)	Patient-level randomization to usual care (no text message reminders) vs generic text message reminders, text message reminders with behavioral nudges, or text message reminders with behavioral nudges and chatbot	Conduct phase: many similar concurrent QI activities
Pragmatic Trial of Population-based Programs to Prevent Suicide Attempt (SPOT) (NCT02326883); Status: ongoing	To evaluate the effectiveness of two population-based outreach programs for preventing suicide attempts among patients identified as at-risk.8	Four HCSs (HealthPartners and Kaiser Permanente Washington, Kaiser Permanente Colorado, and Kaiser Permanente Northwest; ∼19,000 patients)	Patient-level randomization to 1) usual care 2) a care management intervention or 3) an online skills intervention	Pilot phase: Sites adapted a version of the intervention similar to one used in the pilot phase
Pragmatic Trial of User-Centered Clinical Decision Support to Implement EMergency Department-Initiated BuprenorphinE for Opioid Use Disorder (EMBED) (NCT03658642); Status: ongoing	To increase rates of emergency department-initiation of Buprenorphine/naloxone and referral for ongoing treatment for patients with opioid-use disorder. BUP is a well-established effective treatment but its use has not been routinely implemented into emergency department care.9	20 Emergency Departments across five HCSs ∼9900 patients (Yale New Haven Health, University of North Carolina Health University of AlabamaBirmingham Health, Baystate Health, and University of Colorado Health)	Group-randomized trial of user-centered computerized clinical decision support	Design phase: potential for confounding effects due to multiple QI initiatives to address opioid use disorder
Primary Palliative Care for Emergency Medicine (PRIM-ER) (NCT03424109); Status: ongoing	To test the effectiveness of primary palliative care education, training, and technical support for emergency medicine.10	35 emergency departments (EDs) in 18 HCSs ranging from academic medical centers to community hospitals (∼4983 providers; ∼57,717 patients)	Stepped-wedge randomization to asynchronous learning and technical support to bolster emergency providers’ palliative care skills.	Site selection phase: enrollment of early adopters of innovations may lead to multiple competing interventions
Strategies and Opportunities to STOP Colon Cancer in Priority Populations (STOP CRC) (NCT01742065); Status: complete	To determine whether EHR-embedded tools and clinic staff training in how to implement a mailed FIT outreach program could increase colorectal cancer screening uptake among patients with historically lower CRC screening rates and worse CRC outcomes, such as those with low income, or who are on Medicaid or underinsured.11	26 Federally Qualified Health Centers (FQHCs) in 2 states (Oregon and California, ∼41,000 patients)	Group randomized to mailed fecal immunochemical tests (FIT) outreach and use of a real-time EHR embedded tool	Site selection phase: mix of early and late adopting sites led to distinct differences in sites
Time to Reduce Mortality in End-Stage Renal Disease trial (TiME) (NCT02019225); Status: complete	To determine whether treatment with hemodialysis sessions that are longer than many patients in the US currently receive reduces the high rate of mortality among people being treated with thrice-weekly maintenance hemodialysis.12	Two large US dialysis provider organizations (DaVita, Inc., Fresenius Medical Care – North America; 266 outpatient dialysis facilities, 7035 patients)	Cluster randomization to a default hemodialysis session duration of at least 4.25 hours or to usual care (no trial-driven approach to hemodialysis session durations)	Site selection phase: enrollment of late adopters was associated with inadequate implementation of the intervention

3. Problems and solutions

The Collaboratory ePCTs are in various phases—some are currently being designed, launched, or conducted; some are in the data analysis phase; and others have been completed. During the design, pilot, and conduct phases of these trials, a multitude of QI initiatives created different challenges across a number of the HCSs involved in the trials (Fig. 1). The following examples describe the strategies and solutions used to counter the challenges; from these examples, we further develop common, generalizable strategies and recommendations for future ePCTs.

Fig. 1.

Challenges that arose from QI activities by phase of the ePCT.

4. Design

4.1. Challenge: Many pragmatic trials take a long time to complete, and for urgent public health questions, there will be important competing QI activities

4.1.1. Case example: Pragmatic Trial of User-centered Clinical Decision Support to Implement Emergency Department-initiated Buprenorphine for Opioid Use Disorder (EMBED)

Because opioid use disorder is a national public health crisis and progress against opioid-related morbidity and mortality is sorely needed, the study team embraced QI activities at study sites as essential (and inevitable). To ameliorate the potential confounding effects of these QI activities with the ePCT, in the planning phase the study team (1) changed the design from a stepped-wedge to a group-randomized trial to shorten the duration of the trial, thereby decreasing the impact on temporal trends from emerging QI activities, (2) balanced QI activities across sites with constrained randomization, and (3) planned to track specific QI initiatives by site to determine their effect on the primary outcome. The study team felt that these pragmatic approaches might increase the generalizability of the findings given the allowances for real-world QI co-occurring with the trial.

5. Pilot phase

5.1. Challenge: Sites might adopt or modify the trial intervention before an ePCT is complete based on promising groundwork accomplished during the pilot phase

5.1.1. Case examples: Pragmatic Trial of Population-based Programs to Prevent Suicide Attempt (SPOT)

During the pilot phase of the trial, HCS leaders at one site began developing tools and workflows to support the integration of mental healthcare into routine primary care as part of a system-wide QI initiative. These leaders adapted a version of the suicide risk assessment tool the research team had used in the pilot phase for SPOT to monitor patients assigned to the care management intervention arm of the trial. At the same time, the leaders adapted the assessment tool to help ensure primary care patients who screened positive for frequent suicidal ideation received appropriate follow-up care. After trial randomization had begun, the study team collaborated with health system leaders and shared experiences to improve integration of the assessment tool into the EHR and standard primary care workflows. Patient-level randomization planned at the time of grant submission provided protection against temporal biases introduced by this QI initiative that may have been introduced had the team chosen randomization at the provider or clinic level (a common design for ePCTs).

5.1.2. Lumbar Imaging with Reporting of Epidemiology (LIRE)

During the pilot phase, one of the four HCSs independently implemented a QI intervention similar to the LIRE intervention (i.e., epidemiological benchmark text representing the normal range in imaging reports) in the hopes of decreasing inappropriate spine care. After the study team had discussions with local radiology leadership, the site agreed to remove the text from their radiology reports so as not to confound the trial. The stepped-wedge design of the trial facilitated the discussions with site leadership as all of the participating clinics would have the intervention text in the radiology reports by the trial’s conclusion.

6. Recruitment of participants

6.1. Challenge: QI activities during recruitment of an ePCT can create confusion among participants and clinicians

6.1.1. Case example: Pain Program for Active Coping and Training (PPACT)

The study team needed to be aware of, coordinate, and measure QI activities that included both 1) opioid therapy tapering- and safe use-related QI efforts, which did not directly compete with their intervention, and 2) nonpharmaco-therapy for chronic pain as an alternative for opioids, which did directly compete with the intervention. However, the simultaneous QI efforts that appeared similar to the PPACT intervention caused unexpected confusion for both frontline clinicians and potential participants who were concerned that their chronic opioid treatment might be reduced or eliminated. To counter this, at one of the participating HCSs, the investigators intensified orientation efforts to ensure that potential participants fully understood their care options and how the trial offerings fit into the broader array of pain-related services in the healthcare system.

7. Site selection

7.1. Challenge: Health systems that are early adopters of evidence are quick to change practice and have many QI activities. Late adopters may be slow to implement evidence into care and to implement research interventions

7.1.1. Case examples: Primary Palliative Care for Emergency Medicine (PRIM-ER; enrolled early adopters)

The research team targeted collaboration with early adopter emergency departments that were beginning to prioritize palliative care initiatives and had physician and nurse champions. While including motivated sites helps with implementing a complex intervention, these will likely be implementing other related programs, which may in turn impact the same outcomes of the trial. The research team designed an analysis plan that will account for this. Specifically, they will monitor QI initiatives at the site level and plan to negotiate with clinical leadership to delay or replace palliative initiatives with PRIM-ER activities. Through ongoing tracking, the goal is to support and encourage local QI while ensuring the outcomes of the trial are a result of the intervention and not parallel programs.

7.1.2. Time to Reduce Mortality in End-stage Renal Disease Trial (TiME; enrolled late adopters)

The TiME trial set out to test a longer dialysis session duration (4.25 hours) versus usual care (non-trial directed session duration). During facility selection, it was apparent that hemodialysis session durations were already increasing at many facilities operated by the dialysis provider organizations, likely in response to observational studies demonstrating associations between longer session durations and improved patient survival. Because this practice change was expected to decrease the difference in session durations between the intervention and usual care facilities, the study team decided to restrict enrollment to “late-adopter” facilities that had not already implemented longer session durations. While this approach addressed one problem, it had the unintended effect of enriching the trial for facilities that had less enthusiasm to change to practice in the absence of rigorous evidence of benefit, and thus, less willingness to broadly adopt the TiME intervention as routine care during the conduct of the trial.

7.1.3. Strategies and Opportunities to Stop Colon Cancer in Priority Populations (STOP CRC; enrolled mix of early and late adopters)

Whether or not a Federally Qualified Health Center (FQHC) was an early or late adopter to innovations in care was not part of STOP CRC’s selection criteria, although there were distinct differences between the FQHCs. For example, some clinics assigned to usual care did not want to wait to start the intervention because waiting raised some ethical and participatory issues (not wanting to offer differing care across clinics in their centers), so they were more likely to give out FIT kits at routine clinic visits than they might have been had they not been part of the study. Conversely, some sites in the intervention arm were slow to mail the FIT kits. Implementation success varied across intervention clinics, ranging from 21% to 82%, in lagged data. Although STOP CRC enrolled a mix of early and late adopter FQHCs—which had the effect of diluting the ability to detect changes due to the STOP CRC intervention, thereby decreasing the overall intervention effectiveness—the active intervention was still significantly more effective than usual care. In the per protocol analysis, intervention effect was similar to smaller trials implemented in research settings, highlighting the need to carefully design the analysis plan up front to account for these differences.

8. Conduct and analysis

8.1. Challenge: different health systems participating in the same ePCT could have varied implementation of both QI activities and the intervention during the conduct of the trial

8.1.1. Case example: Improving Chronic Disease Management with Pieces (ICD-Pieces)

All the participating health systems conducted different QI initiatives that overlapped with key components of ICD-Pieces, including the intervention, and could potentially affect the conduct and analysis of the trial. For example, one health system has implemented initiatives to promote better blood pressure control and measurement of Hemoglobin AIc. Another has patient-facing education materials for CKD. A third health system has eGFR prompts to trigger further consideration for blood pressure and lipid control medication use. The fourth system aligned provider incentives with best care practices for diabetes control. The study team continues to monitor the QI activities at each HCS for possible conflict or influence with ICD-Pieces in the intervention and control groups.

8.2. Challenge: Competing QI initiatives may impact the ability to measure the primary outcome

8.2.1. Case examples: Active Bathing to Eliminate (ABATE) Infection study

Because hospitals routinely implement new QI interventions and infection prevention is often a target of these QI strategies, the study team needed to have a process for monitoring and addressing potential conflicting QI initiatives that participating hospitals might pursue during the ePCT. As a requirement of participation in ABATE, infection prevention strategies were required to be stable in the baseline year preceding the trial and during the intervention period. Hospitals in both arms were required to report any new QI or other interventions that were being considered or launched during the trial. Reminders for reporting were provided during monthly coaching calls, and early reporting was encouraged when QI strategies were in the planning stage. All reported QI initiatives were assessed weekly by the trial’s Steering Committee. Hospitals that reported a competing intervention based upon the Steering Committee’s concern for a meaningful effect that could conflict with trial outcomes were asked to delay the QI initiatives until the trial was over, or to drop from the trial. During the 21-month trial, 196 QI interventions were reported to the Steering Committee, with 67 (34%) deemed to directly compete with trial outcomes. Three sites dropped from the trial (two in the control group and one in the intervention group) to pursue a competing intervention. Data from these sites were included in the as-randomized trial analysis, but were removed from the as-treated analysis from the time of drop-out.

8.2.2. Guiding Good Choices for Health (GGC4H)

With increasing integration of behavioral healthcare in pediatric primary care settings, parents and adolescents may be exposed during the 4-year GGC4H ePCT to parenting and behavioral health interventions other than Guiding Good Choices (GGC), implemented as part of QI initiatives. These initiatives are unlikely to be offered uniformly across clinics whether or not in the intervention or control arms, raising the possibility that GGC’s impact will be dampened, confounded, or both. The study team has developed several mechanisms to deal with this possibility. First, the team has adopted a theoretical framework-driven implementation monitoring system to record QI initiatives and other external and internal activities that could potentially impact GGC, as they occur throughout the trial. Data collected prospectively will help researchers identify and respond to challenges that arise and interpret findings at the end of the trial. Second, the adolescent behavioral health survey, administered to adolescents annually, will include questions about other behavioral health service utilization. Third, the study team includes pediatricians and embedded research teams with strong working relationships and regular communication with clinic, pediatrics, and adolescent medicine leaders. These relationships can be leveraged to understand QI activities and their motivation, and, though less likely, to influence QI implementation to avoid negative impacts on ePCT results.

8.2.3. Personalized Patient Data and Behavioral Nudges to Improve Adherence to Chronic Cardiovascular Medications (Nudge)

Nudge will provide text message reminders for patients with chronic cardiovascular disease to refill their medications. Concurrent with the study, two of the health systems implemented a medication adherence tool within the EHR where clinicians can see the refill adherence of patients. In addition, at some retail pharmacies where patients fill their medications, there are existing text message reminders sent to patients. The study is monitoring these concurrent QI processes, which should be considered co-interventions. In the analysis, the study team will consider these patients as an important subgroup in the assessment of the effect of the intervention. They will be able to determine whether patients exclusively obtain their medications within the health system pharmacy and/or through retail pharmacies. The study team also plans to assess the effect of the intervention overall as well as within the subgroup of patients who obtain medications via retail pharmacies.

8.3. Unresolved questions and lessons from the field: recommendations for researchers and QI and health system stakeholders

Researchers conducting ePCTs within HCSs have an ethical obligation to give patients the best care possible, and one way to ensure that care is evidence-based and high quality is to test interventions through an ePCT. This evidence can be used to drive broad improvements across many HCSs, change reimbursement policies, or introduce legislation to help improve the care on a population level. Many QI activities, although they tend to be smaller in scale, generally have the same goals as ePCTs. However, some QI activities may, as described above, impact an ePCT (Fig. 1): they may create confusion among participating patients, clinicians and staff; be implemented differently (or not at all) across the various systems; impact health systems differently depending on local workflows and priorities; directly compete with the conduct of the trial and intervention fidelity; increase demands on patients, staff, and resources; sway general opinion; and potentially confound the results of a trial. Solutions to challenges created by QI activities will vary depending on the nature of the trial, challenges, and health systems in which the trials are conducted.

It is important for the researcher to understand that HCS participation in ePCTs is voluntary, and “ongoing commitment, shared vision, a willingness to understand and accommodate different priorities” is critical.¹³ Leaders of HCS participate for a variety of reasons, including that evidence generation is for the greater good, research is in keeping with the mission of the HCS, as a market differentiator, as part of performance improvement initiatives, and to gain early access to new knowledge and best practices.¹⁴ Based on experiences of these ePCTs, PIs, HCS leaders, and other members of the Collaboratory have developed these recommendations to provide future researchers with a roadmap to overcoming the challenges with co-occurring QI initiatives during an ePCT, and for ensuring optimal patient care while preserving the ability to answer important health questions.

1. Collaborate with HCS stakeholders in the design stage of the trial and in the decision-making process. Continue this involvement through each phase of the research to ensure synergy and, where possible, minimize competing interventions that might confound the analysis or contaminate the results. In addition, understanding the landscape of concomitant competing interventions may be an important part of understanding the context of trial findings. Understanding the interaction between QI activities and implementation strategies provides guidance for selection of effective strategies to test in real-world settings.

2. Understand the factors that motivate HCSs to undertake QI activities. A map may be helpful to illustrate relationships among involved health system leaders, their motivations, and the multiple internal and external factors that are associated with the motivators, such as changing policy, changing payor requirements, and other possible constraints and considerations. There are formal ways of developing relationship maps, such as mapping decision makers and influencers or force field analyses used in the social sciences.¹⁵ Frameworks such as the Consolidated Framework for Implementation Research (CFIR) can also be useful in mapping the program/intervention to be evaluated, and individual, internal, and external drivers and barriers.^16,17 As an example, policy changes that incentivize health plans or clinics to achieve targets could motivate new QI activities and cause health system leaders to be reluctant to deliver inconsistent care across their clinics. Thus, they may introduce alternative QI initiatives across usual care sites so all sites are similar. As another example, in the case of FQHCs, a QI activity might be directly linked to the funding stream that established QI priorities. If researchers understand the genesis of potential tensions, they will be better able to find a solution that meets the priorities of the HCS without compromising the outcomes of the ePCT.

3. When ePCTs are embedded in HCSs where QI initiatives are common, trial investigators should establish a reporting and monitoring system to identify and address conflicting interventions. Systematic monitoring of all influences that could affect the implementation of an intervention include relevant health system QI initiatives, organizational changes, as well as policy changes and environmental factors that could affect adoption and implementation of the intervention. This is best accomplished by partnering with personnel responsible for local QI initiatives and health system leaders and by collaborating to develop a mechanism for reporting the potentially competing initiatives. Knowledge of and response to conflicting interventions is critical to ensure that the interpretation of trial results is valid. The possibility that sites may need to drop out of the trial due to competing interventions should be accounted for in trial power calculations, and potential biases introduced by such drop-out should be considered.

4. HCS stakeholders may need to be asked to delay implementing a competing QI initiative during an ongoing trial. As mentioned above, a strong durable partnership based on trust, ongoing commitment, and continuous communication is critical for this type of conversation. QI activities are typically initiated to address a need, and therefore, examining how (and if) the results of the ePCT will address this need both locally and in a broader more generalizable context are important aspects of the ongoing conversation. The success of requests to delay QI activities will be related to the importance of the ePCT question to participating sites and HCS leaders. Before making a request, researchers should consider whether the competing initiative is considered best-practice by national standards, as nationally accepted changes to best practice may need to be equally implemented across all participating arms during the course of the trial. The trial investigators should ensure equal opportunity and encouragement for such changes; training can be implemented to ensure the activities are implemented equally across all sites.

5. Ensure that statistical experts involved with the analyses are aware of QI initiatives (or plans for initiatives), so they can recommend appropriate actions and ideally, protect the validity of ePCT results. For example, investigators could consider ways to shorten the timeline both during the trial planning and trial conduct phases (e.g., stepped-wedge vs grouped cluster designs; larger sample size vs longer follow-up for outcome event accrual) in case systems are motivated to implement either a competing QI activity and/or the research intervention across the HCS.

6. Developing clear communication between the study team and the staff implementing the ePCT intervention before, during, and after implementation within the HCS is critical to success. This includes communicating results in a user-friendly way that can be used by health system stakeholders to make decisions about intervention adoption.

7. Assess the value of the ePCT intervention in the midst of all other QI initiatives and demands on the provider and HCS. This “value” can be considered through the lens of multiple stakeholders, including patients (improved care and outcomes), clinicians (streamlined workflow and processes), and healthcare systems (higher quality care and lower costs). Adoption will more likely occur if the intervention is relevant to multiple stakeholders, such as an improvement that will save the provider time, be patient-centered, and decrease overall costs for the HCS.

9. Conclusions

Health systems are complex, dynamic and constantly evolving. QI implementation within HCSs will continue and, therefore, continue to be a challenge for conducting ePCTs within HCSs. Elucidating an experimental effect in an ePCT can be challenging even without co-occurring or competing QI initiatives. In general, because the interventions in pragmatic trials are designed for heterogenous settings that change over time, QI activities might lead to a dilution of the potential impact of the intervention. This might make it more likely that a pragmatic trial will have a negative, diminished, or inconclusive result compared with an explanatory trial. This happened in several of the NIH Collaboratory trials.

Although there is an ethical imperative to protect the integrity of the trial for the development of much needed evidence, there is also a primary obligation to protect the well-being of participants and provide high-quality care.¹⁸ For ePCTs to be rigorous, study teams must monitor, adapt, and respond to QI during the design and the trial implementation. Both ePCTs and QI happen within the same context and aim to improve patient care, they are inherently interconnected. Indeed, the distinction between QI activities and ePCTs is arguably fuzzy.¹⁹ As we transition from a construct where research is conducted separately from healthcare to one where research is a part of continuous learning, as in a learning healthcare system,²⁰ we expect to find more synergy between QI and research and more robust partnerships and collaboration among those responsible for QI, healthcare, and research. Therefore, routine collaboration between ePCT researchers and HCS stakeholders are critically important for optimally aligning research with QI to support high-quality patient-centered care. Ideally, ePCTs should adapt to best practice changes so that the comparator is always compared to best practice (e.g., preventing out-of-date results at publication). Therefore, in addition to avoiding unnecessary conflicting QI interventions, ePCTs also need to embrace best practice change so that the trial intervention is implemented against a background or comparator of best practice.