Skip to main content
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2021 Oct 1;28(12):2626–2640. doi: 10.1093/jamia/ocab202

Enhancing the use of EHR systems for pragmatic embedded research: lessons from the NIH Health Care Systems Research Collaboratory

Rachel L Richesson 1,, Keith S Marsolo 2, Brian J Douthit 3,4, Karen Staman 5, P Michael Ho 6, Dana Dailey 7, Andrew D Boyd 8, Kathleen M McTigue 9, Miriam O Ezenwa 10, Judith M Schlaeger 11, Crystal L Patil 11, Keturah R Faurot 12, Leah Tuzzio 13, Eric B Larson 13, Emily C O’Brien 2, Christina K Zigler 2, Joshua R Lakin 14, Alice R Pressman 15, Jordan M Braciszewski 16, Corita Grudzen 17, Guilherme Del Fiol 18
PMCID: PMC8633608  PMID: 34597383

Abstract

Objective

We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research.

Materials and Methods

Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements.

Results

We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data.

Discussion

Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries.

Conclusion

We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.

Keywords: electronic health records, pragmatic clinical trials, learning health systems, data standards

INTRODUCTION

Pragmatic clinical trials (PCTs) are randomized controlled trials designed for generalizability, often involving multiple clinical sites with broad eligibility criteria.1,2 Their advantage is the ability to determine if health interventions actually work in practice, and hence such trials can generate “real-world evidence” to inform implementation, clinical practice, and regulatory decision-making.3,4 PCTs are “pragmatic” in that they are embedded in the workflows of diverse healthcare systems and aim to leverage existing data streams in the electronic health record (EHR) to limit the costs and burden of research data collection while maximizing the ability to answer important clinical questions and deliver better-quality care.5

The Health Care Systems Research Collaboratory (hereafter, “Collaboratory”) is supported by the National Institutes of Health (NIH) Common Fund and was designed to strengthen the national capacity to implement cost-effective, large-scale PCTs that are embedded in routine care and conducted in partnership healthcare systems.6,7 The trials often involve cluster randomization (of hospitals, clinics, or primary care providers, etc.), the interventions may be implemented by health system personnel, and data are collected as part of routine clinical care. The Collaboratory is currently supporting 21 large-scale, high-impact PCTs (or Demonstration Projects) that address issues of major health importance, such as hospital-based infections, chronic pain, the opioid crisis, colorectal cancer, medication adherence, and suicide,8–32 all in different phases of completion (Table 1). The portfolio includes 6 new projects that are part of the Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing (PRISM), which are part of the HEALSM (Helping to End Addiction Long Term) Initiative. To apply lessons learned from ongoing projects and to provide expertise to the Demonstration Projects, 7 Core working groups were created: EHRs; phenotypes, data standards, and data quality; patient-reported outcomes; healthcare systems interactions; ethics and regulatory; biostatistics and study design; and stakeholder engagement cores. A centralized Coordinating Center and the Core groups assist with the design, conduct, and logistical challenges of the projects and disseminate generalizable knowledge about PCT methods and enabling factors.

Table 1.

Collaboratory projects and EHR functionality

Study name Project goal Setting/population EHR functionality and data types Status (at the time of survey)
Determine if using antiseptic bathing for all patients and nasal ointments for patients harboring methicillin-resistant Staphylococcus aureus (MRSA) reduces multidrug-resistant organisms and bloodstream infections 53 hospitals/339 902 patients in adult medical, surgical, oncology, and step-down units Census, demographics, administrative codes, nursing documentation, and extensive microbiology results. Completed
  • ACP PEACE16

  • Advance Care Planning: Promoting Effective and Aligned Communication in the Elderly

  • NCT03609177

Test whether clinician communication skills training and patient video decision aids increase completion of advance care planning 36 oncology clinics across 3 health systems/Patients >65 years of age with advanced cancer (∼12 000 patients and clinicians) Free-text notes Recruiting
  • BackInAction26

  • Pragmatic Trial of Acupuncture for Chronic Low Back Pain in Older Adults

Evaluate the effectiveness of acupuncture in older adults with chronic low back pain 4 performance sites/789 adults ≥65 years of age with chronic low back pain Back-end functionality to extract data on potential recruits, and utilization outcomes data for participants. Planning
  • BeatPain Utah

  • Nonpharmacologic Pain Management in Federally Qualified Health Centers (FQHCs) Primary Care Clinics

To compare the effectiveness of nonpharmacologic intervention strategies for patients with back pain. FQHCs throughout the state of Utah Demographics, encounter diagnosis/problem list, and medications Planning
  • EMBED17

  • Pragmatic Trial of User-Centered Clinical Decision Support to Implement Emergency Department-Initiated Buprenorphine for Opioid Use Disorder17

  • NCT03658642

Test the effectiveness of user-centered computerized clinical decision support on rates of emergency department–initiated buprenorphine/naloxone and referral for ongoing medication-assisted treatment in patients with opioid use disorder 4000 patients in 18 sites in 5 large health systems. clinical decision support target is clinicians treating adult patients with opioid use disorder in the emergency department Dozens of data elements, which are not standardized between healthcare systems Not yet recruiting (at time of survey; recruiting now)
  • FM TIPS25

  • Fibromyalgia TENS in Physical Therapy Study

  • NCT04683042

Test the feasibility and effectiveness of adding transcutaneous electrical nerve stimulation (TENS) nonpharmacologic treatment for pain and fatigue in patients with fibromyalgia (FM) 24 routine physical therapy clinics and 5 health systems in rural and urban settings/∼600 patients with FM
  • Episode of care data: identification of participation in research study, flag of fibromyalgia diagnosis; identifiers for patient name and date of birth; start and end date for episode of care, ICD 10 diagnostic codes, number of physical therapy visits scheduled, number of physical therapy visits attended.

  • Patient visit level data: CPT Codes per visit; outcome measure patient-specific functional scale on PT Visit 1 and every 30 days while in PT

Planning
  • GGC4H

  • Pragmatic Trial of Parent-Focused Prevention in Pediatric Primary Care: Implementation and Adolescent Health Outcomes in Three Health Systems (GGC4H: Guiding Good Choices for Health)

  • NCT04040153

Test the feasibility and effectiveness of implementing a universal evidence-based anticipatory guidance curriculum (Guiding Good Choices) for parents of early adolescents 3 large integrated health systems; 75 sites/Pediatric primary care practices serving families with adolescents 11-12 years of age (∼3600 adolescents and families) Diagnoses, symptoms, procedures, encounters, prescriptions (orders and fills), settings, provider characteristics, and demographics variables (eg, sex, race/ethnicity, insurance type) Not yet recruiting (at time of survey; recruiting now)
  • GRACE24

  • Hybrid Effectiveness-Implementation Trial of Guided Relaxation and Acupuncture for Chronic Sickle Cell Disease Pain

A hybrid type 1 effectiveness-implementation trial to assess the effectiveness of acupuncture and guided relaxation on individuals with sickle cell disease while observing and gathering information on implementation in 3 health systems. 3 performance sites/366 individuals with sickle cell disease and chronic pain Recruitment of patients, integration of 4 patient-reported outcomes into clinical care Planning
  • HiLo27

  • Pragmatic Trial of Higher vs Lower Serum Phosphate Targets in Patients Undergoing Hemodialysis

  • NCT04095039

Test the effects of liberalizing the serum phosphate target (“Hi”) versus maintaining aggressive phosphate control (“Lo”) 2 large dialysis provider organizations; 100 sites/∼4400 patients with end-stage renal disease receiving maintenance hemodialysis Automated data on phosphate binder prescription changes Not yet recruiting (at time of survey; recruiting now)
  • ICD-Pieces

  • Improving Chronic Disease management with Pieces

  • NCT02587936

Improve care for patients with chronic kidney disease, diabetes, and hypertension by using a novel technology platform (Pieces) that uses the EHR to identify patients and by assigning practice facilitators within primary care practices or community medical homes 4 health systems; 143 sites/∼11 000 patients with multiple comorbid conditions (chronic kidney disease, diabetes, and hypertension)
  • Medication Rx

  • Vital sign data

  • ICD10CM/ICD10PCD hospital claims data.

  • ICD10CM problem list

  • ICD10CM encounter billed diagnosis

Enrollment completed
Determine if inserting epidemiological benchmarks (essentially representing the normal range) into lumbar spine imaging reports reduces subsequent tests and treatments 4 health systems; 98 clinics/250 401 patients with low back pain ICD-9 and 10 diagnosis and procedure codes; CPT codes; pharmacy prescription and filled data; radiology images and reports; Completed
  • NOHARM

  • Nonpharmacologic Options in Postoperative Hospital-based and Rehabilitation Pain Management

  • NCT04570371

Evaluate the feasibility of EHR-embedded patient- and clinician-facing decision support for nonpharmacologic pain care after surgery 4 large integrated health systems; 22 practice clusters/∼54 000 postoperative patients Epic PROM collection capabilities via MyChart portal and tablet at point of care; portal questionnaire functionality to embed an interactive conversation guide; Clinical decision support to provide alerts, prompt actions; nursing inpatient educational activities; discharge summaries and processes; registries to drive automated, individualized messaging Planning
  • Nudge28,29

  • Personalized Patient Data and Behavioral Nudges to Improve Adherence to Chronic Cardiovascular Medications

  • NCT03973931

Use pharmacy refill data to test effectiveness of medication reminder nudges delivered to patients’ mobile phones, and test an interactive chat bot mechanism that optimizes content 3 large integrated health systems/14 700 patients with chronic cardiovascular conditions who take medications ICD diagnosis codes and pharmacy refill data.

Recruiting

  • OPTIMUM

  • Group-Based Mindfulness for Patients With Chronic Low Back Pain in the Primary Care Setting

  • NCT04129450

Evaluate a group-based mindfulness program (mindfulness-based stress reduction) for patients with chronic low back pain within primary care Primary care clinics in 3 health systems/∼450 patients with chronic low back pain
  • Use of EHR-based data to generate recruitment lists;

  • Referral orders/flags for enrollment, routed to study staff

  • Extraction of data on demographics, diagnoses, procedures, medication prescribing, and health service utilization as study covariates and outcomes

Planning
  • PPACT10

  • Pain Program for Active Coping and Training

  • NCT02113592

Help patients adopt self-management skills for chronic pain, limit use of opioid medications, and identify factors amenable to treatment in the primary care setting 3 staff model health plans; 106 primary care providers/860 patients with chronic pain on long-term opioid therapy Patients eligibility was based on long-term opioid treatment receipt for pain, also used for PCP clustering, and for health services/cost analyses and identification of moderators (clinical and demographic characteristics). Completed
Test effectiveness of primary palliative care education, training, and technical support for emergency medicine 33 emergency departments/patients ≥66 years of age in the emergency department with serious, life-limiting illness (4983 providers) P-CaRES screening process: (1) life-limiting conditions, including end-stage organ disease, advanced cancer, septic shock or multiorgan failure in elderly patients, or a high chance of accelerated death (eg, cardiac arrest); (2) functional decline, uncontrolled symptoms, caregiver distress, or provider gestalt regarding limited prognosis31,33 Not yet recruiting (at time of survey; recruiting now)
  • PROVEN18,21

  • Pragmatic Trial of Video Education in Nursing Homes

  • NCT02612688

Determine if showing advanced care planning videos in nursing homes affects the rates of hospitalization 2 nursing home health systems; 359 nursing homes/nursing home health systems serving long-stay (>12 months) patients with advanced comorbid conditions (166 196 patients) An identical form was introduced into the EMR of each partner Completed
Compare outcomes in patients who receive care-management or online skills training for suicide prevention versus usual care 4 large health systems/18 889 individuals at elevated risk for suicide on a depression scale
  • Encounter diagnoses

  • PHQ9 scores

Analysis in progress
  • STOP CRC8

  • Strategies and Opportunities to Stop Colon Cancer in Priority Populations

  • NCT01742065

Improve the rates of colorectal cancer screening by mailing fecal immunochemical testing tests to patients at federally qualified health centers 26 federally qualified health center clinics/62 155 individuals eligible for colorectal screening per the US Preventive Task Force guidelines A real-time registry in the EHR was used to determine those eligible for screening34. Through a validation study35, accuracy of EHR data was confirmed (88% positive predictive value for identifying eligible participates). Completed
  • TiME11

  • Time to Reduce Mortality in End-Stage Renal Disease

  • NCT02019225

To determine whether increasing hemodialysis session duration reduces mortality and hospitalization rates for patients receiving maintenance hemodialysis care 266 dialysis facilities operated by 2 dialysis provider organizations/7035 adults who have initiated treatment with maintenance hemodialysis within the past 120 days All of the data elements, including outcomes, were obtained from the EHR, including death, hospitalization rate, predialysis BP, postdialysis hypotension, interdialytic weight gain, fluid removal rate, missed dialysis sessions, and change in quality of life as assessed by the Kidney Disease Quality of Life Short Form-36.11 Completed
To coordinate care and improve outcomes for trauma survivors with post-traumatic stress disorder (PTSD) and comorbidity and to provide the American College of Surgeons with multisite pragmatic trial evidence that could further inform regulatory policy 25 US level 1 trauma centers/635 trauma survivors with PTSD and comorbidity Baseline 10-questions PTSD screening, PTSD Checklist, Patient Health Questionnaire, PHQ-9, Alcohol Use Disorder Identification Test (AUDIT), Medical Outcomes Study Short Form (MOS SF) SF-12, Abbreviated Injury Scale (AIS). ICD codes for traumatic brain injury, and comorbid conditions Completed

EHR: electronic health record.

In 2017, the EHR Core reported the challenges from the Collaboratory’s first round of 9 Demonstration Projects. Four broad informatics challenges were identified, including: “(1) using clinical data for research, (2) integrating data from heterogeneous systems, (3) using EHRs to support intervention delivery or health system change, and (4) assessing and improving data capture to define study populations and outcomes.”20,36

Using qualitative data generated from an electronic survey completed by lead scientists for each of the 21 PCT projects in the Collaboratory portfolio, we update and elaborate on the EHR-related challenges. This collection of examples and prerequisites for successful ePCTs aim to improve the ability to use data collected during routine care to support research37 and generate new knowledge in the form of implementable clinical interventions to address important public health issues.

MATERIALS AND METHODS

Study design and population

This online cross-sectional and semistructured survey that included 4 open-ended questions was circulated to all 21 Collaboratory Demonstration Projects (including active and completed studies and 2 new projects in early design stage).8–32 Collectively, these PCTs in the Collaboratory address a range of critical public health problems, including the opioid crisis, suicide, and colorectal cancer, using a variety of interventions, research designs, settings, and patient populations (Table 1).

Survey content and administration

The semistructured survey was created by members of the EHR Core and consisted of 20 questions, informed by known challenges.20,36,37 The survey included 4 open-ended questions: (1) Is there EHR functionality that you would like to have for your study, but didn’t? If yes, please explain. (2) Which EHR data elements are you using for your study? To the best of your knowledge, are they standardized across the EHR systems used in your study? (3) What key question(s) would you ask of EHR vendors to assess their readiness to support research? and (4) What data or functions would you like to see standardized across all EHR systems? Sixteen structured questions pertained to challenges relating specifically to data and data access issues, requiring either “yes” or “no” responses (Supplementary Table S1). If the participants answered “yes”, then they were prompted to elaborate on their experiences. Completion of the survey was voluntary. Data were collected from Collaboratory demonstration project representatives between August 20, 2020, and June 30, 2021.

Data analysis

The responses from participants on the open-ended questions amassed a large amount of rich text, and we conducted a thematic analysis38 on the resulting data. We coded narrative responses from the 4 open-ended questions directed to each project. First, RLR prepared the data for coding by collating and unitizing the responses. Next, coding was completed by RLR to develop an initial codebook. A second coder (KSM) reviewed the initial coding schema and suggested changes as applicable. Group consensus was achieved for each coding unit, resulting in Cohen’s kappa coefficient of 0.72. The resulting 21 codes were iteratively grouped by RLR and BJD into families, eventually resulting in the final 6 themes. This process is summarized in Figure 1.

Figure 1.

Figure 1.

Process for determining themes for EHR challenges in collaboratory projects. EHR: electronic health record.

The consensus themes and underlying codes were compiled and presented to the Collaboratory EHR Core for discussion between November 2020 and December 2020. During these meetings, the EHR Core members provided clarification on the 6 themes and discussed their implications.

Development of prerequisites to facilitate PCTs

The EHR Core used additional calls to brainstorm and discuss strategies and recommendations for enhancing EHR systems to support PCTs. We received input from our expert and diverse members of the EHR Core, Healthcare Systems Interactions Core, and Patient-Reported Outcomes Core. Through the circulation and collaborative writing of this report, members of the EHR, Patient-Reported Outcomes, and HCS Interactions Cores contributed to the list of prerequisites for PCTs and recommended EHR enhancements to improve capacity for pragmatic trials.

RESULTS

A total of 20 out of 21 (95%) Demonstration Projects completed the survey shown in Table 2.

Table 2.

Reported challenges related to using EHR data in pragmatic trials

Survey Question: (Yes/No) Did you experience any challenges related to: # Projects reporting: n = 20 New projects n = 2 (10%) Early phase projectsa n = 4 (20%) Middle phaseb projects n = 7 (35%) Complete or near complete n = 7 (35%)
IT staff turnover 11 (55%) 1 (5%) 1 (5%) 3 (15%) 6 (30%)
Integrating data from heterogeneous systems 11 (55%) 1 (5%) 2 (10%) 4 (20%) 4 (20%)
Using EHRs to support intervention delivery or health system change 10 (50%) 2 (10%) 1 (5%) 3 (15%) 4 (20%)
Missing data 8 (40%) 0 3 (15%) 0 5 (25%)
Assessing the validity or accuracy of EHR data 7 (35%) 0 2 (10%) 1 (5%) 4 (20%)
Acquiring data 7 (35%) 1 (5%) 2 (10%) 2 (10%) 2 (10%)
Use of free-text or narrative data from EHR 6 (30%) 0 0 2 (10%) 4 (20%)
Data cleaning 6 (30%) 0 1 (5%) 1 (5%) 4 (20%)
Completeness of data across EHRs 6 (30%) 0 2 (10%) 1 (5%) 3 (15%)
Interoperability 5 (25%) 1 (5%) 0 1 (5%) 3 (15%)
Data quality assessment 5 (25%) 0 1 (5%) 0 4 (20%)
Requesting data (ie, knowing which data are available) 5 (25%) 1 2 (10%) 0 2 (10%)
Identification and use of clinical phenotype definitions 4 (20%) 0 1 (5%) 1 (5%) 2 (10%)
Determining or negotiating the frequency of data pulls 4 (20%) 0 0 1 (5%) 3 (15%)
Gaining permission to use data 2 (10%) 0 1 (5%) 0 1 (5%)
Need to upload data or write to EHRs 1 (5%) 0 1 (5%) 0 0
a

Early-phase projects are just transiting from planning to implementing the trial.

b

Middle-phase projects are enrolling and collecting data but early; complete or near complete are nearly done with enrollment.

EHR: electronic health record.

The specific EHRs across study sites are presented in Table 3.

Table 3.

Characterization of EHR use across Collaboratory demonstration projects

PRISM projects
Active projects
Completed projects
Back in Action BeatPain Utah GRACE FM TIPS NOHARM OPTIMUM ACP-PEACE EMBED GGC4H HiLo ICD-Pieces Nudge PRIM-ER PROVEN SPOT TSOS ABATE PPACT LIRE STOP-CRC TiME
Use of EHR data
 Determine eligibility x x X x x x x x x x x x x x x x x x x
 Outcomes x x X x x x x x x x x x x x x x x x x
 Patient-reported outcomes x x
 Part of intervention delivery x x x x x x x x x x x x x x
EHR Systems/Data Source
Epic x x X x x x x x x x x x x x x
 5 different physical therapy systems x
 Allscripts x x
 American systems x
 AthenaHealth x x
 Cerner x x x x
 CPRS (VA) x x
 Custom
 DaVita x x
 Fresenius x
 HSCRN-virtual data warehouse x
 Mayo clinic proprietary x
 Medicare claims x
 MEDITECH x
 Point click care x
 Trauma registry x
eClinicalWorks (eCW) x
CompuGroup x

EHR: electronic health record.

The responses to the open-ended questions included 82 unique ideas that were compiled into 21 consensus codes and further refined into 6 broad themes (Figure 1).

Survey themes

Theme 1: Inadequate collection and integration of patient-centered data, including patient-reported outcomes, questionnaires, and documentation of services received

Seven projects—particularly those researching management of pain or psychological trauma—voiced a need for patient-centered data within health systems and EHRs. These studies reported having to collect primary data that were expected to be readily available, including patient-reported outcomes (PROs), previously completed questionnaires, and advance care planning conversations.16 This led to time- and monetary-consuming adjustments as illustrated by a project using Epic when they said, “We would have appreciated greater ease and flexibility of automatically responding to patient-reported information, including PROs to match participants’ needs. Automatically assigning the intervention to specific patients using Epic EHR logic and based on EHR data was clunky, time consuming, and expensive. We wound up using the MyChart questionnaire functionality, but this was cumbersome, and the user experience was not optimal.”

Another study reported that even if patient-centered data were collected, it was not done in a standardized manner across health systems, preventing the ready use of these data for multisite pragmatic studies.

We hoped to rely on self-report/patient reported outcomes that occur during pediatric well visits… many of the items were different across systems and there was a great deal of missing data for these data at each health system.

Finally, while some PROs were collected, not all the PROs needed for a trial were:

We had 8 patient reported outcomes (PROs) that we wanted patients to enter into the EHR. … Only 4 of the PROs were included in the EHR, the other 4 will be collected by REDCap.

Theme 2: Lack of structured research data collection, standard EHR functions, usable interfaces, and standard workflows

Several respondents commented on their experiences with data collection in the EHR environment. Some reports were positive, as the EHR allowed the structured collection of research data, making it easier to collate, analyze, and share across study sites. One study reported that even when data collection was standardized and integrated into the EHR, there were still varying levels of missing data across sites, and another reported notable errors in the accuracy of structured data.39 Several projects shared requirements for very specific EHR functionality (eg, prescribing educational videos to patients, notifications to detect changes in medication dosages, customized communications to patients though patient portals, provider alerts regarding state-specific requirements for life-sustaining treatment) that was not readily in place, necessitating the creation of new EHR-based tools or manual extraction of specific data. One respondent suggested that the variability in EHR functionality across sites can introduce potential usability concerns, potentially affect fidelity to the intervention,40 and hinder scalability of clinical interventions.

We had to use different aspects of EHRs to implement our intervention. Some used the EHR with a static message, another the EHR with a dynamic “pop-up”, one used the Radiology Information System (RIS)/dictation module. Making all of these approaches work smoothly was a challenge.

The EHRs we are working with have little customization capabilities for clinical decision support reminders and alerts. Two of them do not even allow clients to add any new reminders/alerts. Clients can only turn on/off reminder/alerts that are implemented by the vendor and those are often suboptimal. Also, the user interface of reminders/alerts is very basic and does not support actions such as placing an order.

Another project commented on scalability when they stated that, “Better integration of a user-centered interface that is scalable across health systems and EHR vendors [is important for data collection].”

Theme 3: Lack of standardization of structured data for research

Ten studies reported challenges with standardized data across sites. Studies had challenges with standard data elements (n = 5), standard coding (n = 2), inconsistent use of standard coding (n = 2), and lack of standardized or structured (imaging) reports (n = 1). Several studies reported that their study would benefit from the adoption and use of standard code systems—specifically for laboratory test results, procedures, medications, and indications for imaging procedures, even across health systems that use the same EHR product.

The codes are for the most part standardized (same codes used across sites), but their implementation is not. Some settings used ICD-procedure codes and others CPT for the same procedures… One site did not use CPT codes but instead their own proprietary procedure codes. Sites had different limits to the number of diagnosis codes reported for encounters and some sites did not identify primary diagnosis code. Encounter type codes were inconsistent across sites.

Others reported challenges not only in the adoption of standard code systems but also in the consistent use of them. The variation across sites in turn necessitated further processing—and significant resources and time—to aggregate and analyze the data.

We use several EHR data elements. However, they are not fully standardized across systems using same vendor—due to nuances of local build. This is rate limiting in terms of EHR research.

Several projects reported that standard data elements or patient screening and assessment tools would have been useful for their studies. In some cases, these measures exist and in other cases would require the medical and clinical practice communities to develop and advocate for their development and adoption as standard practice. Respondents from a pediatric—adolescent study reported that research and practice would benefit from:

Well Child, Well Adolescent, Adult primary care visit templates using brief, validated, evidence-based screening and assessment tools, … including “gold-standard” behavioral health (mental health and alcohol and drug) measures and clinical decision response algorithms (eg, NIAAA or AUDIT alcohol screening, S2BI or CRAFFT substance use screeners for adolescents, PHQ-2 or 9 + Columbia Suicide Screener, etc.).

The team also suggests we: “Work towards consensus on sets of standardized core clinical measures, perhaps endorsed by bodies such as AAP or AMA or USPSTF [American Academy of Pediatrics, American Medical Association, United States Preventative Services Task Force], perhaps with CMS [Centers for Medicare and Medicaid Services] input, that major health systems could get onboard with, and build them into the platforms.”

Theme 4: Lack of resources to support customization of EHR systems for PCTs

Nine studies identified tools and functions to support research that would be useful for their studies and pragmatic intervention. These included the ability to order an intervention, ability for local customization of EHRs (presumably faster and cheaper than if the EHR company does the work), standardized patient summarizes, configuration of EHR for research (for provider alerts about potentially eligible subjects; tracking patients on protocols; patient portal enhancements, including personalized messaging and randomization; and for research).

Respondents were able to identify several ways that the health system could have supported the EHR configuration for their trial but did not for various reasons, including doubt that the health system had the available resources to do so.

There may be some tension between what helps with clinical research and facility whereas lots of (understandable) caution on clinical delivery system side to allow too much tinkering that may overburden clinical workflow, so I think improved integration of PROs would be terrific but know this can be seen as a double edged sword.

Similarly, another project stated:

Our issues are less about Epic and its capabilities, and more about bandwidth of providers and patients and the supremacy of clinical needs over research needs.

A common feature is that IT departments have demands on them for patient care and billing, and therefore allocating IT resources to research is often not a priority.

Our trial required extensive support from IT programmers and report writers to enable data collection, cleaning, and analysis. EHR vendors allow extensive end user modification and specification that causes end-user engagement in report generation.

Eleven studies reported challenges with IT staff turnover and getting local IT resources to support their project.

Skilled technical staff may leave for better positions—in our health systems or outside companies.

A common feature is that IT departments have demands on them for patient care and billing, and therefore allocating IT resources to research is often not a priority, and a request might take months or years to be started. These challenges seemed to reflect communication issues related to prioritization.

One of the biggest barriers we have experienced is the CMIO and high-level leadership has committed to working on this research study, but when the time comes to actually implement the [EHR alerts] there is a communication gap between the CMIO/Leadership and the IT analyst doing the work. …the person doing the work was not privy to all the previous conversations nor do they realize there is a specific timeline that needs to be followed. This has created some slight delays and a lot of additional and unnecessary conversations.

Another project stated,

Time allocation/resources is very important. If a site has other pressing priorities its difficult sometimes to get an IT analyst dedicated to work on the research items even with a subcontract.

Theme 5: Difficulties aggregating multidata type resources for multisite trials

While many trials were able to collect the data they needed, the aggregation of all the data from multiple sites presented as a common challenge. This presented as both challenges in sites that are not associated with the health system, and as data elements without common models or standards.

We are using pharmacy refill data to identify patients and deliver the intervention to patients. Pharmacy refill data comes from both internal pharmacy data as well as external pharmacies. We are currently able to get daily pharmacy refill data from health system internal pharmacies. For prescriptions filled pharmacies outside of the health system, we are not able to obtain daily pharmacy refill data currently.

Eleven studies faced challenges with the capacity to integrate their data with external sources and to accurately match data across these sources. Among these are challenges with integration with internal/external data sources, access to primary care provider practice data, identity matching across data sources, and lack of integration between care settings (specifically for dosage and medication changes).

[.challenges with]… Integration with specialty care, and procedures. Specifically access to colonoscopy outcomes and pathology lab data from biopsies. Claims data and cancer registry data would also been useful for exploratory outcomes. It would be useful to access data from [Epic's health information exchange module] Care Everywhere for research.

Data harmonization across different EHR products and even the same EHR at different health systems is very challenging and labor intensive.

Theme 6: Difficult and inefficient EHR data access: lack of re-usable and automated queries to support PCTs

Eight studies reported challenges with access to data for analysis. Several survey respondents expressed difficulties with automated data pulls from sites and also coordinating data across sites that update and send data at different intervals. These studies also suggested more widespread use of data warehouses and better access to data, with standard approaches to storing and retrieving data, including common data model (CDM) support.

Using the [site’s] VDW [Virtual Data Warehouse] was problematic due to variation in frequency of updates. Similarly, there were differential and sometimes substantial delays as to when EHR data became available.

…it would be extremely useful to be able to access medication dispensing records through our EHR data.

In addition to standardized/reusable data queries, we had requests for data cleaning support, data quality (including timeliness), and information around data provenance.

It makes a difference if the data is coming from a data warehouse group that has a layer of management of those data rather than a pure export of existing fields. Understanding latency is important. There are also quality differences at the item level, and making that transparent is important.

Other studies reported challenges and lack of tools to pull data from the EHR during the study to support the intervention and conduct of the trials.

We wanted to automate data delivery from EHR systems in order to reduce burden on site research coordinators, but very few study sites used EHRs that were built with an enterprise data warehouse that would permit straightforward data pulls, so complicated workarounds, manual data transcription, and complex electronic reports had to be created.

We had wanted to use the RedCap integration with [EHR product], but it does not pass the Privacy and Safety requirements for 3 of our 4 sites, and for the 4th it won’t pull custom data points from [EHR]. They are limited to pulling predetermined data elements.

Not all of our sites are able to download medication use and healthcare utilization data. One site, a Federally Qualified Health Center (FQHC), has an EHR with limited functionality. In addition, patients from the FQHC may be seen at local hospitals that are not part of the same health system, further complicating the automatic data extractions. Currently, automatic extraction is prohibited across systems via local agreements. Hence, we plan ‘hand extractions’ of EHR data.

PREREQUISITES FOR CONDUCTING PCTS

The EHR Core, PRO Core, and Health Care Systems Interactions (HCS) Core used the thematic EHR-related challenges reported in Collaboratory PCTs to identify prerequisites and recommendations for successful PCTs, which includes enhancing EHR systems to support PCTs. These are provided below.

Integrate the collection of patient-centered data, including PROs, questionnaires, and advance care plans, into EHR systems and clinical workflows to support pragmatic research and personalized clinical care

Healthcare systems need incentives to invest in the routine collection of PROs, and EHR vendors and the Office of the National Coordinator (ONC) for Health Information Technology could better champion, require, enable, or support the collection of PROs. Potential benefits for integrating PROs into the EHR include increasing clinician/patient communication,41,42 improving patient satisfaction with care,42 and increasing symptom monitoring, which can improve clinical outcomes, such as survival.43 In addition, participant-prioritized outcomes (eg, anxiety, pain) might differ from traditional clinical outcomes44 and help with risk prediction.45 Research demonstrates that integration of PROs into the EHR is possible in a number of different types of care settings,46,47 and facilitating additional measures for research is becoming more of a priority.48–50 If PROs have not been established before a PCT is being conducted, clinics could potentially use the trial to help identify domains to include in their EHR, although there is great variability in what patients think is important depending on the disease or condition. However, integrating PROs into the EHR can change workflow and requires buy-in from multiple levels,50–52 including input from clinicians and administrators at each site, as well as alignment with the priorities and the mission of healthcare system, and this is not trivial to accomplish. Finally, integration of PROs requires careful consideration and prioritization of what to measure and how to measure it, as sometimes PROs have not shown any effects/improvements in pragmatic settings.53 Some groups have started to compile recommendations on how to successfully incorporate PROs into the EHR,54,55 and a staged approach is often helpful. Ideally, systems should identify standard mechanisms through which PROs can be added and collected for specific projects, as adding too many PROs can add to burden in the clinical workflow, and it is hard to predict what PROs a future study might need.

Facilitate high-quality structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows

Principal Investigators (PIs) and their teams requested the ability for local customization of EHRs, standardized patient screening and assessments, and configuration of EHRs for research (eg, to provide alerts for potentially eligible subjects or providers, track patients on protocols, and patient portal enhancements, including personalized messaging and randomization, and for research). In many cases, the EHR functionality (eg, flagging potential or enrolled research subjects) that researchers requested was available in the EHR, suggesting that organizations were either unaware, under-resourced to implement them, or do not prioritize the use of EHR data in research.

Many of our challenges are not with the EHRs but with the health systems not using functionality. Although low-resourced settings might need a considerable amount of help to implement and maintain EHR functions, leveraging preloaded standardized EHR functions can accelerate implementation of innovative and transformative clinical practices and aid in generation of data from patient populations, which can contribute to improving the delivery of health care, as in a Learning Health System.56

Ensure adequate IT and other staff or services to support embedded research

The primary role of healthcare organizations is to provide the best patient care possible at the lowest cost, and conducting research is a secondary goal. Staff turnover has been consistently reported as a challenge for Collaboratory PCTs, particularly IT staff turnover.57 All of these activities require local IT support, and local IT departments prioritize clinical care and billing, so obtaining IT support can be a challenge, a problem (understandably) exacerbated by the urgent needs created by COVID-19.

Devoting sufficient resources to IT support of PCTs is needed, and some members of our EHR Core suggested that research teams could hire IT staff solely dedicated to research; this could occur at the institutional level and be funded by research dollars. However, some of our investigators who have done this already report that the amount of funding is negligible next to the institution’s bottom line, and this research-charge model might reduce the amount of organizational HIT funding (in response to transitory grant funding which eventually end), possibly acting as a disincentive. One hired personnel with experience with the most common EHRs and used them to work with FQHCs to implement and configure EHR functionality.

Prioritize the creation of structured research data by using and promoting standards

Healthcare organizations could set priorities for IT staff so they increase use EHR data standards to support quality healthcare delivery, continuous quality improvement, and the generation of new knowledge. A main incentive comes from policymakers and regulations that affect payment, such as Uniform Data Systems (UDS) reporting for FQHCs and Meaningful Use requirements by the Centers for Medicare and Medicaid Services (CMS).58,59

Relevant standards include standard data elements, standardized coding systems and terminology, or structured reports, any of which could be incorporated into common EHR builds. Health systems can request that vendors improve their systems, and perhaps the most efficient way to achieve this is to synergize efforts with agencies that impose requirements tied to healthcare payments, such as CMS/Healthcare Effectiveness Data and Information Set

(HEDIS) quality measures and the United States Core Data for Interoperability (USCDI) data elements set by the ONC/CMS.60

Create aggregate, multidata type resources for multisite trials

Because our pragmatic trials were designed to minimize burden of de novo data collection, all make use of current data in the EHRs, but many required other data types and sources, such as commercial pharmacy data, practice data (on provider and practice features), and PROs, which needed to be accessed and linked. The aggregation of all the data from multiple sites presented a common challenge, due to both the fact that different systems might collect data using different data elements and have varying adoption of different CDMs. Both complicate the development of systems and interfaces for researchers to access the data required for multisite trials.

Create re-usable and automated queries to support PCTs

Respondents also reported the need to re-access data throughout the lifespan of their trials, and new queries, or queries of multiple sites, was required. A need for ongoing support for queries (or potentially an automated data pull) was voiced by the respondents. Data are configured differently for the “same” clinical uses across multiple sites, and this makes developing queries even more complex. EHR infrastructure should be designed so that data query and output is efficient and research friendly. If the future is about big data science and using existing data, then we have to a priori create a system that supports that vision.

As we noted in our previous publications,20,36 the lack of standardized data and EHR functions means that programs for data queries, data management (including data quality assessment), study conduct (eg, automated alerts, notes, and dashboards to support the trial itself), and analysis need to be created for every health system participating in a trial. To overcome this need for custom programming at each site, Collaboratory-affiliated researchers recommended increased collaboration between clinical researcher informaticists and healthcare IT operations professionals, as has been previously described by the Collaboratory and others.20,36,61 The establishment of multisite registries with data warehouses across health systems, especially for low-resourced settings, would help, and this is happening in several states. For example, in Utah, there is a data warehouse that is automating data feeds from 12 different health systems (FQHCs) using different EHRs. Another similar approach is the creation of a network that uses a CDM, such as the PCORnet,62,63 although transforming data into a CDM can be time consuming and resource intensive.64

DISCUSSION

Our findings add examples of specific services and standards, but the overarching question we face remains: how can researchers persuade health care organizations to invest in EHR enhancements and data infrastructure to support PCTs? Our findings from this survey of Collaboratory projects are similar to what we found in 2017: using EHR data for research and obtaining data from multiple sites are still major challenges; the amount of time and resources needed to use the EHR to support the delivery of interventions remains daunting, capturing high-quality data suitable for research is still difficult. While it might seem as if the needle has not moved on any of these issues, it is possible that studies are evolving in their data needs (eg, more complex interventions, more targeted and personalized interventions integrated with EHR systems) and that we are EHR-enabled to answer more sophisticated research questions. To better assess the improvement of these PCT facilitators at a national level, the Collaboratory Coordinating Center explores metrics for interoperability of EHRs and data “readiness”65,66 for pragmatic research. A number of national developments indicated that standardization of EHRs and data are progressing. In addition to the recent ONC report (2020) promoting a research agenda for EHR-enabled research, the use of standards has certainly improved with Meaningful Use certification requirements, and the consolidation in the EHR market has reduced variation in EHR systems. Also, improvements in reporting infrastructure for quality measures have benefits for research.

Our findings also mirror areas of need in the 2020 report from the ONC, which calls for improvements in data quality, harmonization, access to interoperable data, services for efficient storage and data, integration with other health data sources, and data aggregation.37 The use of EHR systems and data takes effort, costs money, and takes away from other health system priorities. This seems fundamentally unchangeable, so the alternative is to change the “story” and the argument for robust and customizable EHR systems. The story needs to not be about promoting research per se but rather about finding and implementing the best and most effective treatments, and continuously learning within and across health systems. However, building and promoting infrastructure for gathering standardized, re-usable data from the EHR in support of continual learning are not without costs, and it is the joint responsibility of funders, researchers, healthcare systems leaders, EHR vendors, and policymakers to work together to accomplish these goals. The National Academy of Medicine called for the development of a learning health system 11 years ago. It stated that 90% of clinical decisions should supported by timely, up-to-date clinical information,67 but achieving that goal remains distant and will require “health system leaders to consider rigorous evidence generation a core function of ordinary health care, research funders to prioritize practical questions relevant to population health and to support infrastructure for embedded research.”56 Currently, patients bear the costs of a lack of evidence, and we must strive to change this. This might be facilitated using expertise and approaches from emerging Implementation Science,68–70 which provides a scaffold for evaluation and bringing about change in the healthcare setting. In a recent article by Collaboratory leadership, the authors suggest re-imbursing healthcare systems for research-related costs and supporting re-useable infrastructure in highly engaged systems as a way to mitigate the costs of engaging in research and encouraging healthcare system participation.71 Similarly, the inclusion of robust cost impact studies might demonstrate a business case for the use of effective treatments (and the deimplementation of ineffective ones). Collectively and over time, these studies might provide compelling justification for continuous pragmatic research in learning health systems.

Research stakeholders (including clinicians, policymakers, research sponsors, patients, and the public/healthy consumers) must persuasively promote the importance of high-quality data. This value is not limited to a specific trial, but extends to other uses such as population health, continuous quality improvement, comparative effectiveness research, pharmacovigilance, and even algorithmic and artificial intelligent safety surveillance in the future. These programs can be conducted more cheaply and effectively with high-quality data. Standards-based EHR systems can empower customers to adopt new practices, which can lead to improvement and implementation in health systems. However, the current status quo is for each instance of an EHR is to not be standardized, even across separate instances or Epic, and the many different data sources (16 across the 21 Collaboratory trials) creates an ongoing challenge for investigators conducting PCTs (ie, for any individual trial, this means that every data pull needs to be configured for each and every participating site). The value of standards-based EHR systems can be promoted to health system leaders as a necessary enabler for integrating evidence-based practices and intervention to improve care and the health of the populations served (ie, a means to link research to quality improvement and continuous learning).

A positive and hopeful development is the HEAL (Helping to End Addiction Long Term) studies in response to the opioid crisis. These 6 studies are supported by the Collaboratory and have helped define a set of common measures72,73 based on research and clinical needs, including a common set of PRO measures. Using this common set of PRO measures will help with comparability across trials, minimize patient burden, as well as provide valid and reliable quality-of-life measures.74 One next step may be to lobby EHR vendors to include these basic measures and for EHR regulators/standards bodies to endorse them. Should the interventions evaluated by these studies (acupuncture, physical therapy, TENS, guided relaxation group-based mindfulness, etc.) be effective, the use of consistent measures across these studies could help with broader implementation. More recent success with N3C and COVID have shown that with the right incentive (and a bolus of centralized resources for harmonization), investigators can pull together disparate data quickly to understand the disease burden and effective targets for limited resources.75

Our study teams also reported the need for configurable EHR systems. Often informaticists tightly control the EHR in an organization, and the key stakeholders, including policymakers, are not fully aware, empowered, or incentivized to champion or facilitate the EHR customizations needed for multisite trials. Enhancing the ability to customize EHRs enables responses to urgent public health needs, delivery of new and evidence-based treatments, and the ability to address emerging local needs and patient-specific care. The number of PCTs funded by NIH and supported by health systems is growing; implementation science has burgeoned as a field; research networks, such as the Patient-Centered Outcomes Research Network (PCORnet) are enabling use of harmonized data; and the FDA is championing the use of real-world evidence. At the same time, the humanitarian and financial issues of poor health metrics, disparities in health, and increases in chronic and multiple diseases are growing. We have not made a dent in the Triple Aim, and the Quadruple aim,76–78 which adds the dimension of preventing clinician burnout and frustration with EHR systems.

Driving change for routine PCTs will require meaningful partnerships between healthcare systems, individual clinics, informatics teams, EHR vendors, and researchers, as well as federally driven standards and policies to ensure that EHR systems can support research.79 Indeed, the 21st Century Cures ACT80 and other developments are laying the case for this, and we assert that clinical researchers and health services researchers—as representatives of their health delivery organizations—should be leading efforts to develop and promote data standards that can enable sharing and rapid adoption of tools that facilitate the trial. These researchers can also leverage new regulations for quality measures, and weigh in and advocate for the importance of PROs. Embedded PCTs are integral to a learning health system and will enable faster changes in practice and improved patient outcomes. In time, the case for standards will grow in clarity and support as the learning cycles get faster and the benefits become more visible and established.

Limitations

Our sample of studies is limited and may not represent all pragmatic trials, and our sample varies in terms of timing of research, from beginning of enrollment to complete, and was conducted during the COVID-19 pandemic, which may have exacerbated issues related to EHR use and enhancements. The types of studies in our sample vary greatly, as the Collaboratory is funded by the NIH Common Fund with the goal of learning how to conduct high-impact trials that matter to public health, although this variation may not be representative of all PCTs and may affect the conclusions we can draw about EHR system readiness for research. Another limitation is that responses were collected in an asynchronous survey, so classic interview methods could not be done; that is, responses could not be expanded upon.

CONCLUSIONS

The case for high-quality, standardized data goes beyond clinical care, as it can impact the population as a whole by providing better care based on reliable evidence. We argue for tailoring the use of EHR systems to enable the collection of patient perspectives and the extraction of high-quality, robust data to support pragmatic research toward the identification of effective treatments and implementation strategies, and to enable learning within and across healthcare systems.

FUNDING

This work was supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by the NIH Common Fund through cooperative agreement U24AT009676 from the Office of Strategic Coordination within the Office of the NIH Director. This work was also supported by the NIH through the NIH HEAL Initiative under award number U24AT010961. Demonstration Projects within the NIH Health Care Systems Research Collaboratory were supported by the following cooperative agreements with NIH Institutes, Centers, and Offices: ABATE Infection (UH2AT007769, UH3AI113337), ACP PEACE (UG3AG060626, UH3AG060626), EMBED (UG3DA047003, UH3DA047003), GGC4H (UG3AT009838, UH3AT009838), HiLo (UG3DK118748, UH3DK118748), ICD-Pieces (UH2DK104655, UH3DK104655), LIRE (UH2AT007766, UH3AR066795), Nudge (UG3HL144163, UH3HL144163), PPACT (UH2AT007788, UH3NS088731), PRIM-ER (UG3AT009844, UH3AT009844), PROVEN (UH2AG049619, UH3AG049619), SPOT (UH2AT007755, UH3MH007755), STOP CRC (UH2AT007782, UH3CA188640), TiME (UH2AT007797, UH3DK102384), and TSOS (UH2MH106338, UH3MH106338). Demonstration Projects within the NIH HEAL Initiative were supported by the following cooperative agreements with NIH Institutes, Centers, and Offices: BackInAction (UG3AT010739, UH3AT010739), BeatPain Utah (UG3NR019943), FM TIPS (UG3AR076387, UH3AR076387), GRACE (UG3AT011265), NOHARM (UG3AG067593, UH3AG067593), and OPTIMUM (UG3AT010621, UH3AT010621). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or its HEAL Initiative.

AUTHOR CONTRIBUTIONS

A study representative and PI reviewed the final summary of each study and its features presented here. All authors listed here participated in the discussion and refinement of the themes and suggestions we present here and provided substantive comments on the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

The following authors report no competing interests: ARP, BJD, CG, DD, EBL, GDF, JMB, JRL, KRF, KS, LT, MOE, RLR, CLP, and JMS. ECO reports research grants to her institution from BMS and Novartis. CKZ reports research grants to her institution from F.D.A., NIH, the Foundation for Angelman Syndrome Therapeutics, and the Childhood Arthritis & Rheumatology Research Alliance (CARRA). ADB reports a research grant to his institution from Alike Health and reports consulting through Boyd Health Information Consulting LLC. KM reports a research grant to her institution from Pfizer. KSM reports research grants to his institution from Novartis, Amgen, Bayer, and Boehringer Ingelheim, consulting support from Novartis, and that he is coinventor of Hive Networks, Inc.

DATA AVAILABILITY

The data underlying this article cannot be shared publicly due to the potential to identify individual PIs and participating providers. The data will be shared on reasonable request to the corresponding author.

Supplementary Material

ocab202_Supplementary_Data

REFERENCES

  • 1. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 2011; 13 (2): 217–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Tunis SR, Stryer DB, Clancy CM.. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003; 290 (12): 1624–32. [DOI] [PubMed] [Google Scholar]
  • 3. Schwartz D, Lellouch J.. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis 1967; 20 (8): 637–48. [DOI] [PubMed] [Google Scholar]
  • 4. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us? N Engl J Med 2016; 375 (23): 2293–7. [DOI] [PubMed] [Google Scholar]
  • 5. Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ 2015; 350: h2147. [DOI] [PubMed] [Google Scholar]
  • 6. Weinfurt K, Hernandez AF, Curtis LH.. Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. https://rethinkingclinicaltrials.org/chapters/pragmatic-clinical-trial/what-is-a-pragmatic-clinical-trial/ Accessed July 9, 2020.
  • 7. Weinfurt KP, Hernandez AF, Coronado GD, et al. Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol 2017; 17 (1): 144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Coronado GD, Petrik AF, Vollmer WM, et al. Effectiveness of a mailed colorectal cancer screening outreach program in community health clinics: the STOP CRC cluster randomized clinical trial. JAMA Intern Med 2018; 178 (9): 1174–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Coronado GD, Retecki S, Petrik AF, et al. Mapping multi-site clinic workflows to design systems-enabled interventions. EGEMS (Wash DC) 2017; 5 (1): 13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. DeBar L, Benes L, Bonifay A, et al. Interdisciplinary team-based care for patients with chronic pain on long-term opioid treatment in primary care (PPACT)—protocol for a pragmatic cluster randomized trial. Contemp Clin Trials 2018; 67: 91–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Dember LM, Lacson E, Brunelli SM, et al. The TiME trial: a fully embedded, cluster-randomized, pragmatic trial of hemodialysis session duration. J Am Soc Nephrol 2019; 30 (5): 890–903. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Grudzen CR, Brody AA, Chung FR, et al. ; PRIM-ER Investigators. Primary Palliative Care for Emergency Medicine (PRIM-ER): protocol for a pragmatic, cluster-randomised, stepped wedge design to test the effectiveness of primary palliative care education, training and technical support for emergency medicine. BMJ Open 2019; 9 (7): e030099. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Huang SS, Septimus E, Kleinman K, et al. Chlorhexidine versus routine bathing to prevent multidrug-resistant organisms and all-cause bloodstream infections in general medical and surgical units (ABATE Infection trial): a cluster-randomised trial. Lancet 2019; 393 (10177): 1205–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Jarvik JG, Comstock BA, James KT, et al. Lumbar Imaging With Reporting Of Epidemiology (LIRE)—protocol for a pragmatic cluster randomized trial. Contemp Clin Trials 2015; 45 (Pt B): 157–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Jeffery MM, D'Onofrio G, Paek H, et al. Trends in emergency department visits and hospital admissions in health care systems in 5 states in the first months of the COVID-19 pandemic in the US. JAMA Intern Med 2020; 180 (10): 1328–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Lakin JR, Brannen EN, Tulsky JA, et al. ; ACP-PEACE Investigators. Advance Care Planning: Promoting Effective and Aligned Communication in the Elderly (ACP-PEACE): the study protocol for a pragmatic stepped-wedge trial of older patients with cancer. BMJ Open 2020; 10 (7): e040999. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Melnick ER, Jeffery MM, Dziura JD, et al. User-centred clinical decision support to implement emergency department-initiated buprenorphine for opioid use disorder: protocol for the pragmatic group randomised EMBED trial. BMJ Open 2019; 9 (5): e028488. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Mitchell SL, Volandes AE, Gutman R, et al. Advance care planning video intervention among long-stay nursing home residents: a pragmatic cluster randomized clinical trial. JAMA Intern Med 2020; 180 (8): 1070–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Simon GE, Beck A, Rossom R, et al. Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial. Trials 2016; 17 (1): 452. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Richesson RL, Hammond WE, Nahm M, et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory: Table 1. J Am Med Inform Assoc 2013; 20 (e2): e226–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Mor V, Volandes AE, Gutman R, et al. PRagmatic trial Of Video Education in Nursing homes: the design and rationale for a pragmatic cluster randomized trial in the nursing home setting. Clin Trials 2017; 14 (2): 140–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Zatzick D, Jurkovich G, Heagerty P, et al. Stepped collaborative care targeting posttraumatic stress disorder symptoms and comorbidity for US trauma care systems: a randomized clinical trial. JAMA Surg 2021; 156 (5): 462–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Jarvik JG, Meier EN, James KT, et al. The effect of including benchmark prevalence data of common imaging findings in spine image reports on health care utilization among adults undergoing spine imaging: a stepped-wedge randomized clinical trial. JAMA Netw Open 2020; 3 (9): e2015713. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Li H, Patil CL, Molokie RE, et al. Acupuncture for chronic pain in adults with sickle cell disease: a mixed-methods pilot study. Acupunct Med 2021; 96452842110173. doi:10.1177/09645284211017303. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. Vance CGT, Zimmerman MB, Dailey DL, et al. Reduction in movement-evoked pain and fatigue during initial 30-minute transcutaneous electrical nerve stimulation treatment predicts transcutaneous electrical nerve stimulation responders in women with fibromyalgia. Pain 2021; 162 (5): 1545–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Nielsen A, Ocker L, Majd I, et al. Acupuncture intervention protocol: consensus process for a pragmatic randomized controlled trial of acupuncture for management of chronic low back pain in older adults: an NIH HEAL initiative funded project. Glob Adv Health Med 2021; 10: 21649561211007091. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Edmonston DL, Isakova T, Dember LM, et al. Design and rationale of HiLo: a pragmatic, randomized trial of phosphate management for patients receiving maintenance hemodialysis. Am J Kidney Dis 2021; 77 (6): 920–30.e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Waughtal J, Luong P, Sandy L, et al. Nudge me: tailoring text messages for prescription adherence through N-of-1 interviews. Transl Behav Med 2021; ibab056.doi:10.1093/tbm/ibab056 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Luong P, Glorioso TJ, Grunwald GK, et al. Text message medication adherence reminders automated and delivered at scale across two institutions: testing the nudge system: pilot study. Circ Cardiovasc Qual Outcomes 2021; 14 (5): e007015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30., Chung FR, Turecamo S, Cuthel AM, et al. ; PRIM-ER Investigators. Effectiveness and Reach of the Primary Palliative Care for Emergency Medicine (PRIM-ER) pilot study: a qualitative analysis. J Gen Intern Med 2021; 36 (2): 296–304. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Hill J, Cuthel AM, Lin P, et al. Primary Palliative Care for Emergency Medicine (PRIM-ER): applying form and function to a theory-based complex intervention. Contemp Clin Trials Commun 2020; 18: 100570. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Zatzick DF, Russo J, Darnell D, et al. An effectiveness-implementation hybrid trial study protocol targeting posttraumatic stress disorder and comorbidity. Implement Sci 2016; 11: 58. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33., Tan A, Durbin M, Chung FR, et al. ; Group Authorship: Corita R. Grudzen on behalf of the PRIM-ER Clinical Informatics Advisory Board. Design and implementation of a clinical decision support tool for primary palliative Care for Emergency Medicine (PRIM-ER). BMC Med Inform Decis Mak 2020; 20 (1): 13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Green BB, Vollmer WM, Keast E, et al. Challenges in assessing population reach in a pragmatic trial. Prev Med Rep 2019; 15: 100910. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Petrik AF, Green BB, Vollmer WM, et al. The validation of electronic health records in accurately identifying patients eligible for colorectal cancer screening in safety net clinics. Fam Pract 2016; 33 (6): 639–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Richesson RL, Green BB, Laws R, et al. Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory. J Am Med Inform Assoc 2017; 24 (5): 996–1001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.The Office of the National Coordinator for Health Information Technology. National Health IT Priorities for Research. 2020. https://www.healthit.gov/sites/default/files/page/2020-01/PolicyandDevelopmentAgenda.pdf Accessed October 16, 2020.
  • 38. Braun V, Clarke V, Hayfield N, et al. Thematic analysis. In: Liamputtong P, ed. Handbook of Research Methods in Health Social Sciences. Singapore: Springer Singapore; 2019: 843–60. doi:10.1007/978-981-10-5251-4_103. [Google Scholar]
  • 39. Lakin JR, Gundersen DA, Lindvall C, et al. ; ACP-PEACE Investigators. A yet unrealized promise: structured advance care planning elements in the electronic health record. J Palliat Med 2021; 24 (8): 1221–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Gearing RE, El-Bassel N, Ghesquiere A, et al. Major ingredients of fidelity: a review and scientific guide to improving quality of intervention research implementation. Clin Psychol Rev 2011; 31 (1): 79–88. [DOI] [PubMed] [Google Scholar]
  • 41. Yang LY, Manhas DS, Howard AF, et al. Patient-reported outcome use in oncology: a systematic review of the impact on patient-clinician communication. Support Care Cancer 2018; 26 (1): 41–60. [DOI] [PubMed] [Google Scholar]
  • 42. Bartlett SJ, De Leon E, Orbai A-M, et al. Patient-reported outcomes in RA care improve patient communication, decision-making, satisfaction and confidence: qualitative results. Rheumatology (Oxford) 2020; 59 (7): 1662–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43. Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017; 318 (2): 197–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Evangelidis N, Tong A, Howell M, et al. ; COVID-19-Core Outcomes Set (COS) Survey Investigators. International survey to establish prioritized outcomes for trials in people with coronavirus disease 2019. Crit Care Med 2020; 48 (11): 1612–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Schnittker J, Bacak V.. The increasing predictive validity of self-rated health. PLoS One 2014; 9 (1): e84933. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Basch E, Barbera L, Kerrigan CL, et al. Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 2018; 38: 122–34. [DOI] [PubMed] [Google Scholar]
  • 47. Richter MF, Storck M, Blitz R, et al. Repeated digitized assessment of risk and symptom profiles during inpatient treatment of affective disorder: observational study. JMIR Ment Health 2020; 7 (12): e24066. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48. Biber J, Ose D, Reese J, et al. Patient reported outcomes—experiences with implementation in a University Health Care setting. J Patient Rep Outcomes 2017; 2: 34. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49. Nelson TA, Anderson B, Bian J, et al. Planning for patient-reported outcome implementation: development of decision tools and practical experience across four clinics. J Clin Transl Sci 2020; 4 (6): 498–507. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Zhang R, Burgess ER, Reddy MC, et al. Provider perspectives on the integration of patient-reported outcomes in an electronic health record. JAMIA Open 2019; 2 (1): 73–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51. Chung AE, Basch EM.. Incorporating the patient’s voice into electronic health records through patient-reported outcomes as the “review of systems”. J Am Med Inform Assoc 2015; 22 (4): 914–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52. Philpot LM, Barnes SA, Brown RM, et al. Barriers and benefits to the use of patient-reported outcome measures in routine clinical care: a qualitative study. Am J Med Qual 2018; 33 (4): 359–64. [DOI] [PubMed] [Google Scholar]
  • 53. Harle CA, Marlow NM, Schmidt SOF, et al. The effect of EHR-integrated patient-reported outcomes on satisfaction with chronic pain care. Am J Manag Care 2016; 22 (12): e403–8. [PMC free article] [PubMed] [Google Scholar]
  • 54. Gensheimer SG, Wu AW, Snyder CF, et al. ; PRO-EHR Users’ Guide Steering Group, PRO-EHR Users’ Guide Working Group. Oh, the places we’ll go: patient-reported outcomes and electronic health records. Patient 2018; 11 (6): 591–8. [DOI] [PubMed] [Google Scholar]
  • 55. Austin E, LeRouge C, Hartzler AL, et al. Capturing the patient voice: implementing patient-reported outcomes across the health system. Qual Life Res 2020; 29 (2): 347–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56. Simon GE, Platt R, Hernandez AF.. Evidence from pragmatic trials during routine care—slouching toward a learning health system. N Engl J Med 2020; 382 (16): 1488–91. [DOI] [PubMed] [Google Scholar]
  • 57. Tuzzio L, Larson EB.. The promise of pragmatic clinical trials embedded in learning health systems. EGEMS (Wash DC) 2019; 7 (1): 10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Uniform Data System (UDS) Resources. Bureau of Primary Health Care. 2018. https://bphc.hrsa.gov/datareporting/reporting/index.html Accessed March 5, 2021.
  • 59.CMS Finalizes Definition of Meaningful Use of Certified Electronic Health Records (EHR) Technology | CMS. https://www.cms.gov/newsroom/fact-sheets/cms-finalizes-definition-meaningful-use-certified-electronic-health-records-ehr-technology Accessed March 5, 2021.
  • 60.United States Core Data for Interoperability (USCDI) | Interoperability Standards Advisory (ISA). https://www.healthit.gov/isa/united-states-core-data-interoperability-uscdi Accessed April 2, 2021.
  • 61. Marsolo K. Informatics and operations–let’s get integrated. J Am Med Inform Assoc 2013; 20 (1): 122–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62. Forrest CB, McTigue KM, Hernandez AF, et al. PCORnet® 2020: current state, accomplishments, and future directions. J Clin Epidemiol 2021; 129: 60–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63. Fleurence RL, Curtis LH, Califf RM, et al. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc 2014; 21 (4): 578–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64. FitzHenry F, Resnic FS, Robbins SL, et al. Creating a common data model for comparative effectiveness with the observational medical outcomes partnership. Appl Clin Inform 2015; 6 (3): 536–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65. DeBar L, Jarvik J, Tuzzio L, et al. Assessing fitness for use of real-word data sources. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory. doi: 10.28929/053. https://rethinkingclinicaltrials.org/chapters/conduct/assessing-fitness-for-use-of-real-world-data-sources/introduction/ Accessed July 22, 2021. [Google Scholar]
  • 66. Douthit BJ, Del Fiol G, Staes CJ, et al. A conceptual framework of data readiness: the contextual intersection of quality, availability, interoperability, and provenance. Appl Clin Inform 2021; 12 (3): 675–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67. Institute of Medicine (US) Roundtable on Evidence-Based Medicine. Leadership Commitments to Improve Value in Health Care: Finding Common Ground: Workshop Summary. Washington, DC: National Academies Press; 2009. doi:10.17226/11982 [PubMed] [Google Scholar]
  • 68. Allotey P, Reidpath DD, Ghalib H, et al. Efficacious, effective, and embedded interventions: implementation research in infectious disease control. BMC Public Health 2008; 8: 343. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69. Glasgow RE, Eckstein ET, ElZarrad MK.. Implementation science perspectives and opportunities for HIV/AIDS research: integrating science, practice, and policy. J Acquir Immune Defic Syndr 2013; 63 (Suppl 1): S26–31. [DOI] [PubMed] [Google Scholar]
  • 70. Peters DH, Adam T, Alonge O, et al. Republished research: Implementation research: what it is and how to do it: implementation research is a growing but not well understood field of health research that can contribute to more effective public health and clinical policies and programmes. This article provides a broad definition of implementation research and outlines key principles for how to do it. Br J Sports Med 2014; 48 (8): 731–6. [DOI] [PubMed] [Google Scholar]
  • 71. Platt R, Simon GE, Hernandez AF.. Is learning worth the trouble?—improving health care system participation in embedded research. N Engl J Med 2021; 385 (1): 5–7. [DOI] [PubMed] [Google Scholar]
  • 72. Staman K. November 5, 2020: NIH HEAL Initiative Common Data Elements Published in the Living Textbook. Rethinking Clinical Trials. 2020. https://rethinkingclinicaltrials.org/news/november-5-2020-nih-heal-initiative-common-data-elements-published-in-the-living-textbook/ Accessed April 16, 2021.
  • 73.HEAL Initiative Research Plan. Research Plan | NIH HEAL Initiative. 2019. https://heal.nih.gov/about/research-plan Accessed April 16, 2021.
  • 74. Zigler C, Debar L, Weinfurt K.. NIH HEAL, FDA, and other core outcome sets. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory.doi:10.28929/139. https://rethinkingclinicaltrials.org/chapters/conduct/real-world-evidence-patient-reported-outcomes-pros/nih-heal-fda-and-other-core-outcome-sets/ Accessed July 27, 2021. [Google Scholar]
  • 75. Haendel MA, Chute CG, Bennett TD, et al. ; N3C Consortium. The National COVID Cohort Collaborative (N3C): rationale, design, infrastructure, and deployment. J Am Med Inform Assoc 2021; 28 (3): 427–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 76. Bodenheimer T, Sinsky C.. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014; 12 (6): 573–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 77. Obucina M, Harris N, Fitzgerald JA, et al. The application of triple aim framework in the context of primary healthcare: a systematic literature review. Health Policy 2018; 122 (8): 900–7. [DOI] [PubMed] [Google Scholar]
  • 78. Mery G, Majumder S, Brown A, et al. What do we mean when we talk about the Triple Aim? A systematic review of evolving definitions and adaptations of the framework at the health system level. Health Policy 2017; 121 (6): 629–36. [DOI] [PubMed] [Google Scholar]
  • 79. Richesson RL. Learning health systems, embedded research, and data standards—recommendations for healthcare system leaders. JAMIA Open 2020; 3 (4): 488–491. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80.Office of the National Coordinator for Health Information Technology. 21st Centure Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program Proposed Rule. 2019. https://www.healthit.gov/sites/default/files/nprm/ONCCuresActNPRM.pdf Accessed February 21, 2019.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ocab202_Supplementary_Data

Data Availability Statement

The data underlying this article cannot be shared publicly due to the potential to identify individual PIs and participating providers. The data will be shared on reasonable request to the corresponding author.


Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES