Collecting Patient-Reported Outcome Measures in the Electronic Health Record: Lessons from the NIH Pragmatic Trials Collaboratory

Abstract

The NIH Pragmatic Trials Collaboratory supports the design and conduct of 27 embedded pragmatic clinical trials, and many of the studies collect patient reported outcome measures as primary or secondary outcomes. Study teams have encountered challenges in the collection of these measures, including challenges related to competing health care system priorities, clinician’s buy-in for adoption of patient-reported outcome measures, low adoption and reach of technology in low resource settings, and lack of consensus and standardization of patient-reported outcome measure selection and administration in the electronic health record. In this article, we share case examples and lessons learned, and suggest that, when using patient-reported outcome measures for embedded pragmatic clinical trials, investigators must make important decisions about whether to use data collected from the participating health system’s electronic health record, integrate externally collected patient-reported outcome data into the electronic health record, or collect these data in separate systems for their studies.

Introduction

Patient-reported outcomes (PROs) are meaningful aspects of health that include concepts like health-related quality of life, fatigue, emotional distress, social functioning, pain, and physical functioning reported directly by patients. PRO measures are the questions and/or surveys used to collect this information without interpretation from a third party such as a clinician or researcher.¹ PRO measures are often the best approach to collect information about outcomes that are experienced uniquely by the patient (e.g. pain intensity, sleep quality, fatigue, satisfaction with social roles) and thereby provide critical insights into ways that respondents are feeling and functioning. As such, PRO measures are critical components of clinical trials,² including embedded pragmatic clinical trials (ePCTs) that evaluate interventions as part of routine care and frequently use information from the electronic health record (EHR) as a data source. When available via the EHR, PRO measures can be used in many valuable ways to enhance and individualize care, including exploring the clinical utility of treatments and enabling shared decision making between clinicians and patients. PRO data from the EHR can also be used by researchers to better understand how symptoms change over time for groups of patients. However, problems arise when PRO measures are not routinely integrated into an EHR system or are inconsistently adopted across patient groups and settings. Moreover, even when PRO measures are documented in the EHR, there can be issues with administration of PRO measures, including provider and patient burden, that may result in inconsistent and biased data.

Even with support and funding for PRO measure collection and integration, as is typical in ePCTs, major health system barriers may impede the collection, use, and interpretation of PRO measures. Competing health care system priorities constrains incentives for addressing financial and logistical challenges to PRO data collection in EHR systems, as does lack of clinician buy-in for adoption of PRO measures. The limited adoption and reach of technology, such as patient portals, impacts the completeness of PRO data and may introduce bias through the differential use by demographic subgroups. Further, there is a lack of consensus and standardization regarding which measures are important for different health conditions, how to assess them, and how PRO data may inform clinical decisions. Due to these challenges, many study teams collect PRO data in separate systems (e.g., REDCap^3,4) and may encounter technical and governance challenges when attempting to integrate these data into a participant’s EHR. Ultimately, any solution involving routine collection of PRO measures via the EHR, or the integration of PRO data collected outside of the EHR must primarily benefit patients and clinicians, with secondary benefits for researchers, administrators, and payors.

In this article, we share lessons learned in the NIH Pragmatic Trials Collaboratory, which supports the design and implementation of 27 ePCTs that address issues of major public health importance and are conducted in diverse settings (e.g., federally qualified health centers, hospitals, clinics, primary care, pediatrics). Many of these projects collect PROs (Table 1). Principal investigators (Pis), Demonstration Project staff, and the leaders from the EHR and Patient-Centered Outcomes (PCO) Core Working Groups share the barriers they have encountered regarding the collection of PRO measures and their integration into the EHR, along with success stories and lessons learned. We conclude with next steps and future directions for research to address the challenges above.

Table 1.

NIH Pragmatic Trials Collaboratory Demonstration Projects^* and PRO measures

Acronym	Description
BackinAction	Evaluates acupuncture vs usual care in patients >65 with chronic low back pain
BeatPain Utah30	Compares the effectiveness of nonpharmacologic intervention strategies for patients with chronic low back pain seeking care in Community Health Centers (CHCs) throughout the state of Utah.
FM-TIPS	Compares the use of TENS for movement evoked pain in patients with fibromyalgia in physical therapy
GRACE	Evaluates the effects of guided relaxation and acupuncture for people with sickle cell disease, the majority of whom identify as Black.
Guiding Good Choices for Health31	Tests the feasibility and effectiveness of implementing a universal evidence-based anticipatory guidance curriculum (Guiding Good Choices) for parents of early adolescents to promote family bonds, healthy development, and reduce risky behaviors.
NOHARM33	Aims to encourage use of validated nonpharmacologic approaches to manage peri-operative pain.
	Part of NIH HEAL Initiative
PRO measures	Backin Action	BeatPain Utah	GRACE	FM-TIPS	NOHARM	Guiding Good Choices for Health
Adverse events	x		x	x
Antisocial behavior						x
Brief pain inventory - short form				x
Telephone Interview for Cognitive Status (TICS)	x
CRAFFT Substance use screening tool						x
Euro-Quality of Life (QOL)-5d	x
EXPECT acupuncture expectation questions	x
Fibromyalgia Impact Questionnaire-Revised (FIQR)				x
Frailty scale	x
Generalized Anxiety Disorder (GAD-2 or GAD-7)	x	x	x	x	x	x
HEAL common data element demographic questions + body mass index	x
High Impact Chronic Pain (HICP)	x	x
Impact of COVID on overall health and access to healthcare	x
Movement evoked (5x sit to stand) pain				x
Movement evoked fatigue				x
Multidimensional Assessment of Fatigue (MAF)				x
National Institutes of Health Lower-Back Pain Task Force fear avoidance	x
Pain Catastrophizing Scale (PCS) – 6 or 13 item scale	x	x	x	x	x
Pain intensity, interference with Enjoyment of life, interference with General activity (PEG)	x	x	x
Pain numeral rating scale					x
Pain Self-Efficiency Questionnaire (PSEQ-4)		x
Patient Global Impression of Change (PGIC)	x	x	x	x
Patient Health Questionnaire (PHQ-2, 8, or 9)	x	x	x	x	x	x
Patient specific functional scale (PSFS)				x
Peer Substance Use						x
Perceived risk of drug use						x
PROMIS Ability to participate in social roles and activities 4a	x
PROMIS computerized adaptive test: Physical Function					x
PROMIS fatigue scale	x
PROMIS Gastrointestinal/Constipation 9a			x
PROMIS pain interference 4a			x		x
PROMIS physical functioning 6b	x	x	x	x	x
PROMIS sleep disturbance, duration or quality 6a or 8a	x	x	x	x
Rapid Assessment of Physical Activity (RAPA)				x
Resting fatigue by numeral rating scale				x
Resting pain by numeral rating scale				x
Roland Morris Disability Questionnaire (RMDQ)	x
Screen and social media time						x
Symptom severity score				x
Tobacco Alcohol Prescription Medication Substance Abuse (TAPS 1)	x	x	x	x	x
Widespread Pain Index (WPI)				x

Patient-Reported Outcomes Measurement Information System (PROMIS), National Institutes of Health (NIH), Helping End Addiction Long-Term (HEAL)

^*Note that all of the trials listed in this table except for GGC4H are part of the NIH HEAL Initiative (Helping End Addiction Long Term Initiative) which required similar PRO measures to be used across all trials. Thus, many of these were selected for comparative purposes rather than clinical utility. Please also note, the measures listed here are not an exhaustive list of all data collected directly from patients within the trial.

Methods

The EHR Core discussed challenges with collection of PRO measures on a series of monthly calls. These challenges were collated into an initial draft of the document by Core leaders, and then circulated to the Patient-Centered Outcomes Core leaders and principal investigators and study teams (who are co-authors). They were asked to share concrete lessons learned and ideas about what is needed to better incorporate PRO measures into ePCTs and the EHR.

PRO Measure Integration Decision Framework

Figure 1 presents a decision framework for pragmatic trial investigators when considering integration of PRO measures into the EHR. Investigators should identify which PRO measures are already available within the EHR at their participating health system. If the required PRO measures are already supported, investigators should evaluate the existing PRO measures for consistency across sites and quality of historical PRO measure data previously collected via established mechanisms (e.g., data missingness, completeness over time). If the PRO measures of interest are not already available in the EHR, the study team must consider the utility of integrating external PRO measures into the EHR to help guide clinical care and support future research projects. This decision comes with several challenges also outlined in Table 2 and below, including technical and governance considerations. If the PRO measures are deemed to be primarily useful for the ePCT, or if there are ethical or legal issues that prohibit integration into the EHR, the study team may choose to use an external system for their collection (such as REDCap).

Figure 1.

Considerations for Integrating Patient-Reported Outcome Measures (PROMs) in Electronic Health Records for Pragmatic Trials

Table 2.

Common Challenges and Considerations for adding PRO measures into the EHR for ePCTs.

Competing health care system priorities
Challenge	Recommendation	Case Example
Healthcare systems do not collect the necessary PRO measures for research	Understand system priorities and policies. When exploring EHR updating opportunities, invite multiple people of different roles to group meetings to understand barriers facilitators and workflow perspectives. Ask partners in the HC system to explain how the changes would impact their everyday work – and do this prior to implementation.	For the GRACE trial, the current policy, set by the hospital’s suicide prevention committee, only supports EHR integration of the PHQ-8 for hospital care; therefore, researchers can only use data from PHQ-9 if it is collected outside of the EHR, so the GRACE study collects this remotely.
Integrating PROs into the EHR may not be useful/necessary	Assess whether integrating (any/all) PRO measures into the EHR is necessary for a given ePCT. Before engaging the health care system about PRO measures that are not routinely collected in the EHR, determine whether there is clinical or scientific utility in integrating PRO measures into the EHR. This may not be necessary for all ePCTs.	BackInAction trial sites do not collect the primary outcome in the EHR, and the intervention is not administered within the health care system at 3 of the 4 sites, making integration of the Roland Morris Disability Questionnaire into the EHR unnecessary. All of the PROs, including those required by the HEAL Initiative as Common Data Elements are collected in REDCap.
HC systems are complex and often have “information overload”	Rather than providing raw PROM data, the EHR must display relevant data points or summaries, and provide actionable information to a provider. When available, clinical decision support can automate appropriate responses to actionable PROM scores, thereby avoiding the need for clinician review and management.	The NOHARM trial EHR build included rule-based logic to determine which automated response should be triggered by PROM-assessed self-efficacy for pain management. Low and very low scores led to portal delivery of needs-matched self-management materials.
It is costly and time-intensive to add PRO measures, and HC systems may ask researchers to pay for the IT support.	Build this cost into grants and/or identify HC system stakeholders who may be amendable to cost-sharing	The FM-TIPS trial built IT set-up and periodic EHR extraction into the grant funding. The primary limit to EHR extraction was time to complete the task by a PT clinician rather than an IT representative.
Healthcare systems have unique processes, cost structures, and timelines for prioritization Additionally, health system/health system’s IT timelines often don’t match the grant timelines	Develop short-term solutions to collect the necessary PRO data while waiting for EHR integration.	BeatPain hoped to implement e-referrals with some PROs to be transmitted from CHCs to an academic health system’s EHR prior to enrollment, but the healthcare system was going through a large implementation of their EHR at an affiliated hospital and had no bandwidth to implement the requested functionality. The alternative solution was to purchase a license to a cloud-based inbox that served as the project’s e-referral inbox, independent from the EHR. The entire approach had to be implemented by the BeatPain team. For the GRACE trial there was a fee at 2 of the 3 healthcare systems to build and turn on the PROs so they may be integrated into all 3 healthcare systems’ EHRs. All 3 healthcare systems had their own unique processes, cost structures, and timelines for the prioritization of the study to activate requisite PRO measures. The trial was not high on the priority list. As a short-term solution, GRACE collects these data on REDCap.
Clinician’s buy-in for adoption of PRO measures
Challenge	Recommendation	Case Example
Data entry is an additional burden on clinical staff	Utilize electronic PRO measures whenever possible, considering alternative administration methods, to limit the data entry burden on staff and minimize errors.	To minimize EHR data collection burden at study sites in BeatPain Utah, research coordinators and physical therapists collected PROs for patients enrolled in the trial via REDCap.
Overburdened care team may not see the value	• Have PROM data populate the EHR for clinician use during a visit because this saves time. Use targeted PROs to improve clinical care and provide value • Use PROs at specific encounters • Select PROs to be embedded in the EHR such that they provide value. • Consider collecting disease- or condition-specific PRO measures through other mechanisms, such as REDCap.	The NOHARM trial used a portal-based PRO measure to query patients scheduled for qualifying surgeries about their preferred nonpharmacological pain care approaches. Patients selected validated approaches as part of their individualized pain management plans. The selections were recorded in the EHR and used to prompt providers to encourage and support patients’ use of their selected nonpharmacological modalities in lieu of opioids. The NOHARM EHR build made it easy for providers to order and administer patients’ selected modalities, to document modality use, to automatically provide educational and self-management materials, and remain informed about patients changing modality use and preferences. The build was designed to autopopulate with default values based on patients’ modality choices, thereby reducing data entry requirements. Mixed methods assessment revealed a perception of improved efficiency and high levels of satisfaction among clinical stakeholders.
Scores must be interpretable; clinicians don’t always know what is clinically significant for a particular PROM. There is also a perception that PROM assessment will raise patients’ expectations and make clinicians responsible for conditions that fall outside their training, resourcing, and scope of practice.	Make sure that the PROM are meaningful, results are interpretable by clinicians and patients, and provide guidance on what is clinically significant	See NOHARM example above
Low adoption and reach of technology
Challenge	Recommendation	Case Example
Low adoption and reach of technology such as personal health records (PHRs) at low resource settings such as safety net community health clinics (CHCs)	Use other technology. Enable interventions based on bidirectional text messaging that aim to connect patients to health services	To address technology access barriers among low socio-economic status and rural populations, some trials have enabled interventions based on bidirectional text messaging that aim to connect CHC patients to health services such as a tobacco quitline, COVID-19 testing and vaccination, and colorectal cancer screening.34,35
A growing digital divide between patients seen at high-resource healthcare systems and socioeconomically disadvantaged and rural systems such as safety net CHCs	PROs can be collected as a part of the engagement between ePCT staff and patients and reduce existing disparities in data collection.	BeatPain Utah collected PROs as a part of the engagement with patients in the delivery of telehealth-based interventions. These interventions have been implemented to serve patients who receive care at 13 CHCs with 62 clinics across Utah.
Lack of Consensus and Standardization of PROM selection and administration in the EHR
Challenge	Recommendation	Case Example
PRO measures are typically chosen based on a specific context of use,36 the ‘best’ PROM	Capture of a limited core set of cross-cutting PRO measure data on all patients and those	The NIH requires all HEAL Initiative (The Helping to End Addiction Long-term Initiative)37
for an ePCT could be less optimal for use in clinical care settings.	with a given health condition and supplement with additional PRO measures selected by the patient and clinician to support individual patient treatment goals. It is difficult to find general measures that are appropriate for entire populations.	trials for the reduction of chronic pain and opioid use to use the NIH Common Data Elements measures that are housed in the NIH Common Data Elements Repository. This will facilitate data sharing amongst researchers. All the projects mentioned in this manuscript are HEAL projects except for GGC4H.
Lack of consensus as to which PRO to use for a specific condition; in many disease states there are multiple PROs	Agree on a limited set to use. This requires patients, clinical experts, including professional medical societies and NIH research groups in specific clinical areas, to define.	The 6 trials that are also in the PRISM Collaboratory (HEAL Initiative) use the requisite HEAL Initiative Common Data Elements, which measure pain and other symptoms associated with pain. GGCH4H involves three health systems, all of which have prioritized different measures of adolescent substance use and mental health outcomes, many of which are not collected systematically in pediatric primary care.

Abbreviations: community health clinics (CHCs); electronic health record (EHR); embedded pragmatic clinical trials (ePCT); information technology (IT) patient-reported outcome (PRO); Patient Health Questionnaire (PHQ), personal health records (PHRs), physical therapy (PT), Helping to End Addiction Long-term (HEAL) Initiative,

Competing health care system priorities

Health care systems are complex and have many competing priorities,^5,6 which must be understood by ePCT investigators. Adding a PRO measure to an EHR system requires money and time, and may increase or disrupt the workflow of overburdened clinicians. Therefore, the broader focus of the ePCT and PRO measure strategy need to align with the healthcare system’s goals (Table 2).

Clinician’s buy-in for adoption of PRO measures

In order for PRO measure integration to be successful, the clinical staff and researchers will need to fully understand how PRO data will contribute to clinical decision-making, facilitate communication and high-quality patient care,⁷ and impact clinical workflow. Importantly, the care team will need to know who to speak to if the changes cause major disruptions or become too burdensome for the patient or care team. In our experience we have found that high quality training and clear, comprehensive explanations are essential when updating or adding a PRO measure to the EHR for research. Where possible, clinical staff input can also be informative.

For PRO measures to be more widely accepted by the care team, they must reduce rather than contribute to the information chaos.⁸ Care team EHR data entry is associated with high levels of professional burnout (Table 2). ^9–12 EHR usability challenges have resulted in systems that are complex and error prone, thereby increasing care team cognitive load and errors, leading to patient harm in some cases.^13–16 A major barrier to buy-in from stakeholders when collecting PRO data is interpretation of scores by both the care team and patient,¹⁷ along with contextual differences that may affect score interpretation within a specific use case (considering patient population, age, time in disease course, treatment, and intended use of the measure).

Low adoption and reach of technology in low resource settings

A major barrier to implementing interventions that enable collection of PRO measures is the low adoption and reach of technology (Table 2). Despite ubiquitous adoption of EHR systems, there is a growing digital divide between patients seen at high-resource healthcare systems and socioeconomically disadvantaged and rural systems such as safety net Community Health Centers.^18,19 For example, underserved healthcare settings are 17% less likely to allow patients to access health information; less likely to have adopted patient engagement functions; and have lower portal-based patient-provider communication. ^18,20–23 However, when portal-based PRO measures are unavailable, other methods of outreach for PRO measures can be considered; underserved populations have almost universal access to technology such as cell phones that provide opportunities for large-scale patient outreach interventions and collection of PRO measures. According to Pew Research, even in households with annual incomes less than $30,000, 97% own a cellphone and 76% own a smart phone.²⁴ Therefore, digital health interventions based on technology such as text messaging and interactive voice response have a strong potential to reach individuals in low socioeconomic status and rural communities.

Lack of Consensus and Standardization of PRO selection and administration in the EHR

Across healthcare systems—and even within systems—there is a lack of consensus about which PRO measures are the most appropriate. Further, PRO measures can be domain and even disease-specific, and there is a lack of standardization across PRO measures, although efforts have been made to bolster standardization. For example, the 2004 NIH Common Fund program invested in efforts to facilitate the integration of PROs in research, leading to the Patient Reported Outcomes Measurement Information System (PROMIS), which includes well-validated and reliable measures for a myriad of chronic disease conditions.²⁵ The PROMIS measures are available in a variety of forms from paper and pencil, web and mobile platforms, and through EHR data capture. However, despite attempts to create universally sensitive and valid PRO measures, there is no one ‘perfect’ and valid PRO measure for every context of use. For example, for pain outcomes, PROMIS measures were not as sensitive or specific to fluctuations in patient status as needed for the demonstration projects described here. Therefore, challenges remain when trying to choose PRO measures that are useful across different contexts.

Next Steps

An important step in the process of integrating PRO measure data into the EHR is selection of the specific measures for collection and integration. In some cases, the challenge is finding areas of overlap between PRO measures that are useful for patient care and measures that are useful for research. For patient care, there is widespread agreement that the PRO measures collected should align with patient and clinician priorities for treatment. However, even within the same disease area, different patients have different treatment priorities, so there will be inherent variation in the PRO data captured to support treatment goals. This variation is challenging to researchers aiming to use EHR data to evaluate PROs across large populations. A possible solution is capture of a limited core set of PRO data on all patients with a given condition and supplementing with additional PROs selected by the patient and clinician to support individual patient treatment goals, as is being done with the NIH HEAL Initiative (Helping End Addiction Long Term) Initiative, which is focused on pain; many of the Demonstration Projects used as examples in this paper are part of the HEAL Initiative (Table 1). Developing a core set of measures requires multi-stakeholder consensus, as well as commitment to a shared decision-making process by the patient and clinician. Achieving this consensus will take time, and platforms for communication and consensus building are needed that leverage existing communities and standards development processes, such as HL7.^26–28

Integrating External PRO measure data into the EHR

Given the multiple barriers described above, study teams may choose to use separate data collection systems (e.g., REDCap) to support capture of PRO measure data required for their trial. If these PRO measure data have clear clinical utility or compelling broader applications, investigators may wish to explore integrating them into the participant’s health record. Several standards currently exist to support the technical work associated with this integration (e.g., Fast Healthcare Interoperability Resources [FHIR] questionnaire resource²⁹), and most EHR systems allow clients to create custom questionnaires. However, there are several important issues that the study team must consider, including, whether treating clinicians are expected to review these data and act on results. Also, integration of external data may have broader implications beyond individual patient care and may therefore require input from a wider range of healthcare system stakeholders (e.g., quality improvement teams, value-based payment leadership). Finally, the use of emerging standards for data exchange such as FHIR does not directly address the need for standard use of the same PRO measures across sites, which requires stakeholder alignment on the choice of PRO measures.

Limitations

The information presented in this paper was collected on a series of working group calls, drafted into a paper, and then refined by the Principal Investigators of the trials and their teams. We did not conduct thematic analysis or gather information in a systematic way, although this could be accomplished in future work.

Conclusion

PRO measures are critical to patient-centered research, including ePCTs. When using PRO measures for ePCTs, investigators must make important decisions about whether to integrate PRO measure data collection into participating health system EHRs or to integrate externally collected PRO measure data from their studies into individual records to support clinical care and future research. When making these decisions, investigators should consider the clinical utility of PRO measures, healthcare system priorities, clinician buy-in, adoption and reach of technology in different settings, and current efforts related to standardized capture of these measures.