All together now: findings from a PCORI workshop to align patient-reported outcomes in the electronic health record

Abstract

In recent years, patient-reported outcomes have become increasingly collected and integrated into electronic health records. However, there are few cross-cutting recommendations and limited guidance available in this rapidly developing research area. Our goal is to report key findings from a 2013 Patient-Centered Outcomes Research Institute workshop on this topic and a summary of actions that followed from the workshop, and present resulting recommendations that address patient, clinical and research/quality improvement barriers to regular use. These findings provide actionable guidance across research and practice settings to promote and sustain widespread adoption of patient-reported outcomes across patient populations, healthcare settings and electronic health record systems.

A patient-reported outcome (PRO) is a measurement obtained directly from the patient about health, disease or treatment, without amendment or interpretation by a clinician or anyone else [1]. PRO is a term that can refer to symptoms, functioning, treatment satisfaction or health-related quality of life [1,2]. For many years, PRO collection for research purposes has been distinct from clinical practice applications. Due to technological innovations, there are increasing opportunities for PROs to be simultaneously collected and applied in both research and practice [3].

Electronic data reporting provides a platform for consistent and real-time documentation of PROs. When integrated into an electronic health record (EHR) in the same structured clinical data format (e.g., provider actions, laboratory tests and referrals), PROs can be used to inform individual patient and population-level care [4–6]. PRO-structured data can inform many stakeholders: provider, patient, health system and researcher. As the US healthcare system moves toward large-scale EHR-based data collection, there is increasing interest in collecting and standardizing PRO information [7]. However, beyond a handful of published case studies and academic pilots [8] little is known about current integration efforts.

Integrating Patient-Reported Outcomes into the electronic health record Workshop

In November 2013, the Patient-Centered Outcomes Research Institute (PCORI) and its Methodology Committee hosted a PRO Infrastructure Workshop (Atlanta, GA, USA) to understand current practices integrating PROs into EHR systems, and to generate broad stakeholder priorities to advance and sustain use. This meeting had three main goals: a landscape review, panel presentations and stakeholder discussion groups.

Patients, clinicians, researchers, healthcare system leaders and policymakers were invited to present on current use, identify common implementation barriers, share examples of clinical, research and quality improvement uses, and discuss strategies and priorities for increasing collection and use of PROs in EHR systems. A vision was articulated for integrating PRO measures within EHRs and for creating a larger national data infrastructure for medical care and public health. At the local level, this infrastructure would permit concurrent research and evaluation to be conducted more routinely within a real-world delivery system. It would also allow the care of individual patients to benefit from the knowledge and experience of those previously treated. This infrastructure would be fractal, locally used but also aggregating up to the regional and national levels. At the national level, the overall structure would support widespread capability for the US healthcare system to learn from practice, quality improvement and research [5,9,10].

Based on a pre-meeting landscape review of the field [11], we designed workshop presentations and discussion groups to learn about the range of current integration trends. Below are highlights and the key messages from the workshop, with a focus on how PROs most often collected for clinical practice applications can be leveraged for broader quality improvement and research applications. Following that is a summary of progress on the resulting agenda to date.

Workshop agenda: presentation of successful models, action agenda formulation

Workshop participants included in-person and online attendees (see Table 1 for description of participants). Panel presentations focusing on specific examples of PRO integration in the EHR were followed by full group discussion, interspersed with small group breakout sessions, with an opportunity for all attendees to rotate through breakout sessions on three topics: patient and PRO data, clinical use of PRO data and research and quality measurement. The small group breakout sessions proceeded through semi-structured discussion to address barriers and facilitators of PRO integration in the EHR and to submit recommendations for concrete actions for prioritization through consensus in the final session with all attendees. Data were analyzed and guided by basic principles of grounded theory [12]. Our thematic content analysis began with line-by-line examination of the transcripts. Two of the authors independently identified provisional categories and themes (RE Jensen & AW Wu). Initial theories were compared against newly identified themes, synthesis from the discussion, and author notes, to ensure validity. Nominal grouping was used to reach consensus regarding final categorization (Boxes 1 & 2).

Table 1. . Participant location and affiliation (n = 519).

Affiliation	Primary	Secondary	Total
In-person (n = 115)
Researcher/academia	17	20	37
Direct care provider	24	9	33
Patient/patient advocate	13	3	16
Industry representative	19	1	20
Researcher (private or nonprofit organization)	15	4	19
Government agency	14	0	14
Healthcare payer	6	0	6
Funder	3	0	3
Health system representative	3	1	4
Healthcare purchaser	1	0	1
Online (n = 408)
Researcher/academia	159	13	172
Direct care provider	41	3	44
Patient/patient advocate	24	4	28
Industry representative	22	0	22
Researcher (private or nonprofit organization)	50	7	57
Government agency	46	0	46
Healthcare payer	2	0	2
Health system representative	51	5	56
Funder	13	0	13
Healthcare purchaser	0	0	0
Missing/not reported	0	0	4

Box 1. . Recommendations to address current integration barriers.

Addressing patient concerns & promote engagement

• Make it easy to complete

  • – Provide multiple modes of data collection

  • – Reduce patient burden when possible

  • – Confirm preferred PRO data collection channels with patients

• Make it relevant to patients

  • – Involve patients in PRO development

  • – Give patients control of data and how it is displayed to them

  • – Better understand PRO relevance and importance of specific questions for individual patients prior to administration

  • – Provide immediate information and create immediate benefits for patients in order to demonstrate and advance their interest in PROs

• Clarify and address possible ethical issues directly

  • – Collect patient consent in a way to encourage collection of PRO measures, and assure that consent documents use simple language

  • – Provide guidance to Institutional Review Boards for consent and data use

Individual clinical care

• Improve clinical interpretability of PROs

  • – Measure other patient-reported information in addition to outcomes, such as their experiences with care, goals and preferences

  • – Determine which measures are most appropriate for specific patients and settings

  • – Use PROs to inform shared decision-making and promote individualized care plans

• Incentivize regular use

  • – Translate PROs into specific actions for clinicians

  • – Establish clear recommendations on how to respond to PRO scores in clincal settings

  • – Create value-based payment and reimbursement based on collection of PROs or PRO scores achieved

• Optimize workflow integration

  • – Reduce known technological barriers to electronic health record integration, and real-time, in-clinic review

  • – Move away from a measurement model that assumes care is centered on the health system. It will be valuable to assess PROs in other settings, including at home or at work

  • – Expand options to maximize accessibility of data collection to the greatest number and variety of patients

  • – Identify PRO measures and best practices for different contexts – for whom, when and what

Research and quality improvement

• PRO standardization and measurement equivalence

  • – Better understanding and education of the methodological requirements and differences between research and clinical use of PRO information

  • – Promote uniformity of PRO scoring, through methodological applications (i.e., IRT-derived crosswalks) and setting methodological standards

• PRO infrastructure and software needs

  • – Establish standards for the development for interoperable PRO tools

  • – Develop standards that allow data to be transmitted across different vendors

  • – Better understanding of metadata collected along with PRO scores and how it can be used to support adoption in quality improvement efforts

• Promoting and supporting use

  • – Leverage large research networks that have an infrastructure to support the development of PRO-based quality measures

  • – Work on developing well-defined, quality performance measures at the national level

  • – Demonstrate the value of database linkages (e.g., electronic health record, claims data) incorporating PROs in the evaluation of clinical care, CER and research applications

Box 2. . Immediate priorities to advance use of patient-reported outcome data through integration into the electronic health record.

Standards and education

• Adopt national standards for PRO measurement, data collection, and appropriate use across clinical care, research and quality improvement settings

• Develop educational materials to guide PRO measure selection and interpretation, with specific attention to facilitating EHR integration

Research

• Demonstrate usefulness of combining and reconciling PRO and clinical EHR data

• Determine patient preferences for privacy and sharing of PRO data

• Identify optimal strategies to reduce and manage missing data in PRO collection

Infrastructure

• Develop tools and resources needed for large-scale, sustainable use of PROs within healthcare through EHR platforms

• Promote an infrastructure and interoperability for large-scale PRO collection. (e.g., PCORNet)

• Collaborate with payer organizations to evaluate payment models for PRO data collection and use

Workshop results: current integration trends & stakeholders

We found that current integration efforts focus on informing and improving clinical care across different types of illness and healthcare settings. Very few examples were identified of PROs being pulled from EHRs for research-specific purposes. We identified four predominant EHR/PRO system designs: physician use during a patient visit; quality assessment and improvement; population-based risk screening and clinical and health services research. Each design illustrates the potential of collecting PROs and integrating results into the EHR. Current efforts are tailored to specific contexts, resulting in a wide range of target populations, assessment modalities, data collection schedules and external PRO reporting. This degree of specificity limits widespread collection across different systems. While goals and methods are similar, better understanding of commonalities and a core set of cross-cutting standards is necessary to provide a platform for widespread adoption.

Leading examples in patient-level symptom tracking tend to be for specific diseases that benefit from long-term monitoring, such as rheumatoid arthritis [13–15], HIV/AIDS [16] and cancer patients undergoing chemotherapy [17]. In these cases, PRO data modified treatment. Risk-screening applications focused on linking high-risk scores to targeted interventions. For example, at Kaiser Permanente Colorado a positive previsit PRO screen triggers standardized order sets to direct testing, treatment and referrals for geriatric patients.

Quality improvement efforts on symptom identification and management were identified at all healthcare levels: clinics, practices and populations. For example, Minnesota Community Measurement tracks and publicly reports PRO-based depression screenings and 6-month remission rates at the clinic level from over 80,000 patients annually [18]. In these cases, PRO measurement provides information on the PRO screening use, symptom prevalence and management.

The final workshop goal was to identify and understand stakeholder perspectives and their interest in promoting PRO EHR integration. Healthcare payers, such as Medicare, are looking to demonstrate value in healthcare beyond mortality and costs, which PRO-based quality of care indicators could provide. Healthcare plans, specifically integrated healthcare delivery systems such as Kaiser Permanente and Group Health Cooperative, are looking to leverage their existing clinical electronic data collection infrastructure. Clinicians and researchers were identified as ‘early adopters’, as patient-level PRO collection and use is most commonly driven by academic medical centers, often for use in specific clinics or conditions funded through research grants. Finally, patients themselves are important stakeholders, demonstrating increased interest in understanding their symptoms and functional issues, while current PRO data capture options and interpretation can be provided directly to the patient [19], aligning with growing patient expectations for transparency and access to medical information.

Meeting conclusions: the paths forward to widespread adoption

Stakeholders met in small groups to identify barriers and successful strategies to inform a research agenda: patient concerns, individual clinical care and large scale evaluation (quality improvement and research applications). Differences in needs, interests and expertise resulted in a range of opinions, but each of the groups was successful at achieving consensus. Below is a discussion of the main elements from each topic (Box 1) and actionable next steps (Box 2).

The benefits of patient engagement

Discussions regarding patient concerns focused on PRO collection. This action requires engaged patients, as they must be willing to complete assessments and provide responses that accurately reflect their current health. Reasonably complete and accurate data are needed for the timely identification of new concerns and to prevent biased assessment of care quality at the population level due to nonresponse.

Patient burden and the amount of time patients invest to complete PRO assessments were also important considerations. Research indicates that patients report high satisfaction completing assessments if they think data are used to inform their care [20]. However, it is possible that patients with different levels of functioning may value the usefulness of PRO data differently. Understanding how best to promote relevant PRO assessments while identifying the maintaining content relevance and low burden may help address these concerns.

Beyond promoting patient engagement, ethical issues related to the potential harms, respect for persons and privacy concerns related to PRO collection were identified. Just as a patient nonresponse to a PRO screen may delay issue identification, poor integration of PRO collection into patient care may disrupt or delay care. Additionally, PRO collection can be annoying to patients, if it feels redundant or repetitive with other information collected in the provision of care.

Finally, workshop participants raised concerns over the privacy and protection of their PRO data. Currently, many institutions require a blanket consent process, where patients give general consent for use of all of their data, clinical and patient-reported. To date, no published work has explored the extent to which sensitive PRO data is restricted within the EHR (e.g., assessments of a patient’s satisfaction with care), or outside the EHR (e.g., disclosure of depression severity to family members). Providing clear assurances of data privacy, clinical benefit and recognizing patient burden may help address these concerns and help promote patient involvement.

Themes from these small group sessions focus on improving the individual-level patient experience by increasing the clarity and relevance of clinical PRO assessment. Due to the limited information available and strong interest in ethical and privacy issues surrounding EHR integration, a targeted recommendation was identified as a necessary and immediate priority.

Simplifying & promoting clinical use

Clinical use focused on how PRO collection could be best integrated into every-day clinical care. A key issue voiced by workshop participants was the current established dichotomy between clinical and PRO-based measures in care settings. Many participants felt the benefits of PRO integration are not currently well-communicated and PROs often feel irrelevant and unimportant relative to clinical laboratory values. Clinicians also need guidance for interpreting PRO data, to ensure the data are translated into clinical actions as appropriate. A provider may see an ‘anxiety’ score of 60, but not know if it is a concerning score or if follow-up is necessary.

Many possible solutions were identified. One was to move beyond symptoms and function by including patient goals, treatment preferences and experience of care. Financial incentives were also highlighted as a way to promote and disseminate use. Establishing reimbursements for PRO screening or achieving quality of care benchmarks provides actionable patient follow-up while supporting quality improvement.

The final topic discussed by workshop participants was the necessity of minimizing workflow disruption. Participants agreed that workflow is often unique to each clinic, and general PRO standardization guidelines would not be effective. However, many agreed that ‘success stories’ of early PRO system adopters, and establishing meaningful PRO-based evaluation outcomes demonstrating institutional value and/or cost savings, would encourage adoption.

Recommendations for PRO integration in the clinical setting focused on promoting the value of PRO assessment in routine care. For clinicians, recommendations centered on clinical situations where PROs provide immediately actionable information. For system-level stakeholders, recommendations look to establish PRO value with respect to quality-of-care evaluation.

Leveraging current clinical PRO systems to promote population-level use

Participant discussions examining the promotion of large-scale research and evaluation focused on overcoming one key barrier: that the majority of current efforts to integrate PROs in the EHR have focused on demonstrating clinical relevance at the individual patient-level. Therefore the patient visit has been identified as the focal point for engagement, whereas population-level efforts may require collection and evaluation outside the clinical encounter. Opportunities identified focused particularly on demonstrating how PRO information in large-scale quality measurement and improvement extend beyond the clinical visit. Research and quality improvement recommendations focused on the necessity and complexity of large-scale PRO surveillance. Workgroups identified and prioritized a wide range of methodology, content, infrastructure and dissemination goals necessary to promote and sustain successful surveillance efforts.

Stakeholders also recommended that current collaborations and data networks be enlisted as contributors to standardize PRO integration across systems. PCORNet was specifically highlighted as a resource to promote best practices in population-level PRO use. PCORNet is a PCORI-funded effort which has created a national, highly representative, patient-centered network to support more efficient clinical and observational research [21]. Even in development, PCORNet accelerated the adoption of routine PRO measurement within EHRs by convening a national PRO taskforce charged with identifying a common core PRO data collection.

Personalized medicine from the patient perspective

A meeting theme across all groups was that PROs are not ‘one size fits all’. PROs comprise a broad range of information, including hundreds of symptoms, functional issues and topics that relate to quality of life. Effective measure deployment from this armamentarium differs based on context, as different PRO information will be clinically relevant and actionable at different times. More research is necessary to better understand how individual patient goals and preferences can tailor PRO selection for maximum, immediate and long-term patient benefit. For population-level PRO use, clinicians and health system representatives must also weigh in on matching measures to specific goals.

Immediate priorities & progress made

The EHR is a natural vehicle to collect and disseminate PRO data across a broad variety of clinical situations. However, current projects integrating PRO measures in EHRs are often designed for specific populations, clinical applications and quality improvement, which limits opportunities for expansion and scaling up across settings. Based on this workshop, immediate priorities were identified in three areas to promote PRO integration: education, research and infrastructure (Box 2). First, for education, guidance for providers [22], institutions [23] and healthcare systems are limited [24,25], often (but not always [22]) focusing on PRO selection and score interpretation. Expanding efforts to address the complexities of workflow and administration is necessary. Second, for research, directed methodological research to address gaps in PRO conceptualization, measurement and implementation can provide the information needed to promote interpretability and relevance. Third, for infrastructure, additional tools and resources are needed to support broad infrastructure development and demonstrate the value of PROs to major stakeholders. Together these elements best leverage EHR integration for widespread PRO adoption and regular use.

Priorities identified by workshop participants were incorporated into PCORI Funding Announcement development and, as a result, approximately 10% of the methods priority area portfolio reflects projects based on the priorities identified from this convening of diverse stakeholders. In addition, in follow-up to the 2013 PRO EHR meeting and in response to these immediate priorities, PCORI is now funding an effort to develop a ‘Users’ Guide for Integrating Patient-Reported Outcomes in Electronic Health Records' (PCORI contract #MC-0013; PI: Snyder). This Users’ Guide will review the considerations for including PROs in EHRs, offering a range of alternatives rather than one ‘right’ answer, thereby allowing clinics and institutions to determine what will work best in their environments. This effort will also identify key research questions for the field. A public meeting presenting the Users’ Guide will offer an opportunity for participants to discuss voluntarily implementing PRO integration in consistent ways to enable data pooling.

Conclusion

Diverse stakeholders were convened to discuss examples of successful integration of PROs into the EHR to support clinical care, health system- or population-level monitoring, quality management and research. Through group discussion and guided workshop sessions, barriers to use of PRO data through the EHR were identified, allowing for creation of an agenda for future work. PCORI has begun to address this agenda through funding solicitations based on the priorities identified at the workshop, through PCORNet and through a focused contract to create a guide for PRO integration in the EHR applicable across care settings.

Involving multiple stakeholder groups in the problem identification and articulation of priorities is a first step toward ensuring future work is responsive to the differing needs across the care delivery spectrum. Patient and clinician engagement addresses feasibility and relevance of PRO data collection. Health and IT professionals, as well as researchers, are necessary to address system interoperability while ensuring data accuracy. Healthcare plans, professional groups, accreditation organizations and the government can provide incentives to integrate, report and evaluate PROs as part of routine care. PCORI and other influential groups, such as the NIH Collaboratory, are well-positioned to guide this effort [26]. Joint interdisciplinary participation in this effort to standardize the patient voice within the EHR will ensure PROs are actionable, positioned to improve care quality and the patient healthcare experience at both the patient and system-level.

Executive summary.

• Curently, most EHR/PRO integration efforts are implemented solely to inform individual patient care.

• Targeted research is needed to highlight the direct clinical applications and value of PROs to patients, providers, and healthcare systems.

• National Standards for PRO collection and infrastructure are necessary.