Visualizing the Patient-Reported Outcomes Measurement Information System (PROMIS) Measures for Clinicians and Patients

Abstract

Congruent with the nationwide movement toward patient-centered healthcare, an increasing number of organizations collect and assess patient-reported outcomes (PROs). The standardized NIH PROMIS measures represent one of the most widely used PRO questionnaires, but organizations still face challenges with conveying PROMIS outcomes to clinicians in clinically relevant ways. Our proposed solution, the ProVis application, uses visualizations to engage heart failure patients with PROMIS questionnaires in the waiting room, and conveys PROMIS data to clinicians through longitudinal visualizations in iNYP, our institution’s electronic health record (EHR) interface. Here, we discuss the design and development of ProVis, the alternative strategies we considered, the strengths and weaknesses of ProVis, and our future dissemination and evaluation plans.

Background and Definition of the Challenge

Patient-reported outcomes (PROs) constitute an important subset of patient-generated health data. When collecting PROs, healthcare organizations use standardized survey questions to understand patient’s experiences, and the data may inform each patient’s care plan individually. Recently, the National Institutes of Health (NIH) created the Patient-Reported Outcomes Measurement Information System (PROMIS) measures.^1,2 PROMIS is a validated and standardized set of PRO measures applicable to a range of chronic conditions. The PROMIS scales map directly to PRO measures such as the PHQ-9 depression questionnaire. As such, PROMIS measures occupy an important position at the center of national “Meaningful Use” efforts to incorporate PROs into patient care. Epic Systems now offers PROMIS measures through the MyChart patient portal, and the NIH’s EASI-PRO initiative promotes integration of PROMIS into major electronic health records systems, including Epic and Cerner.

Despite PROMIS’s success, challenges remain. An ideal PRO measure is efficient (minimizes the number of items without compromising reliability) and precise (minimizes error in estimate).² Despite efforts to maximize efficiency of individual scales, clinically useful combinations of PROMIS scales may take too long (>10 minutes) for older patients and patients with low technology literacy. Despite rigorous effort to increase precision, patients with a lower literacy level may demonstrate lower precision.³ Such limitations may decrease clinician’s confidence in PROMIS outcomes, especially clinicians who work with disadvantaged populations. One untested solution is to visualize each PROMIS item’s answer choices to help patients select answers more precisely and efficiently.

Another challenge is defining appropriate cutoff points for clinical action based on PROs. Indicating the patient-reported symptom’s severity is necessary, but insufficient. Clinician-facing systems must also report changes in each symptom’s severity. For example, a patient with longstanding severe depression known to a psychiatrist is less clinically concerning than a patient with newly-reported depression that requires a psychiatric consult. Currently, physicians use clinical notes to track symptom changes over time, a practice that presents challenges due to copy-paste conventions and the time needed to re-read the note. Overcoming these challenges will require better presentation of longitudinal PROs in the EHR to support clinical decision making.

In collaboration with the Columbia University Center for Advanced Cardiac Care at NewYork-Presbyterian Hospital (NYP), our proposed solution is to 1) visualize PROMIS answer choices for heart failure patients to increase PROMIS’s precision and efficiency, especially in disadvantaged populations, 2) incorporate these visualizations into an application, ProVis, for heart failure patients to report PROMIS measures from the waiting room, and 3) visualize ProVis data for cardiologists in the EHR, in a manner that predicates clinical action and facilitates longitudinal symptom tracking. We use the heart failure population to demonstrate our application’s utility for multiple reasons.

Heart failure affects more than 5.7 million Americans and is the leading cause of 30-day hospital readmissions in the United States.⁴ Ineffective symptom recognition and management is the primary cause of repeated hospital readmissions in heart failure.⁵ Clinicians typically spend the majority of the outpatient visit discussing symptoms, although time constraints may prevent the full elucidation of clinically relevant symptoms. As such, PROMIS measures hold great potential to transform clinical decision making in heart failure care.

Proposed Solution

At this time, the patient-facing component of ProVis is fully developed and communicates with the EHR, and the interface for the clinician-facing component is designed. During the design process for ProVis, we applied techniques from biomedical informatics (see Integration with the EHR section), human-computer interaction (see Patient-Facing Interface section), and information visualization (see Clinician-Facing Interface section) to enhance usefulness and usability. To integrate clinical expertise, we conducted participatory design sessions with two clinicians to ensure each visualization’s clinical relevance, including card sorting to discover clinicians’ mental models of heart failure symptoms. To integrate patient expertise, we refined the application after usability testing with 6 hospitalized patients (poster accepted to 2017 AMIA proceedings).

Conceptual Framework: We used the updated technology acceptance model (TAM) to guide the development of the patient-facing interface.⁶ TAM is a validated model that considers how users come to accept and use technology. Figure 1 displays the TAM, and indicates which factors the 4 patient-facing features of ProVis specifically support.

Figure 1.

Technology Acceptance Factors⁶ Influenced by the Proposed Solution, the ProVis application

Use Case Scenario: When the heart failure patient arrives, the receptionist enters the patient’s MRN into the ProVis application, installed on an iPad (see video demonstration). The MRN triggers ProVis to display the patient’s identifying information. The receptionist hands the iPad to the patient, and the patient verifies their identifying information. Then, the patient completes the PROMIS measures and views their summary visualization on ProVis while in the waiting room. When the patient presses the “submit” button, the application prompts the patient to return the iPad to the receptionist. Then, provider and patient review the PROMIS outcomes together, in conjunction with medication data, through the EHR interface, during the actual clinic visit.

Innovation: As the use case scenario describes, the proposed solution: (1) integrates PROMIS measures into the EHR with medication data to inform clinical decision making, (2) presents PROMIS outcomes to patients and clinicians using novel visualizations, (3) applies PROs to a population, heart failure, where symptom management is critical, and (4) facilitates communication between the patient and provider during the clinic visit.

Patient-Facing Design Process: We applied material design to create an industry-standard interface for use in populations with low technology literacy (see Supplementary Materials). Launched in 2014, Google’s material design is among the most widely accepted design frameworks. Advantages over previously popular minimalist frameworks include three-dimensional layering effects which enhance system usability for older users, as well as updated design patterns. A design pattern is a highly-tested solution to a commonly occurring problem in interface design. ProVis uses a responsive mobile-first framework, Bootstrap 3, for viewing on various screen sizes, including both the vertical and horizontal iPad orientations. To improve the interface’s learnability and memorability for older adults, we use clearly identifiable buttons, progress bars, consistent linear navigation, large font, colorful icons, high contrast, and system response times tailored to older users. To increase precision and efficiency, we coupled previously published infographics tested in low health literacy populations with PROMIS answer choices.⁷ Additionally, we created visualizations to summarize PROMIS outcomes for the patient directly after survey completion. In creating patient-facing summary visualizations, we provide immediate feedback to patients, engage them in their care, and highlight the survey’s usefulness to them. Within the summary visualization, we used locally developed materials to educate patients about their symptoms and enhance their situational awareness.

Integration with the EHR: The NewYork-Presbyterian (NYP) hospital enterprise uses multiple EHR vendors, including Epic, Cerner, Athena, and Allscripts. Instead of working with each vendor separately, we will integrate our clinician-facing visualizations into iNYP, an EHR-viewing system interoperable across multiple commercial EHRs within NYP. 12,000 physicians use the iNYP daily to view 4.5 million patient records. iNYP uses “clinical dashboards” to summarize and synthesize patient information for the provider. Our department oversees the development and maintenance of the dashboards. To prepopulate patient demographics in the ProVis interface, a back-end programmer integrated it with iNYP using Microsoft’s .Net. To integrate patient-reported data into the dashboard, iNYP will query the ProVis database server (see Hardware section) for the patient’s MRN.

Clinician-Facing Design Process: Clinicians view PROs within discrete packets of information, termed “cards”, within the iNYP clinical dashboard. The cards separate data into portions and support one particular clinical action each, to prevent information overload. One card summarizes recommended clinical actions for the provider based on the novelty and severity of each symptom. Because heart failure symptoms differ dramatically between individuals, and clinicians may wish to track different symptoms in different patients, we offer clinicians the ability to customize cards to individual patients. We anticipate that customization will increase utilization of PROs in the EHR. We used visualization techniques for time-oriented data⁸ to visualize PROs longitudinally in cards. For example, we visualize how patients’ symptoms scores vary according to medication dose and type (see Supplementary Materials).

Training: The Patient-Centered Outcomes Research Institute (PCORI) user’s guide on integrating PROs in EHRs recommends training administrators and clinicians prior to implementing PROs to prevent non-use.⁹ To help clinicians understand how to manipulate provider-facing visualizations and discuss them with patients, we will offer training at monthly cardiology team meetings. Additionally, we will train the clinic receptionists to engage patients in ProVis and provider user support. We will provide a phone number to report hardware problems.

Hardware: ProVis will be accessed using iPads located in cardiac clinic waiting rooms. The iPads access secure WiFi through an existing hospital network. To protect the iPads, the hospital uses rubberized cases and installs theft-deterrent software. The information systems, including the ProVis application itself and the databases housing patient-reported data from ProVis, use secure, encrypted, HIPAA-compliant servers. Physical access to the servers requires keycard authentication, and we secure electronic access via the Secure Shell (SSH) protocol.

Alternative Solutions, Strengths, and Weaknesses

We choose to implement ProVis in the waiting room for five key reasons. First, although our institution offers an online patient portal, monthly use for patients with access lies below 2%, consistent with the low adoption of portals across the US.¹⁰ Second, physicians can immediately act on information collected in the waiting room, decreasing legal liability. For example, if the patient screens positive for depression in the waiting room, the physician can immediately place them on an antidepressant trial. Third, because every patient the provider sees reports PROs from the waiting room, the provider can integrate PROs into their standard clinic workflow, which may increase PRO utilization. As such, we do not need to prompt providers with notifications to view PROs, and this may prevent alert fatigue. Fourth, the clinic receptionist can assist patients with the application in the waiting room, potentially increasing reliability and therefore clinician’s trust in the patient-reported information. Fifth, restricting the frequency of PRO collection to clinic visits reduces the burden of reporting on patients. Although collecting PROs in the waiting room demonstrates many strengths, one weakness is the inability to monitor symptoms between visits.

We chose to use an external, or “wraparound”, system for PRO collection, rather than collecting PROs using an application within the EHR system itself. External systems offer several strengths over internal systems, including the easy customization necessary to create our visualizations. However, external systems also demonstrate several weaknesses. First, although our system does not require any username or password, the clinical receptionist must links the external system directly to the EHR using the patient’s MRN. Second, our external system uses a unique protocol to communicate with iNYP, which is not easily generalizable to other EHR systems.

We chose to normalize patient-reported symptoms relative to the population distribution in only one clinician-facing visualization. In participatory design sessions, our clinician participants described population-level data as irrelevant to the heart failure clinic, which sees primarily vulnerable populations. Clinician participants placed more emphasis on patient perceptions of severity and changes to severity over time, so we emphasized that instead. Furthermore, the current version of ProVis does not normalize to the population distribution in patient-facing visualizations. This is a weakness, since seeing their information relative to the population may motivate behavior change in some patients. However, it is also a strength, since participants in usability testing worried that population information may prompt complacency (“I’m where everyone else is”) or despair (“I’ll never be able to fix this”).

Implementation and Dissemination Plan

Prior to implementation and dissemination, we will demonstrate the usability and usefulness of our visualizations to patients and clinicians, as well as their impact on PROMIS’s precision and efficiency (see Evaluation Plan section). As discussed above, we will work with the Center for Advanced Cardiac Care to integrate ProVis into electronic intake systems in our hospital’s heart failure clinics. Going forward, the iNYP clinical dashboard team at Columbia University Department of Biomedical Informatics will govern the cardiology PRO system in collaboration with the Center for Advanced Cardiac Care. Independent of implementation of the ProVis application, the visualizations hold potential for adaptation and use in any system incorporating PROMIS measures. Ideally, we would translate our visualizations to other systems through close collaboration with institutions participating in the NIH EASI-PRO initiative. We will share all of the tested visualizations and interfaces on GitHub.

Dissemination beyond the clinic to hospital administrators and policy makers is an important consideration. For example, Franklin and colleagues recently demonstrated that incorporating PRO data dramatically improved their readmission risk prediction model (c-statistic of 0.86 vs 0.65).¹¹ Hospitals use readmission risk prediction models to distribute resources, and the Centers for Medicare and Medicaid Services uses such models to standardize readmission rates between hospitals. Making PRO data available to administrators and policy makers may improve readmission risk prediction, among other pursuits.

Evaluation Plan

In addition to the evaluations our team already conducted (see first paragraph of the Proposed Solution section), we plan to complete the evaluations described below.

Further Usability Testing with Patients: We will recruit 20 patient users from the heart failure clinic for laboratory-based usability testing. The research coordinator will guide the end-users through an IRB-approved usability assessment protocol. The users will complete goal-oriented tasks such as reporting specific symptoms. The coordinator will video-record the screen during each interview using QuickTime Player and ask predetermined questions to encourage concurrent think-aloud. After each task, the coordinator will assess satisfaction, ease-of-use, and confidence with retrospective probing techniques. Quantitative measurements will include time to task completion (approximating efficiency) and the task error rate (approximating precision).

According to the ISO / IEC 9126-9 quality model, usability involves 4 dimensions: (1) understandability, (2) operability, (3) learnability, and (4) attractiveness.¹² Only 32% of usability-related publications focused on mobile health applications assessed all 4 dimensions.¹² However, assessing all dimensions is critical. For example, attractiveness, the least assessed dimension, plays the greatest role in a user’s decision to being using an application.¹³ To ensure we assess all dimensions, we will administer a field-based usefulness, ease-of-use, and ease-of-learning questionnaire¹⁴ to 20 randomly-selected patients after their clinic visit.

Validation of PROMIS Visualizations: We will conduct a two-arm randomized trial to evaluate the impact of visualizing answer choices on PROMIS’s precision and efficiency. A 20 person pilot study prior to the trial will generate parameters to estimate effect sizes in an a priori power analysis. General medicine patients will be block randomized to complete PROMIS measures with visualizations or without visualizations. For the gold standard, all patients will complete “legacy” questionnaires (well-validated and widely-used measures of the same concepts). At baseline, both arms will complete a survey to assess demographics, health literacy, and prior technology use.¹⁵ The primary outcome will be efficiency (time to completion) and precision in comparison with the gold standard.

Semi-Structured Interviews with Clinicians: We will conduct semi-structured interviews with clinicians to assess the predisposing factors (attitudes towards use), enabling factors (skills and support required for use), and reinforcing factors (perception that use makes a difference) for use of clinician-facing PRO visualizations in the EHR. We will analyze the interviews in multiple steps using a qualitative descriptive approach to uncover common themes. First, 2 researchers with training in qualitative methods will independently read each transcript, and define codes in a dictionary for the remaining analysis. Then, the 2 researchers will independently code all transcripts using codes corresponding to the coding dictionary. We will conduct 1 round of inter-coder comparison queries. The coders will meet to review, discuss, and arrive at consensus for the content coding. To enhance rigor, a third researcher with extensive experience in qualitative research will conduct an audit trail of the coding processes.¹⁶ To enhance confirmability, we will use “member checks” and peer debriefing among the data collection team.¹⁶ We will share summaries of the coded data with 3 clinicians and ask for their confirmation or revisions to interpretation.

Conclusion

Our proposed solution, ProVis, uses visualizations to display PROMIS measures and outcomes to heart failure patients in the waiting room, and display PROMIS outcomes to clinicians in the EHR. We believe ProVis will impact heart failure symptom management and prevent readmissions, as well as increase PRO utilization.