Using Health Information Technology to Support Use of Patient-Reported Outcomes in Rheumatology

INTRODUCTION

The growing measurement of patient-reported outcomes (PROs) combined with the increasing adoption of health information technology (IT) and electronic health records (EHRs) as a means of collecting PRO data at scale present an unprecedented opportunity for advancing the science and practice of rheumatology.¹ The standardization of electronic collection of PRO data and the incorporation of these data into routine clinical practice is anticipated to move the field of rheumatology closer to its goal of providing high-quality, affordable care, in ways that matter most to patients.² In addition, by facilitating timely and effective use of PROs without unreasonably adding to clinic workload, health IT has potential to bridge the gap between health care providers and patient perspectives in rheumatology by focusing the clinical encounter more on individual patient perceptions of disease impact.

Although a great number of validated PROs are available and commonly deployed in clinical practice, particularly for rheumatoid arthritis (RA), clinical workflows to meaningfully incorporate these data to monitor disease trajectories, guide treatment decisions, engage patients in disease management, and improve the patientcenteredness of care are lagging.³ Challenges remain in adequately and meaningfully leveraging these data in rheumatology, and few studies have examined the impact and feasibility of using health IT–based tools to standardize PRO collection during outpatient visits.^4,5

In this article, we review experiences with existing electronic efforts to collect and evaluate electronic PRO data in routine practice. Our findings are drawn from a structured review of the literature and include examples from 2 different national electronic collection efforts, 1 in the United States and 1 in Denmark. We discuss the lessons learned from previous work using health IT tools integrated with the EHR or as stand-alone applications to collect PROs in clinical care in rheumatology. Finally, we frame an agenda for future work supporting ways to meaningfully leverage electronic collection of PROs to improve the patient-centeredness of rheumatologic care.

ELECTRONIC COLLECTION OF PATIENT-REPORTED OUTCOMES AT INDIVIDUAL CLINIC LEVEL

We performed a semistructured review of the literature (detailed in Appendix 1) to identify published health IT tools to collect PROs in clinical rheumatology settings in the past 10 years. We searched for both EHR-enabled tools and stand-alone applications to aid collection of PROs. Special attention was paid to the design process, adoption and integration into routine care, patient and physician satisfaction with use, and evaluation of patient outcomes after implementation. We excluded studies about pediatric or orthopedic management as well as self-monitoring tools without health care provider participation.

An overview of the 10 health IT tools identified and evaluated in our review can be found in Table 1. We review each of these electronic tools, categorized by mode of PRO collection and EHR integration status (Fig. 1), and discuss what challenges and opportunities each of these integration methods provide.

Table 1.

Characteristics of electronic tools designed for PRO collection in clinical rheumatology

Reference	Tool/Intervention	Population (n)	EHR Integration	Mode of Collection	Outcome
Chua et al,11 2015	Electronic version of RAPID- 3 questionnaire. Patients access it by logging into an online Patient Portal within 7 d of visit.	RA (68)	Yes	Patient portal	Validation of performance characteristics of HER version of RAPID-3 compared with paper version
Collier et al,10 2009	Rheumatology OnCall (ROC). Facilitates access to data relevant to encounter, but located in separate parts of the EHR. Automatic calculation of DAS28 with a homunculus for joint counts.	RA (15 physicians)	Yes	Full HER integration	Physician satisfaction based on weekly and close-out survey
Dixon et al,14,15 2016	REMORA. Remote electronic collection of PROs between clinic visits through an app. Data automatically integrates with the EHR.	RA (21)	Yes	Remote monitoring	Patient and physician satisfaction
Li et al,13 2018	Online Epic EHR patient portal accessible from home to collect PROMIS PF-10a within 7 d of visit.	RA (1078)	Yes	Patientportal	Comparison of proportionof visits with documented PROMIS PF scores across age, race and ethnicity, and language and examination of trends over time
El Miedany et al,7 2010	Online PRO questionnaire accessible from any device the patient has available. Enables automatic collection of DAS28 and RAPID-3.	RA (211)	No	Web-based stand-alone	Equivalence of outcomes (RAPID-3 and DAS28) between electronic and paper formats; patients’ medication adherence
Newman et al,3 2015	Rheum-PACER. Integrates information from 4 data sources and reassembles them into actionable views and functions during clinic visit. Facilitates automatic calculation of CDAI.	RA (6725)	Yes	Full HER integration	Adoption (RAPID-3, CDAI), efficiency (time to review information), productivity (level of service and corresponding relative value units), and patient perception (patient activation, adherence, satisfaction)
Salaffi et al,16 2016	RETE-MARCHE. Telemonitoring system. Online platform to administer computerized questionnaires to collect RAID and CDAI.	RA (44)	No	Remote monitoring	Proportion of patients in clinical remission at 1 y (CDAI <2.8); radiographic progression; patient acceptance
Salaffi et al,8 2013	SPEAMonitor. Electronic touchscreen tablet to complete computerized versions of the BASFI and BASDAI.	Axial spondylo-arthritis (55)	No	Web-based stand-alone	Usability (in terms of patients’ acceptance, preference, and reliability) of computer-based questionnaires; feasibility (time taken to complete electronic vs paper questionnaires)
Schougaard et al,17 2016	AmbuFlex. Clinical telePRO tool that provides clinical decision support to suggest whether an outpatient visit is necessary.	RA (not known)	Yes	Remote monitoring	Describe experiences with respect to PRO data collection, the PRO-based automated decision algorithm, and PRO-based graphical overview for clinical decision support
Yen et al,6 2016	READY. Tablet application to collect PROs in the waiting room. Uses multiple validated PRO questionnaires that reflect RA symptoms and quality of life.	Not disease-specific; tested in RA (33)	Partiala	Stand-alone	Usability evaluation (think-aloud) and a time-motion study to observe changes in clinical workflow

Abbreviations: BASDAI, Bath Ankylosing Spondylitis Disease Activity Index; BASFI, Bath Ankylosing Spondylitis Functional Index; CDAI, Clinical Disease Activity Index; EHR, electronic health record; PRO, patient-reported outcome; PROMIS PF, Patient-Reported Outcomes Measurement Information System, physical function; RA, rheumatoid arthritis; RAID, RA Impact of Disease; READY, RhEumAtic Disease activity; REMORA, REmote MOnitoring of Rheumatoid Arthritis.

^aIntegration with a commercial EHR vendor occurred after the manuscript was published.

Fig. 1.

Depiction of PRO landscape. From left to right: “Paper” collection entails the patient filling out PRO questionnaires in the waiting room, before medical assistant enters results into the EHR system. In “Tele” systems patients can deliver PROs via personal devices at their convenience, and data are automatically transferred to the EHR. “EHR-based” is similar to the tele solution, but patients either enter PROs in the waiting room before their encounter on devices linked directly to the clinic’s EHR system or deliver PROs before their visits through the online patient portal. Finally, “stand-alone” tools are separate from the EHR system, and both patient and provider access questionnaires and results through an external online platform. Stand-alone systems additionally provide the opportunity for patients to monitor their PROs on personalized dashboards on their smartphone or feed data into online patient communities.

We found significant heterogeneity among the health IT–enabled PRO collection tools, which emphasizes the need for a general framework for systematic incorporation of the patient’s perspective in routine clinical care. We also found that published reports of rigorous digital tools are relatively few, and they are predominantly designed for use in caring for patients with RA rather than other rheumatological conditions.

Stand-Alone Digital Tools

We identified several examples of stand-alone digital tools. These tools exist entirely outside of the EHR, and are accessed by patients as well as clinicians through online links or Web-based applications.

READY (RhEumAtic Disease activitY) is an app (iPad)-based stand-alone ePRO application that enables the PRO data capture process to occur at physician offices using a tablet.⁶ It uses multiple validated PRO questionnaires that reflect symptoms and quality of life of rheumatic diseases, with the option to show trended scores. Usability issues were found in READY, including touchscreen sensitivity (not sensitive enough when users attempted to complete an action), interface design (layout and font size), and instruction and error messages. Despite these issues, most patients with RA who tested the app found READY easier to use than the current PRO paper questionnaire. This was due mostly to the larger font size and the ease of “tapping” rather than writing-out or circling answers. Workflow and activity changes were observed in a time-motion study, and overall the tool did not extend the total time of patient visit.

El Miedany and colleagues⁷ also developed a Web-based stand-alone system in which clinic staff manually entered ePROs into the EHR. A double-blinded, randomized controlled study including 211 patients with early RA was carried out. Patients were randomized to group 1, and completed an ePRO questionnaire monthly, or group 2, and continued to complete standard paper PROs. The primary endpoint was the difference in outcomes of RAPID-3 and DAS28 in both groups and secondary outcomes included medication adherence. After 12 months, there was no statistically significant difference in disease activity measures between the 2 groups. Based on pharmacy data, 89.6% of group 1 patients were adherent to their medications in comparison to 70.5% in the control group (P<.01). In group 1, fewer patients (5.7%) stopped medication because of intolerance compared with controls (19%). Unfortunately, it was unclear from the methods exactly how the collected PROs were implemented in clinical care or how they were acted on to reach these notable results.

In axial spondyloarthritis, a touchscreen tablet application for ePRO collection (SPEAMonitor) resulted in a lower mean amount of time spent completing the questionnaires (Bath Ankylosing Spondylitis Functional Index and Bath Ankylosing Spondylitis Disease Activity Index) than on paper (5.1 vs 7.9 minutes).^8,9 The tablet was also reported to be easier to use by patients, and age, computer experience, or education level did not influence results. The tool was designed to be user-friendly to patients by combining cartoon, writing, and voice and presenting only one question at a time. It was validated against a standard paper questionnaire and there was excellent score agreement. As only the validity of the tool was assessed, implementation into actual clinical workflow was not discussed.

Fully Integrated Electronic Health Record–Based Tool

Several studies examined tools that were fully integrated with an EHR. Full integration involves developing methods to capture key PRO data elements in structured fields within the EHR (just as vital signs or laboratory tests are currently captured).

The Rheum-PACER software integrates and reassembles information from the patient (via a touchscreen questionnaire), nurse, physician, and Epic EHR into a series of actionable views in an application outside the EHR, while feeding information back to the EHR. Core functions include tools to facilitate documentation of key aspects of RA care for patient and provider and graphical displays to examine outcome trends over time. The tool was integrated and tested in 3 rheumatology clinics. Over 2 years, 86% of patients and rheumatologists used the software. Physician-reported chart review time decreased from a median of 5 minutes before Rheum-PACER to 4 minutes after Rheum-PACER and implementation and documentation time decreased from a median of 7 to 5.5 minutes, although neither reached statistical significance. In addition, a strong, significant correlation was seen between physician use of the software and disease control (weighted Pearson correlation coefficient 0.59), and the investigators showed a relative increase in patients with low disease activity of 3% per quarter.³

Similarly, the Rheumatology OnCall (ROC) application gathered rheumatology-pertinent data from laboratory, microbiology, pathology, radiology, and pharmacy information systems within the EHR, and also included a disease activity calculator.¹⁰ In total, 15 rheumatologists (of 47 in the clinic) accessed ROC during outpatient visits at the time of the clinical encounter. Trended clinical and laboratory data populated a dashboard that allowed the clinician to quickly assess trends and facilitated the tracking of disease activity. The investigators also conducted a 12-week prospective cohort study to assess physicians’ attitudes toward use. By administering weekly physician surveys, they demonstrated that most physicians found graphing trends useful. In a close-out survey, the most physicians reported that use of ROC improved patient care, and that they would continue to use ROC in daily RA patient care. However, the frequency of use of the ROC was inconsistent among physicians, pointing to general barriers to health IT adoption. No objective assessment of patient outcomes was carried out. Planned enhancements to the application include disease activity calculators for systemic lupus erythematosus, ankylosing spondylitis, and osteoarthritis.

Patient Portals Within the Electronic Health Record

Similar to fully integrated tools, 2 studies examined the use of online EHR patient portals to collect ePROs. These portals permit patients to retrieve their health records and to enter additional information available to the clinician. Chua and colleagues^11,12 prompted patients with RA to complete an online version of the RAPID 3 via the patient portal anytime in the week before their upcoming appointment. The EHR automatically calculated and interpreted the disease activity score for the electronic RAPID 3, and prior scores could be easily accessed to facilitate comparisons. The study mainly focused on validating the electronic RAPID 3 compared with the paper version and they found no statistical difference by collection method. Limitations included the lack of a qualitative component to the study to help understand patients’ personal experiences in terms of acceptance, convenience, and preference when answering the online questionnaires. In addition, only 52.5% of the clinic’s 1130 patients with RA had an active EHR patient portal, possibly representing a subpopulation of patients more comfortable with using technology.

Li and colleagues¹³ likewise assessed the collection of PROs from patients with RA using a paper version and an online form through an Epic EHR patient portal (MyChart). Patients with an activated portal received a message 7 days before their appointment with a link to the PROMIS (Patient-Reported Outcomes Measurement Information System) physical function (PF) form. They examined the proportion of patients completing the form electronically or on paper and explored patient factors associated with method of completion. One and a half years after the online patient portal survey became available, only 19.3% of visits had an associated online PROMIS PF score, and no patients used it persistently; more than half of patients abandoned the online portal for PRO reporting after a single use. In addition, the investigators showed that use lagged among racial and ethnic minorities. Online portal use did not decrease the burden of data collection and data entry for staff as anticipated, because the clinic workflow still required that medical assistants assessed whether a patient already completed an online version of the survey at the time of patient checkin. This process was time-consuming and faulty, as evidenced by the numerous patients who completed both online and paper surveys within 7 days of an in-person visit. Limitations included the lack of data collection regarding patients’ home Internet and computer access as well as health literacy level, which might have explained low uptake in some groups. Issues of portal access, enrollment, satisfaction, and persistence of use were considered in both of the patient portal studies, but not addressed in the study design.

Digital Tools for Use Outside Clinical Settings that Communicates with the Electronic Health Record (Tele)

Digital tools that communicate with the EHR include tele solutions that enable patients to generate important data outside of the hospital or clinic setting as often as needed.

REMORA (REmote MOnitoring of Rheumatoid Arthritis) examines whether electronic collection of PROs directly from patients daily between clinic visits can enhance clinical care.¹⁴ The “beta app” was tested by 8 patients for 1 month during which they completed routine questionnaires at home, with the data integrated into the EHR. Question sets included 7 visual analogue scales for daily symptoms including pain and fatigue; weekly self-reported counts of tender and swollen joints, flares, and impact on work; and monthly completion of the health assessment questionnaire (HAQ).¹⁵ Patients were told that patient-generated health data would only be reviewed in clinic at the time of the encounter. Both patients and providers reported that the app and graphed results improved clinical care, made the consultation more personal, and demonstrated gradual improvements in symptoms in response to treatment that may otherwise have been missed, thereby supporting decision making. The app was therefore well received among a small, enthusiastic group of patients and providers, but future studies must determine feasibility of large-scale implementation and real impact on patient outcomes.

Salaffi and colleagues¹⁶ showed that a scheduled intensive strategy, based on a tele monitoring system, was a useful approach to achieve remission after 1 year in 44 early patients with RA. Their telemedical tool, “RETE-MARCHE” included a Web site platform that patients accessed from their personal device to answer questions related to RA Impact of Disease (RAID). Importantly, the system immediately generated warnings to both the patient and the clinician case manager whenever it detected that the patient’s self-monitoring showed deterioration in one or more of the symptoms monitored with RAID, in contrast to the REMORA study in which data were reviewed only at the next clinic visit with the provider. A higher percentage of patients in the telemonitoring group achieved Clinical Disease Activity Index (CDAI) remission versus patients managed conventionally (38.1% vs 25% at year 1, P<.01), and time to achieve remission was significantly shorter in the active group. In addition, the investigators determined the degree of patients’ acceptance of the telemonitoring platform. An overall average of 4.28 on a scale of 5 indicated high patient satisfaction with the system. Furthermore, 90.5% said that they would keep using the system in the future. However, the study lacked information about clinicians’ attitudes toward using the system, and the sample size of patients was small.

Finally, one study also described experiences with implementing a telePRO system, AmbuFlex, as the basis for follow-up in malignant and chronic diseases, including RA.¹⁷ The patients defined the need for an outpatient consultation by delivering PROs using an online or paper questionnaire. The results were visually summarized and accessible via a link from the EHR, and the PROs were used to decide whether a patient needed an outpatient visit, based on an automated decision algorithm and published cutoff values. A total of 300 patients with RA from 2 outpatient clinics were referred to telePRO follow-up with an initial response rate to the questionnaire of 93% among these patients. No data were shown separately for patients with RA about how many of the incoming PRO questionnaires that could be handled with no further contact to the clinic than the PRO.

ELECTRONIC COLLECTION OF PATIENT-REPORTED OUTCOMES AT NATIONAL HEALTH SYSTEM LEVEL

As seen from the previous examples of digital tools, patient data are typically collected in a plethora of ways in individual health systems. There are, however, examples of nationwide efforts to reduce the heterogeneity in the types of data collected. Collecting PRO data from the patient once in connection to a clinical encounter in a structured and standardized manner has potential to help rheumatologists to provide better patient-centered care, researchers to have access to more accurate and comprehensive datasets, and allows departments to meet quality measures and conduct audits.

One example of a national PRO collection effort is the Danish DANBIO register. It is a national clinical research register and a data source for rheumatologic diseases (RA, axial spondyloarthritis, and psoriatic arthritis) for monitoring clinical quality at the national, regional, and hospital levels.¹⁸ Patients report their symptoms on a touchscreen computer in the waiting room before the clinical encounter, and the data are later supplemented with clinician-derived data. Core variables, such as diagnosis, year of diagnosis, age, and gender, are registered at the first visit to a specialized hospital department or specialized private practice. Data entered at later visits include PROs for disease activity, pain, fatigue, functional status, and physician-reported objective measures of disease activity, treatment, and C-reactive protein. For some patients, variables such as quality of life, sociodemographic factors, lifestyle, and comorbidity are also registered.

DANBIO has high nationwide coverage and completeness on key data variables, and reporting is mandatory by law for all patients with RA. In 2015, the DANBIO cohort comprised approximately 26,000 patients with RA, 3200 patients with axial spondyloarthritis, and 6200 patients with psoriatic arthritis. Digital data are summarized graphically and used to populate a scoreboard for shared decision making between clinicians and patients during the visit (Fig. 2). The system is not yet integrated with the EHR, but works as a stand-alone tool, accessible from a secure Web site. DANBIO also works as a quality registry, an audit and feedback tool, and provides secondary use of data for research while fulfilling its primary purpose of supporting clinical care. No studies have yet been carried out to assess physicians’ use of the DANBIO scoreboard to guide clinical decision making during patient visits.

Fig. 2.

Patient scoreboard in the Danish DANBIO register. The patient’s disease is followed over time, and the scoreboard provides the physician and the patient with a comprehensive view of the treatment. Red: Severely active disease. Yellow: Moderately active disease. Green: Mildly active disease. (Courtesy of DANBIO, Denmark; with permission.)

Building a national infrastructure around the electronic collection of PRO data is also one of the aims of the American College of Rheumatology’s RISE registry (Rheumatology Informatics System for Effectiveness). RISE passively extracts EHR data from participating practices, aggregates and analyzes the data centrally, and feeds this information back to clinicians as actionable data using a Web-based interface.¹⁹ RISE is also a vehicle for quality reporting in the Merit-based Incentive Payment System (MIPS), a pay-for-performance program for physicians caring for Medicare beneficiaries. RISE currently houses one of the largest collections of outcomes among individuals with RA in the world, with almost half of 95,600 patients having an RA functional status assessment PRO measure and RA disease activity assessment in 2017.²⁰

Unlike research performed within DANBIO, which is not integrated with a specific EHR, RISE has tackled the challenge of aggregating data from many different EHR systems that collect different measures during routine care. Although structured PRO data for patients with RA are available for many practices and collected via a diversity of methods (eg, paper, and then manually inputted into structured data fields in EHR note templates), some rheumatologists still record this information in the text of clinical notes. Therefore, RISE is working toward including critical information from unstructured fields like clinical notes by incorporating algorithms for text mining and natural language processing. These enhancements will increase the availability of PRO and other data for clinical research, performance measurement, and population health management at the individual practice level.

DISCUSSION

This review illustrates that current electronic PRO systems within EHRs and freestanding systems outside the EHR have varied significantly to date in their focus and features. PRO assessments have been incorporated into the clinic visit by several electronic means, and workflow considerations have been fundamental to the successful integration into care settings. Overall, well thought-out collection of electronic PROs has potential to bridge the gap between health care providers and patient perspectives in rheumatology by focusing the clinical encounter more on the individual patient’s experience and perceptions of disease impact. We summarize the key lessons learned from the literature to date in the following sections.

Lesson 1: the Importance of Attention to Workflow

Studies of many of the previously mentioned health IT tools, whether fully integrated or stand-alone, raised concerns about workflow changes in clinical practice and patient engagement after implementation as barriers to adoption. Systematic ePRO collection and delivery to health care providers to use for clinical decision making will clearly not be fully embraced if it is perceived to disrupt or impede clinical workflow. This was exemplified by Li and colleagues,¹³ in which one of the investigators’ goals for online collection was to decrease the burden of data collection and data entry for the clinic staff. However, the workflow still required that medical assistants assessed whether a patient already completed PROs through the patient portal at the time of patient check-in, and this process was both time-consuming and error prone. Yen and colleagues⁶ also received mixed feedback from clinicians toward their stand-alone PRO tool with concerns mainly questioning the implementation of a mobile application into the clinical workflow and how it may impact patient care.

One way to avoid the pitfall of negative or unintended consequences during implementation is to identify all stakeholders who are likely to be involved with the tool early in the development and engage these stakeholders in the design and testing process and ultimately in ongoing improvements. Participatory design, such as human-centered design thinking, deeply engages the end-user in the design process.²¹ Similarly, process redesign methodologies, such as plan-do-study-act (PDSA), focus intensively on changing the workflows by small-scale rapid-cycle tests of change as a form of iterative “learning in action” to ensure that any modifiable obstacles to implementation are addressed.²² Based on our review, few studies have used human-centered design thinking and few report systematic redesign or quality improvement methodologies. Attention to workflow considerations will reduce burden of collection, increase reliability of PRO capture, and reduce nonuse of results. For example, the most successful efforts to date, such as Rheum-PACER, undertook a rigorous workflow redesign process, with careful attention to physician and staff workload.³

Before building Rheum-PACER, Newman and colleagues²³ additionally explored how a computerized version of the MDHAQ (Multidimensional Health Assessment Questionnaire) could be adopted for routine use in a busy, complex, and resource-constrained rheumatology department. PDSA was used in 2 cycles of workflow redesign during implementation of a touchscreen questionnaire, and it was shown that the routine use of a touchscreen questionnaire can be tightly integrated with routine clinical care by focusing on continuous problem solving as well as design and workflow issues.

Our own experiences using a human-centered design process included patients, providers, and staff at 2 outpatient clinics, and evolved around the development of a PRO dashboard for patients with RA.²⁴ We conducted clinic observations, interviews with stakeholders, patient and provider focus groups, and iteratively prototyped and tested the dashboard in clinical settings in close collaboration with health IT designers. The result was a PRO-focused dashboard that supports a conversation about treatment, centered on the patient’s most salient goals, concerns, and experiences, with the overall aim of facilitating shared decision making.

Although the process of designing the tool’s user interface and the selection of PROs included in the tools are infrequently discussed,^25,26 these are critical steps in the development of successful digital tools.⁴ This ensures that the integration happens in a way that meets the needs of the various users, and may support relevance and appropriateness with the goal to reduce nonresponse or sluggish uptake.

Lesson 2: Integration into the Health Information Technology Ecosystem

Increasingly, workflow redesign requires consideration of the health IT ecosystem, including the capabilities of EHR systems and the capacity of those systems for data integration. This has been a significant barrier for many efforts, and there is a clear lack of interoperability between systems, without the possibility of easily customizing data flow into EHRs. As an example, despite being designed for quality and research purposes and as a point of care tool to aid in clinical decision making, the DANBIO register is still, almost 20 years after go-live, not integrated into hospital EHRs due to interoperability issues between IT systems and different EHR vendors.

As evidenced by the tools reviewed here, success with EHR integration has varied significantly. To be useful, PRO data should be easy to interpret and actionable.²³ Integrating electronic PRO data directly into the EHR ensures that clinicians get the right information at the right time (during the encounter) and allows tracking over time in relation to other clinical information available in the EHR (eg, laboratory tests and medication). In contrast, incorporating stand-alone applications into the clinic visit often requires that the physician exit the EHR to retrieve and view data. Alternatively, clinic staff can manually enter data such as calculated disease activity scores from the stand-alone application into the EHR, again adding an extra manual task to the workflow.²⁷ Stand-alone tools, on the other hand, are often less expensive, and provide a good opportunity to test if an electronic capture system works for patients and providers in practice, or to assess the validity of new ePROs against their paper counterparts as seen in several of our included studies.⁷ Some EHR vendors, including Epic and Cerner systems, are creating systems and interfaces that allow outside applications to be supported and integrated with their software, although successful demonstration projects in rheumatology are still lacking, and interfaces to write to the EHR (rather than merely support read-only access) has been elusive.

Lesson 3: Minimize Patient Burden

From the patient’s point of view, minimizing data collection burden is essential and fatigability of use is important. Features enabled by online EHR patient portals offer a platform to further coordinate and develop PRO collection beyond the clinical encounter, further enhancing patient PRO monitoring.²⁸

Li and colleagues¹³ found that even though most of their included patients had an active online portal account, only a fraction (fewer than 20%) completed the patient portal PROMIS survey before an in-person visit. Uptake of portals was found in a systematic review to differ by patient-specific factors with lower use by racial and ethnic minorities and lower use with lower education level or literacy,²⁸ and this was furthermore echoed by Li and colleagues.¹³ More widespread acceptance will require attention to overcoming these barriers and addressing usability and patient-perceived value to engage certain populations. This might include providing materials about enrollment, tools, and surveys in multiple languages and for patients with low literacy. In addition, more work is required to ensure that collected data are useful to patients and meaningfully impacts factors such as their ability to monitor and manage their condition and communicate with their physician. Future studies should address issues of portal access, enrollment, satisfaction, and persistence to accommodate the needs and preferences of diverse populations.

Lesson 4: What Happens Between Visits Matters

Technology, such as smartphone apps, enables patients to generate important data outside of the hospital setting as often as needed and share it with their providers to expand the depth, breadth, and continuity of information available.²⁹ Digital tools relying on these tele-strategies allow PRO data capture between visits, and provide a more accurate and quantifiable representation of disease activity that can be incorporated into clinical decision making as seen with REMORA.¹⁴ Rather than relying on retrospective patient recall at clinic encounters, with its inherent biases, remote collection of PROs can help develop a clear picture of symptoms over time and highlight areas of importance that might be missed when relying on patients’ subsequent recall during the consultation.³⁰ Patients care about what happens between visits, and feel that reporting between hospital or clinic visits better reflects their health. Patients using the REMORA app reported that the app made care “more personal to you” and “found it made a difference, because it wasn’t all me telling him and trying to remember. The information was there so you’ve got solid proof straightaway.”¹⁵ This feedback suggests that the data recorded remotely between visits matters, and that it most likely will have a number of benefits for both patients and clinicians.

This functionality, though, presents challenges, such as reaching and engaging patients without Internet access at home and determining the appropriate clinical response (if any) for PRO scores that indicate problems warranting immediate clinical attention. This is particularly true because physicians currently express that data review burden in clinical practice is already high, even without data collected outside the clinical setting. Finally, strict attention to the accuracy of clinical decision support that recommends a clinical response should be given, to ensure that safety in relation to detecting patients in need of attention is not compromised.¹⁷

Lesson 5: Pragmatic Study Designs

Despite the significant enthusiasm about digital tools in clinical practice, rigorously tested tools that clearly improve patient outcomes are scant. Internally, the use of evaluation and quality improvement techniques can be used to illustrate how PRO implementation adds value in a clinical encounter. Demonstrating value to a broader audience requires additional outcomes, including the user experience (patient and provider attitudes, satisfaction with the system, doctor-patient communication), health services outcomes (clinical actions taken, referrals), and patient outcomes.³¹

A recent editorial from The Lancet pointed out that “continuing to argue for digital exceptionalism and failing to robustly evaluate digital health interventions presents the greatest risk for patients and health systems.”³² Randomized controlled trials, the gold standard of evidence, are rarely used in evaluation of digital health interventions and are often challenging to fund; only one of the rheumatology-specific tools we included underwent this sort of rigorous testing. This might be due in part to the costs of conducting such a trial, which are high relative to the tools’ perceived very low level of risk for the patient. Randomized controlled trials also can be difficult in the experimental evaluation of these tools because of cost, time, and difficulty (including acceptability) in randomizing different providers or health systems to the health IT intervention.

However, rheumatology still needs to build an evidence base showing which strategies are most successful in improving patient outcomes, minimizing adverse events, and maximizing patient as well as provider satisfaction. Quasi-experimental or pragmatic study designs without randomization might be one way to thoroughly evaluate digital interventions. Such studies compare an outcome of interest by using pre- and postintervention measurement of the same measure, and have been shown to be useful when evaluating new health system interventions.³³

SUMMARY

Technology exists that can foster understanding the patients’ experience of their illness in a way that was previously not possible. Early experience in using technology to capture this information through PROs within the field of rheumatology suggests that careful attention to human-centered design, including detailed workflow planning, consideration of patient and physician burden, integration into the health IT ecosystem, and delivering information to the right person at the right time to ensure it is actionable are important. For PRO collection to have a meaningful impact on patient outcomes, all of the previously described considerations need to be woven into the design of applications, and importantly, they must be tested in diverse health systems and in diverse populations, including diseases other than RA, to ensure they are simple to interpret, useful for clinical decision making, and effective in impacting outcomes.

KEY POINTS.

• Well thought-out collection of electronic patient-reported outcomes (PROs) may bridge the gap between providers and patient perspectives by focusing the encounter more on the patient’s experience and perceptions of disease impact.

• This review illustrates that a number of digital tools to collect PROs have been developed both within electronic health records and in freestanding applications, most focused on rheumatoid arthritis.

• Work to date demonstrates that attention to design, including workflow planning, consideration of user burden, integration into health–information technology ecosystems, and delivery of accurate and timely information are important to successful implementation.

• Tools must be tested in diverse health systems and populations to ensure they are simple to interpret, useful for clinical decision making, and effective in impacting outcomes.

APPENDIX 1: STRUCTURED REVIEW

We performed a structured review of the literature to assess the nature and extent of current digital tools being used to collect PROs in clinical rheumatology settings. Specifically, we focused on the design; implementation strategies; EHR integration status; whether clinics, providers, and patients adopted the new form of administration; and how it impacted clinic workflow. In addition, we focused on evaluation of patient outcomes, time to document, and satisfaction as measures of effectiveness.

With the assistance of a professional librarian, we searched 2 electronic databases (PubMed and EmBase), from August 2018 and 10 years back; MeSH terms and search concepts are listed in Table 2. We evaluated gray literature, including proceedings from major rheumatology meetings (American College of Rheumatology [ACR] and the European League Against Rheumatism from 2008 to 2018), and conducted hand searches of reference lists of retrieved articles.

Table 2.

MeSH terms used for structured review of digital tools to collect PRO

Search Concepts	Search Terms
1. Patient-reported outcomes	((((“Patient-Reported Outcome” OR “Patient-Reported Outcomes” OR “Patient Reported Outcomes” OR “Patient Reported Outcome” OR “Patient Reported Outcome Measure”[MeSH] OR “Patient Reported Outcome Measures” OR “PROMs” OR “PROMIS” OR “PRO-PM” OR “e-PROM” OR “e-PROMS” OR “Self Report”[MeSH] OR “Disease Activity”))
2. Digital tools	(((“Mobile Applications”[Mesh] OR mhealth OR m-health OR “mobile health” OR “health IT” OR “health information technology” OR digital OR ipad OR “mobile apps” OR “Smartphone” OR “EHR” OR “electronic health record” OR “electronic health records” OR “mobile device” OR “mobile devices” OR “EMR” OR “electronic medical record” OR “electronic medical records” OR “patient portal” OR “Patient Portals”[Mesh] OR “Electronic assessment”)))
3. Diseases	(((“Rheumatic Diseases”[Mesh] OR “rheumatoid arthritis” OR “arthritis, rheumatoid”[mesh] OR rheumatology OR rheumatologic OR rheumatologist OR rheumatologists OR Spondylarthropathies [MeSH] OR “Lupus Erythematosus, Systemic” [MeSH] OR “Scleroderma, Systemic” [MeSH] OR “Vasculitis” [MeSH] OR “Myositis”[MeSH] OR “Mixed connective tissue disease” [MeSH] OR “rheumatic disease” OR “rheumatic diseases” OR “rheumatoid arthritis” OR lupus OR scleroderma OR vasculitis OR myositis OR “ankylosing spondylitis” OR “psoriatic arthritis” OR “reactive arthritis”)))

Studies evaluating use of PROs in a clinical trial or in a pediatric setting were excluded, as were articles describing use of PROs in evaluating orthopedic management of musculoskeletal conditions such as total joint arthroplasty for osteoarthritis. Studies on applications that allow patients to track symptoms and self-manage their disease without involvement of the clinic or clinician were also excluded.

We identified 192 titles by the search, and following screening of titles and abstracts, 10 tools were included in this review. One additional tool was identified by screening the ACR abstract archive. Nine of 10 studies targeted patients with RA and their providers. Patient and provider satisfaction was the most common outcome among the included studies. Validation of electronic collection of validated PRO questionnaires against their traditional paper counterparts were reported by 5 studies. Design, workflow changes, and adoption were less frequently mentioned. Explicit discussions of whether this strategy led to improvements in patient outcomes, were rarely included, demonstrating the need for guidance on how to best evaluate tools to maximize their use.