Abstract
Objective
Integrating patient-reported outcomes (PROs) into electronic health records (EHRs) can improve patient-provider communication and delivery of care. However, new system implementation in health-care institutions is often accompanied by a change in clinical workflow and organizational culture. This study examines how well an EHR-integrated PRO system fits clinical workflows and individual needs of different provider groups within 2 clinics.
Materials and Methods
Northwestern Medicine developed and implemented an EHR-integrated PRO system within the orthopedics and oncology departments. We conducted interviews with 11 providers who had interacted with the system. Through thematic analysis, we synthesized themes regarding provider perspectives on clinical workflow, individual needs, and system features.
Results
Our findings show that EHR-integrated PROs facilitate targeted conversation with patients and automated triage for psychosocial care. However, physicians, psychosocial providers, and medical assistants faced different challenges in their use of the PRO system. Barriers mainly stemmed from a lack of actionable data, workflow disruption, technical issues, and a lack of incentives.
Discussion
This study sheds light on the ecosystem around EHR-integrated PRO systems (such as user needs and organizational factors). We present recommendations to address challenges facing PRO implementation, such as optimizing data collection and auto-referral processes, improving data visualizations, designing effective educational materials, and prioritizing the primary user group.
Conclusion
PRO integration into routine care can be beneficial but also require effective technology design and workflow configuration to reach full potential use. This study provides insights into how patient-generated health data can be better integrated into clinical practice and care delivery processes.
Keywords: patient-reported outcomes, electronic health record, clinical workflow, sociotechnical system, health information technology
BACKGROUND AND SIGNIFICANCE
Patient-reported outcomes (PROs) are a key measurement of the effectiveness of patient-centered care.1,2 PROs include patients’ self-reported symptoms, functional status, and health-related quality of life.2 Studies show that symptoms directly reported by patients more accurately reflect their health status than through clinician report.3,4 Potential benefits of using PROs include more accurate symptom detection, enhanced patient-provider communication, and improved patient outcomes.5–8 Additionally, PROs are beginning to be utilized to evaluate the quality of health-care delivery.9 PRO-based performance measures quantify care quality based on the outcomes experienced by patients and can compare the performance of different provider groups.9–11
While PROs have been widely used in clinical research, their use in the provision of clinical care is relatively new.2 A major step toward weaving PROs into routine care is PRO integration into the electronic health record (EHR). Currently, EHR patient portals enable patients to report symptoms electronically and the results are auto-populated within the EHR.1,12 PROs with EHR integration have been leveraged to facilitate automated triage for psychosocial care12 and can achieve more standardized and efficient clinical documentation and workflow.7,13
However, research has revealed technical, logistical, social, and organizational challenges of integrating PROs into clinical care.1,14,15 These challenges include uncertain clinical benefits, task overload, clinical workflow disruption, and lack of longitudinal data display1,14,15. A recent study found that physician buy-in and technology usability were key to successful PRO implementation.15 Additional challenges may arise when PROs are integrated into the workflows of different clinical stakeholders, such as physicians, social workers, and medical assistants (MAs).12,15 Studies have found that different types of providers interact with health information systems with distinct needs and goals.16–18 While much attention has been paid to physicians’ feedback on PROs,1,15,19 little research has examined the perceptions and needs of other clinical stakeholders.
In this study, we examine how well an EHR-integrated PRO system fits the needs and clinical workflow of various clinical stakeholders. According to the fit between individuals, task and technology (FITT) framework,20 IT adoption in clinical settings depends on the fit between the attributes of individual users (eg, needs, motivation), the technology (eg, usability, functionality), and the clinical tasks and processes (eg, organization, task complexity). Drawing on the FITT framework, we conducted in-depth interviews with providers to examine user preferences, contextual factors, and pain points that need addressing in their use of a PRO system. Understanding end-user perspectives will help us address the challenges facing PRO integration into routine clinical care.
MATERIALS AND METHODS
Research setting and system design
Northwestern Medicine has developed and implemented a system that integrates PROs into the EHR (Epic)—the Northwestern Medicine Patient Reported Outcomes (NMPRO). Two clinics within the orthopedics and oncology departments have administered Patient-Reported Outcomes Measurement Information System (PROMIS) computer adaptive tests (CATs) to quantify important symptoms and functions. Unlike traditional, static measures, PROMIS CATs offer high measurement precision with very few questions.21,22 The NMPRO workflows in the 2 clinics are different. In orthopedics (Figure 1), assessments are administered both presurgery and postsurgery to monitor patient pain and function over time. In oncology (Figure 2), the main aim of NMPRO is to optimize PRO completion prior to clinic visits and provide triage for psychosocial and supportive care.
Figure 1.
NMPRO Workflow in the Department of Orthopaedic Surgery’s Joint Replacement Program. NMPRO: Northwestern Medicine Patient Reported Outcomes.
Figure 2.
NMPRO Workflow in the Robert H. Lurie Comprehensive Cancer Center (RHLCCC). NMPRO: Northwestern Medicine Patient Reported Outcomes; RHLCCC: Robert H. Lurie Comprehensive Cancer Center.
NMPRO has been in use within the Department of Orthopaedic Surgery’s Joint Replacement Program since October 2016. Patients complete PROMIS CATs that assess pain interference, physical function, and social function, as well as a Pain Intensity Short Form both prior to surgery and postsurgery at 3 weeks, 5 weeks, 3 months, 6 months, 1 year, and 2 years. The preoperative assessment is triggered by an assessment order once surgery is scheduled. Patients can complete the assessment either in clinic with the aid of MAs or remotely using a patient portal (MyChart). Following surgery, a provider places an order for a series of assessments. Patients receive an email with instructions to complete the assessment via MyChart when each postoperative time point becomes available. Patients who do not complete the assessment prior to visit are asked to fill it out in the clinic using the exam room workstation, either on their own, with the help of a family member, or aided by MAs. It takes patients less than 5 min to complete the assessment. As of May 2018, 1167 patients have completed at least 1 assessment.
NMPRO has also been active in the Robert H. Lurie Comprehensive Cancer Center (RHLCCC) since September 2017, after replacing a different technical approach.12,23 The NMPRO assessment includes PROMIS CATs for pain interference, physical function, fatigue, depression, and anxiety, as well as 2 custom questions to assess social work needs and nutritional status. Assessments are triggered by appointments with oncologists. An email reminder is automatically sent to patients 72 h prior to an appointment. If a patient does not complete the assessment before a visit, MAs will launch the PRO assessment for patients in the exam room. The PRO assessment contains about 40 questions and takes less than 10 min to complete. As of May 2018, 5394 RHLCCC patients have completed at least 1 assessment.
In both clinics, PROMIS CAT scores are automatically populated in the EHR. PROMIS uses T-scores with a mean of 50 and a standard deviation of 10 in the US general population.22 High scores indicate more of the concept being measured. In orthopedics, after the patient has completed more than 1 assessment, a trending graph is available to show scores from past assessments. Within RHLCCC, scores in the severe range automatically trigger notifications within the EHR. For severe pain, fatigue, and physical function issues, a message is sent to the patient’s treating oncologist and nurse. Additionally, severe depressive or anxiety symptoms, social work needs, and nutrition concerns trigger messages to the social work and dietitian pools.
Data collection
This study employed a qualitative interview method to examine provider perspectives on the NMPRO system. Qualitative interviewing has been widely used to examine the usability of technologies and to understand how users integrate technology into their everyday lives16,24. We conducted in-depth interviews with 11 providers between September 2017 and January 2018 to understand their experiences interacting with NMPRO. Participants were recruited via a purposive sampling method. We first identified a list of providers who had interacted directly with NMPRO in the 2 clinical settings. Recruitment messages were sent via email, describing the purpose of this study and the voluntary nature of participation. We recruited participants until thematic saturation was achieved.25 Interviews averaged 37 min (range 19–63 min) and followed a semistructured protocol (see Supplementary Appendix A).
Of the 11 participants, 2 were from orthopedics and 9 were from the Cancer Center. The uneven number of participants from the 2 clinics reflected the different scales of NMPRO implementation. All interview participants had to some extent interacted with NMPRO. The 11 participants and their roles are presented in Table 1. Four (36%) participants were male, and 7 (64%) were female. This study was approved by the Northwestern University Institutional Review Board and written consent was obtained from all participants prior to the interview.
Table 1.
Characteristics of interview participants
| Clinic | Role | Number |
|---|---|---|
| Orthopedics | Surgeon | 2 |
| Oncology | Oncologist | 3 |
| Psychologist | 1 | |
| Social worker | 2 | |
| Medical assistant | 3 |
Data analysis
All interviews were audio-recorded and transcribed. We conducted open coding and axial coding to analyze the data.25 The first 2 authors open coded 4 interviews together to identify each instance in which participants talked about their interactions with and attitudes toward NMRPO. This process produced 56 distinct codes, such as “administering PROs is burdensome” and “scores are easy to interpret.” The 2 authors then conducted axial coding by grouping open codes that were conceptually similar.25 Examples of axial codes included: “PRO collection,” “PRO review,” and “usability issues.” We resolved discrepancies and developed a consensus codebook encompassing 18 distinct codes (see Supplementary Appendix B). The remaining eight transcripts were then evenly divided between the 2 authors and were coded independently following the codebook.
After completing axial coding, the whole team met and collectively identified preliminary themes. Themes that lacked representation in the data were dropped and similar themes were combined. The final 9 themes were selected via consensus among all the authors to represent the most salient perspectives of users. Following Ammenwerth et al’s FITT framework (Figure 3),20 we grouped these themes into 3 categories: individual, task, and technology. Analyses were conducted using Dedoose, a qualitative data analysis software.
Figure 3.
The FITT framework: IT adoption depends on the FITT. FITT: fit between individuals, task and technology.
RESULTS
We identified 9 themes regarding participants’ interactions with and attitudes toward NMPRO. Since physicians, psychosocial providers, and MAs interacted with the system differently, we examined their perspectives separately (Table 2).
Table 2.
Summarized findings of user feedback on NMPRO based on the FITT framework
| Role | Task(s) | Technology | Individual |
|---|---|---|---|
| Medical Assistants |
|
|
|
| Physicians |
|
|
|
| Psychosocial providers (psychologists & social workers) |
|
|
|
EHR: electronic health record; FITT: fit between individuals, task and technology; NMPRO: Northwestern Medicine Patient Reported Outcomes; PRO: patient-reported outcome.
Task and workflow
NMPRO implementation has generated new tasks in the clinical workflow. The following themes illustrate these tasks—PRO collection, PRO review, and care delivery—and the associated challenges of incorporating PROs into clinical practice.
PRO collection
When patients did not complete an assessment in advance, a new task was added to MAs’ workflow: PRO administration in the exam room. While MAs in RHLCCC did not view this as a heavy burden, they experienced several challenges. The first challenge concerned patient participation. Although most patients were receptive to completing the assessment when asked, it was harder to engage patients who viewed it as a waste of time or who had lower levels of computer literacy. Some patients had physical function limitations that interfered with using the desktop computer or were too sick to complete the assessment. MAs usually had to spend more time with these patients, explaining the purpose and assisting with completing the assessment. This was considered “time-consuming” (P9) and sometimes hindered MAs’ ability to room the next patient on busy days.
Further, PRO collection in the exam room did not integrate seamlessly with the existing clinical workflow. MAs noted that whether or not patients could complete PROs completely depended on the flow of the clinic. As 1 MA explained, “On the days where things are very fast-paced, by the time I maybe barely finish vitals, I will already have the doctor knocking on the door. So, there’s really no time [for PROs]” (P7). Once interrupted by the arrival of the physician, patients were unlikely to complete the assessment. MAs suggested that computers and tablets should be provided in the waiting room so that patients could start doing PROs before they entered the exam room.
PRO review
Providers were expected to review PRO results to identify patient issues, but many physicians did not do so routinely, considering PRO review to be time-consuming, disruptive, and a distraction from discussion with the patient. As an oncologist said, “When you’ve got 20 min with the patient before the next one is in the room, it needs to flow, not be interrupted” (P12). Moreover, oncologists regarded PROs as redundant because they “already asked patients things like fatigue and distress” (P12) in routine visits. Given that PROs were administered presurgery and postsurgery in orthopedics, surgeons reviewed PROs occasionally during the visit if a patient was unsatisfied with his or her improvement. In this specific situation, PROs were used to compare the patient’s preoperative- and postoperative-health status to demonstrate progress.
In contrast, psychologists and social workers in RHLCCC reviewed PROs as part of their routine care. Psychologists often reviewed PROs before they met with patients, which informed “target areas to evaluate” (P2) and helped guide their upcoming patient discussion. Similarly, social workers reviewed PRO scores to discern the severity of a patient’s issues and what resources the patient might need before outreach. However, PROs per se were not sufficient to inform clinical decisions. Social workers considered the threshold for depression (T-score above 60) too low and often used their “clinical judgment” (P8) as a supplement. “If it’s under 70, I don’t think of it as acute and I don’t know if that’s accurate. But if it’s 70 or above, I think that is very acute,” P8 said. The perceived severity influenced whether or not she would make an effort to reach out to the patient.
Care delivery
NMPRO was used differently by physicians and social workers in their care delivery process. Specifically, physicians seldom acted upon PROs due to a lack of actionable data: “PROs have only been validated against legacy tools, but is there a way that we can develop patient trajectories for common conditions?”(P6). The patient trajectories could help surgeons to “identify patients who don’t follow normative recovery courses or who present later on with new problems” (P3). But depicting such trajectories entailed an enormous amount of patient data and these data were not available yet. Moreover, a surgeon (P3) contended that “PROMIS does not have a lengthy heritage in joint replacement surgery. So we are in the process of doing studies to relate and validate PROMIS within this population.”
In contrast, social workers actively leveraged NMPRO to deliver mental health resources. They were required by clinic guidelines to respond to patients with severe emotional distress or social work needs within 72 h, either via MyChart messages or phone calls. Social workers noted that their workload increased vastly after the introduction of NMPRO. “It’s challenging because there’s more people to follow-up with and not enough staff to do so.” (P5). Given the large volume of patients in need, social workers often prioritized outreach to patients who reported a higher level of distress.
Technology: usability and functionality
The technology’s material features are another integral part of the FITT framework. In this section, we examine to what extent the usability and functionality of NMPRO facilitate or hinder the work processes of its users.
Technology roadblocks to PRO collection
Although MAs noted that NMPRO was easy to use, they highlighted some roadblocks in PRO collection. First, the PRO assessment was only available in English, which excluded patients who were non-English speakers. Second, desktop computers were immobile so that patients with mobility issues such as wheelchair users had a hard time filling out PROs. Third, the EHR was locked when patients were taking the assessment to protect data confidentially and security, but the screen could only be unlocked by MAs. Therefore, “If the doctor chooses to use that computer, we have to go in to log ourselves out of that computer in order for them to get in” (P10). This created extra work that disrupted both the MA and physician workflows.
Data visualization for PRO review
Physicians desired easier access to PRO results and more visualization options. For example, they expected to pull up the PRO graph using “fewer clicks” (P6). When viewing the results, they wanted to have a variety of data display options that allow providers to “select the representation of data and the interpretation of results in the manner that best works for them” (P3). Physicians also wanted the functionality to automatically insert PRO results into the clinical note, in conjunction with a description of the meaning of the score. This would help them to identify patient issues “within a few seconds” (P12) and save time for the clinic visit.
Automated triage for care delivery
Social workers mentioned that their care delivery process was complicated by a lack of patient tagging in the system. In RHLCCC, social workers were specialized by cancer type and worked with specific providers’ patients. However, notifications for all patient referrals were sent to an EHR pool of all social workers, rather than routed to a specific person. Consequently, social workers had to go through the entire pool to “look for patients I am supposed to be responding to based on who their oncologist is” (P5). Social workers suggested “an automatic separation” (P8) or adding a label indicating the patient’s attending oncologist to expedite this process. Additionally, some referrals were duplicated in the EHR pool because each measure (eg, depression, anxiety, social work need) generated its own notification: “So, if I had nine referrals, they could potentially be three patients” (P5). This made it difficult to effectively plan the needed outreach activities.
Individual: motivations and barriers to use
The FITT framework posits that the new system and tasks should fit user needs and goals. Therefore, in this section, we draw attention to individual motivations of and barriers to using NMPRO.
MAs: well-trained but lacking perceived benefit
MAs had sufficient training to administer PROs but did not fully understand the benefits of routine PRO assessment. Prior to rollout, MAs had received several training sessions regarding the purpose of PRO assessment and associated workflows. They found these introductory presentations very helpful in guiding them to prompt patients and navigate the system. However, most MAs had little knowledge of the content of the assessment and thus had a hard time explaining how patients would reap direct benefits from completing it. “We [administer PROs] because we are required to do so,” P9 noted.
Physicians: interested but unmotivated to use
Physicians generally viewed NMPRO as valuable. They regarded PROs as a “conversation starter” (P5) that could drive patient-provider communication, which “improved the efficiency of the visit” (P6). PROs also allowed physicians to “track any symptom over time from visit to visit” (P2), which was conducive to “quality improvement evaluations” (P6). Additionally, physicians commented that PROs in the future could eliminate unnecessary office visits: “Maybe we don’t need a 2-year follow-up visit. Patients can do the rest of their follow-ups remotely if they’re doing fine, and we’re just following their x-rays…more like tele-medicine,” a surgeon (P6) envisioned.
However, physicians identified 4 barriers to using PROs. First, they were concerned about task overload and burden because “there are so many aspects of care and adding in another element is a lot to deal with” (P4). Second, physicians found PROs to be “pretty early and sporadic” (P12), without enough patient data to depict “normative recovery courses” (P3) that could guide clinical action. Third, physicians were skeptical about the validity of PRO scores due to “little research around it [PRO] to help frame decision making or thoughts around it” (P3). Fourth, there was a lack of financial incentives to use PROs. An oncologist P4 explained: “We have a ton of work we’re doing that we don’t bill for. I feel like unless the payment models change to reflect this work, we’re not going to do it.”
Psychosocial providers: enthusiastic but concerned about data accuracy
Psychologists and social workers benefited greatly from NMPRO and viewed it as integral to patient care. Using PROs enabled psychologists to “know patient concerns upfront” (P2) and have “more targeted conversations with patients” (P2). Social workers appreciated PROs for identifying patients in need and enabling just-in-time service delivery. Receiving PRO notifications allowed them to “reach out speedily to patients in need” (P5).
Yet social workers voiced concerns about the accuracy of PROs. They noted that distress measures reflected patients’ feelings at a certain point in time which may have changed by the time social workers reached out to them. Social workers also mentioned that patients might misreport their issues due to “misunderstanding, mental health stigma, and cultural reasons” (P5), or “the way they were primed” (P9). Furthermore, social workers found it challenging to gauge the severity of patients’ social work needs, which were identified in the assessment through a checklist rather than on a scale. “I think it [NMPRO] is overwhelming, because the assessment can’t tell you who is in urgent need” (P5). If a patient identified financial concerns, for example, social workers hardly knew how severe the concern was and thus could not prioritize outreach accordingly.
DISCUSSION
This study demonstrates that EHR-integrated PROs can facilitate patient-provider communication and improve care delivery.2,7,15 These findings also show that physicians, psychosocial providers, and MAs face different challenges in their use of the PRO system. Barriers mainly stemmed from a lack of actionable data, current workflow configurations, and technical issues. Based on our findings, we propose solutions to facilitate routine use of PROs (Table 3). These suggestions, mapped onto the FITT framework, show the importance of considering both technical and organizational solutions to guide future design.
Table 3.
Summary of NMPRO challenges and recommended solutions
| Aspect | Challenges | Recommendations |
|---|---|---|
| Task/workflow | Patient resistance to PRO completion | Provide patients with educational materials regarding the purpose, use, and security of PROs via handouts, videos, and online resources |
| Limited time for patients to complete assessment in exam room | Increase opportunities for PRO collection in various settings, such as patient’s home, waiting room, and exam room | |
| Reviewing PROs disrupts clinical encounters | Provide simple and actionable PRO results with vivid data visualization to save time for providers | |
| Technology | Lack of various types of data visualization | Incorporate tools to represent and view data in multiple ways |
| PRO results are not available for direct integration into clinical notes | Pull PRO results into the Note using existing EHR tools | |
| Desktop computers are inconvenient for certain patient populations (eg, wheelchair users) | Provide tablets or other mobile devices for PRO collection in the clinic | |
| PRO assessments are only available in English | Offer PROs in other languages and allow patient to select desired language | |
| Patient referrals are not routed to specific social workers | Provide automatic separation/labeling of patients based on their clinic and primary physician | |
| Individual | Not fully aware of the value of PROs | PROs should include relevant and validated measures based on the needs of a clinic; explicate the clinical utility of PROs to providers through training |
| Unsure how to make use of the data | Need more research and guidance on recommended clinical action with PRO scores; provide actionable data presentation | |
| Lack of incentives | Incentivize and support routine use of PROs through improved payment models (eg, incentivize providers based on the number of clicks on PRO results) | |
| Lack of training and education | Conduct training tailored to specific providers through emails, clinic meetings, and one-on-one meetings; engage physician champions |
EHR: electronic health record; NMPRO: Northwestern Medicine Patient Reported Outcomes; PRO: patient-reported outcome.
Our findings suggest that the “individual” component of the FITT framework may be the main determining factor of routine PRO use.20 We found that providers’ perceived value of PROs largely facilitated or hindered their PRO use, beyond the effects of workflow change and technology usability. Consistent with previous studies, we found that physicians were generally skeptical of PRO utility,1,19 largely due to a lack of guidelines for clinical action and the absence of normative data for PRO interpretation. Conversely, PROs have long been used to screen for emotional distress and to evaluate outcomes of clinical interventions. Thus, psychologists and social workers had a clearer understanding of what clinical action was appropriate for specific scores and therefore found more value in NMPRO. In this sense, PRO assessments should include measures that are identified by providers as clinically relevant and meaningful to improve their delivery of care.1,14 When patient-generated health data26 can aid in guiding clinical action, providers will be more likely to use them.
We also suggest that PRO systems should be oriented to the distinct needs of different provider groups in varied clinical settings. When the needs of multiple stakeholders cannot be satisfied concurrently, it would be helpful to prioritize those who interact most frequently with the system. Given that the primary goal of NMPRO implementation in the Cancer Center was screening and triaging severe emotional distress,27 it was unsurprising that psychosocial providers interacted most frequently with NMPRO. However, the orthopedics clinic focused on presurgery and postsurgery assessments to highlight the patient function over time, resulting in most surgeon interaction with PROs. In this case, a clinic might be better served by focusing more on the needs of psychosocial providers than on oncologists who found less benefit in PROs. Thus, future design and implementation of PRO systems may benefit from identifying the primary user group(s) and making iterative improvements driven by feedback from end-users.19
Additionally, changes in organizational culture and payment model are needed for routine PRO use to be embraced. Despite physicians’ skepticism, we observed that people involved in the PRO planning team were more open to using and talking to others about PROs. Therefore, engaging physician champions and having training sessions can help providers better understand the value of PROs and integrate PROs into their workflow.19 Incentives also matter. In our case, oncologists were not incentivized to review PROs, whereas surgeons could receive additional payment for including PRO assessment, according to the Comprehensive Care for Joint Replacement (CJR) model.28 The CJR model holds hospitals and providers financially accountable for the quality and coordination of care.28 In this sense, payment models should adequately incentivize providers to use PROs for patient care.
However, this study has several limitations. First, this research was conducted within a single health-care institution, thus limiting the generalizability of our findings. As EHR-integrated PRO systems have been increasingly implemented within health institutions, similar studies should be conducted to examine if our findings can apply to other organizational settings. Second, our sample size was relatively small, although thematic saturation was reached.25 Future research could expand on this work with a large-scale survey to understand provider perspectives on patient-generated health data across provider types and clinical specialty. Third, we conducted this study at an early stage of NMPRO implementation. Longitudinal studies are warranted to examine how providers’ perceptions might change over time. Fourth, we did not interview patients due to our focus on providers’ feedback. Future research should draw attention to patient needs and experiences using PRO tools.
CONCLUSION
New system implementation in clinical settings is often accompanied by a change in clinical workflow and organizational culture. In this study, we evaluated the adoption of an EHR-integrated PRO system in 2 major clinical areas of a tertiary care hospital. Drawing on the FITT framework, we described the opportunities and challenges facing providers in using the system in terms of clinical workflow, individual needs, and technical features. We also provided recommendations to inform better integration of patient-generated health data into routine care.
FUNDING
This research was supported by the National Center for Advancing Translational Sciences (NCATS) 1U01TR001806.
CONTRIBUTORS
RZ was responsible for the study design, data collection, data analysis, and initial drafting of the manuscript. ERB was responsible for the data analysis and initial drafting of the manuscript. All authors contributed to the interpretation of data, the revision, and final approval of the manuscript.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.
Supplementary Material
ACKNOWLEDGMENTS
We thank the individuals who participated in this study. We also thank the reviewers, the editor Dr Indra Neil Sarkar, and the NMPRO Steering Committee for their valuable feedback.
Conflict of interest statement. None declared.
REFERENCES
- 1. Harle CA, Listhaus A, Covarrubias CM, et al. Overcoming barriers to implementing patient-reported outcomes in an electronic health record: a case report. J Am Med Inform Assoc 2016; 231: 74–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Black N. Patient reported outcome measures could help transform healthcare. BMJ 2013; 346: f167. [DOI] [PubMed] [Google Scholar]
- 3. Fromme EK, Eilers KM, Mori M, Hsieh Y-C, Beer TM.. How accurate is clinician reporting of chemotherapy adverse effects? A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire C30. J Clin Oncol 2004; 2217: 3485–90. [DOI] [PubMed] [Google Scholar]
- 4. Basch E, Jia X, Heller G, et al. Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes. J Natl Cancer Inst 2009; 10123: 1624–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016; 346: 557–65.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Chen J, Ou L, Hollis SJ.. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 2013; 131: 211.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Chung AE, Basch EM.. Incorporating the patient’s voice into electronic health records through patient-reported outcomes as the “review of systems”. J Am Med Inform Assoc 2015; 224: 914–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol 2015; 269: 1846–58. [DOI] [PubMed] [Google Scholar]
- 9. Basch E, Snyder C, McNiff K, et al. Patient-reported outcome performance measures in oncology. J Oncol Pract 2014; 103: 209–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Cella DF, Hahn EA, Jensen SE, et al. Patient-Reported Outcomes in Performance Measurement. Triangle Park, NC: RTI Press Research; 2015. [PubMed] [Google Scholar]
- 11. Basch E, Torda P, Adams K.. Standards for patient-reported outcome–based performance measures. JAMA 2013; 3102: 139–40. [DOI] [PubMed] [Google Scholar]
- 12. Wagner LI, Schink J, Bass M, et al. Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer 2015; 1216: 927–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Estabrooks PA, Boyle M, Emmons KM, et al. Harmonized patient-reported data elements in the electronic health record: supporting meaningful use by primary care action on health behaviors and key psychosocial factors. J Am Med Inform Assoc 2012; 194: 575–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Nelson EC, Eftimovska E, Lind C, Hager A, Wasson JH, Lindblad S.. Patient reported outcome measures in practice. BMJ 2015; 350: g7818.. [DOI] [PubMed] [Google Scholar]
- 15. Rotenstein LS, Agarwal A, O’neil K, et al. Implementing patient-reported outcome surveys as part of routine care: lessons from an academic radiation oncology department. J Am Med Inform Assoc 2017; 245: 964–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Sanger PC, Hartzler A, Lordon RJ, et al. A patient-centered system in a provider-centered world: challenges of incorporating post-discharge wound data into practice. J Am Med Inform Assoc 2016; 233: 514–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Hassol A, Walker JM, Kidder D, et al. Patient experiences and attitudes about access to a patient electronic health care record and linked web messaging. J Am Med Inform Assoc 2004; 116: 505–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Urowitz S, Wiljer D, Dupak K, et al. Improving diabetes management with a patient portal: qualitative study of a diabetes self-management portal. J Med Internet Res 2012; 146: e158. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Locklear T, Miriovsky B, Willig J, et al. Strategies for overcoming barriers to the implementation of patient-reported outcomes measures. An NIH health care systems research collaboratory patient reported outcomes core white paper; 2014. https://www.nihcollaboratory.org/Products/Strategies-for-Overcoming-Barriers-to-PROs.pdf (accessed September 2018)
- 20. Ammenwerth E, Iller C, Mahler C.. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak 2006; 61: 3.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Wu AW, Kharrazi H, Boulware LE, Snyder CF.. Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol 2013; 668: S12–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care 2007; 45 (5): S22–31. [DOI] [PubMed] [Google Scholar]
- 23. Pearman T, Garcia S, Penedo F, Yanez B, Wagner L, Cella D.. Implementation of distress screening in an oncology setting. J Community Supportive Oncol 2015; 1312: 423–8. [DOI] [PubMed] [Google Scholar]
- 24. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC.. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 226: 1179–82. [DOI] [PubMed] [Google Scholar]
- 25. Strauss A, Corbin JM.. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Thousand Oaks, CA, US: Sage Publications, Inc; 1990. [Google Scholar]
- 26. Shapiro M, Johnston D, Wald J, Mon D.. Patient-Generated Health Data. RTI International; 2012. https://www.rti.org/sites/default/files/resources/patientgeneratedhealthdata.pdf (accessed September 2018) [Google Scholar]
- 27. American College of Surgeons Commission on Cancer. Cancer Program Standards 2012: Ensuring Patient-Centered Care. American College of Surgeons; 2011. https://www.facs.org/~/media/files/quality%20programs/cancer/coc/programstandards2012.ashx (accessed September 2018) [Google Scholar]
- 28.Centers for Medicare & Medicaid Services. Comprehensive Care for Joint Replacement Model; 2018. https://innovation.cms.gov/initiatives/cjr (accessed September 2018)
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.