Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Mar 21.
Published in final edited form as: Med Care. 2015 Feb;53(2):153–159. doi: 10.1097/MLR.0000000000000289

The Role of Technical Advances in the Adoption and Integration of Patient-Reported Outcomes in Clinical Care

Roxanne E Jensen 1, Nan E Rothrock 2, Esi Morgan DeWitt 3, Brennan Spiegel 4,5,6, Carole A Tucker 7, Heidi M Crane 8, Christopher B Forrest 9,10, Donald L Patrick 11,12, Rob Fredericksen 8, Lisa M Shulman 13, David Cella 2, Paul K Crane 14
PMCID: PMC4801509  NIHMSID: NIHMS695612  PMID: 25588135

Abstract

Background

Patient-reported outcomes (PROs) are gaining recognition as key measures for improving the quality of patient care in clinical care settings. Three factors have made the implementation of PROs in clinical care more feasible: increased use of modern measurement methods in PRO design and validation, rapid progression of technology (e.g., touch screen tablets, Internet accessibility, and electronic health records (EHRs)), and greater demand for measurement and monitoring of PROs by regulators, payers, accreditors, and professional organizations. As electronic PRO collection and reporting capabilities have improved, the challenges of collecting PRO data have changed.

Objectives

To update information on PRO adoption considerations in clinical care, highlighting electronic and technical advances with respect to measure selection, clinical workflow, data infrastructure, and outcomes reporting.

Methods

Five practical case studies across diverse healthcare settings and patient populations are used to explore how implementation barriers were addressed to promote the successful integration of PRO collection into the clinical workflow. The case studies address selecting and reporting of relevant content, workflow integration, pre-visit screening, effective evaluation, and EHR integration.

Conclusions

These case studies exemplify elements of well-designed electronic systems, including response automation, tailoring of item selection and reporting algorithms, flexibility of collection location, and integration with patient health care data elements. They also highlight emerging logistical barriers in this area, such as the need for specialized technological and methodological expertise, and design limitations of current electronic data capture systems.

The Case for Patient-Reported Outcomes

Patient-Reported Outcomes (PROs) are assessments of a patient’s health and disability experiences in a structured and standardized format directly from the patient.1 PRO adoption and implementation in clinical care are rapidly increasing as a result of several key developments. New PROs have been developed, and older scales have been evaluated with modern measurement models. Technological infrastructure has progressed rapidly, leading to the expanded incorporation of touch screen tablets, Internet-based applications, and electronic health records (EHRs) into clinical care. PROs are increasingly demanded by regulators, payers, accreditors, professional organizations, and clinicians to measure and improve PRO-based outcomes at the patient, clinic, and healthcare system levels.

Legislative demands to improve health care outcomes and contain costs are changing the United States health care system to one that puts greater emphasis on quality of care, value-based reimbursement, and patient engagement.2 PROs are increasingly identified as the most direct and relevant measures of success for demonstrating high-quality patient-centered care. The integration of PRO scan support patient care and quality improvement by enabling the identification and engagement of patient services.

PROs are gaining acceptance across a variety of different clinical settings.3–6 Evidence suggests they aid in the management of chronic conditions,3, 7 improve patient-provider communication,8–11 and increase patient satisfaction with care.11 While PROs are most commonly used to inform and track patient-level information,12 electronic data capture and EHR integration capabilities have expanded PRO use in novel ways, such as population-based monitoring. (Table 1)

Table 1.

Taxonomy of Patient Reported Outcome (PRO) Use in Clinical Care and Benefits of Electronic Integration

Use Description Example (Setting) Electronic Integration
1 Needs Assessment Identifies the necessity for a formal patient evaluation or therapeutic intervention Depression screening (primary care) High score triggers a more in depth depression symptom evaluation or a behavioral health referral
2 Shared Decision-Making Presents PRO scores to patients and clinicians before selecting a new treatment Information about current pain, physical function provided to patients before scheduling knee replacement surgery (acute care) Allows real-time modeling of predicted improvement over time based on a patient’s baseline clinical and demographic characteristics
3 Symptom Management Tracks intended and unintended treatment effects and identifies symptom-management opportunities Monitoring chemotherapy treatment for unexpected adverse effects (cancer treatment) Scores are monitored in real time, at any internet-accessible location
4 Outcome Assessment Tracks recovery after an intervention and evaluates treatment effectiveness Regular measurement of fatigue, sexual function, and depression after starting anti-hypertensive treatment. (specialist care) Provides a number of customizable real-time reporting options to evaluate patient recovery
5 Quality Improvement Provides population-level scores to evaluate clinical practice Measurement symptom prevalence (health system) Necessary to capture symptom-linked clinical care actions
Open in a new tab

Successful integration of PRO collection and review into the clinic workflow has always been a key aspect of PRO implementation. Recent advances in PRO instrument development and technology have increased the feasibility and practicality of sustained use. Most studies discussing PRO implementation in clinical settings focus on patient and provider attitudinal barriers13, 14 and general implementation guidance.15 Issues specific to electronic system integration have been identified, but these discussions have not been updated to reflect current PRO technology and data infrastructure,16 or have focused on PRO integration within a single institution.17

In this article we provide an update on PRO integration in clinical care, focusing on recent technological and organizational advances with respect to measure selection, clinical workflow, electronic infrastructure, and outcomes reporting (Figure 1). We highlight each element with examples from ongoing implementation efforts across different settings and patient populations.

Figure 1.

Figure 1

Open in a new tab

Key Electronic Patient-Reported Outcome Workflow and Infrastructure Integration Considerations

Why Now? Rapid Increase in Policy Relevance and Value of PROs in Clinical Care

Recent changes in the field have facilitated the increasing use of PROs in clinical care settings. Here we address several of these, including content tailoring, infrastructure, and increased demand for PROs from multiple stakeholders.

Increased Demand by Regulators, Payers, Accreditors, and Professional Organizations

There is an increasing emphasis on integrating PRO data into clinical care as a means of promoting patient-centered care. One catalyst has been the creation of the Patient-Centered Outcomes Research Institute (PCORI).18 PCORI focuses on the inclusion of the patient voice when setting funding priorities. This has increased awareness of the patient’s perspective in health care and of the importance of PROs in clinical care and quality improvement. The Centers for Medicare and Medicaid Services (CMS) incentive program includes mandates for EHR reimbursements. Meaningful Use (Stage 3) implementation is scheduled for 2016 with a focus on improving patient outcomes. Consequently, multiple organizations (including CMS, the National Quality Forum,19, 20 and the National Committee for Quality Assurance) are engaging in efforts to identify and recommend patient-centered outcome measures. These organizations laud the ability of PROs to provide “rigorous, disciplined measurement” of patient outcomes,21 to capture more complete pictures of patient experiences with multiple chronic conditions,2 to promote patient-centered communication,22 and to provide information on outcomes that are meaningful to patients.23

Infrastructure: Rapid Progression of Computer Hardware, Internet Accessibility, and EHRs

Ease of administration has increased with the availability of electronic PRO data collection software and Web-based data entry options, allowing for immediate scoring that can be displayed for review during clinical encounters. Several studies demonstrate that electronic collection is preferred in clinical care4, 5, 7–11, 13, 24 and is associated with lower rates of unanswered questions than paper forms.4, 5, 11, 25, 26 Reproducibility of electronic data collection is high,3, 5, 7 lowering missing data and allowing complex skip patterns.4, 5, 11, 25–27 Modern systems can also provide a consistent look for all content administered to patients.

In the past decade, the barriers to widespread electronic PRO assessments have diminished, making clinical applications more affordable and feasible to implement. Computer hardware has become both increasingly affordable and capable. At the same time, Internet connectivity is rising in both private and public locations, with 80% of American households indicating at least one regular Internet user.28 This rise in connectivity has increased the range of locations where patients can complete assessments (e.g., at home, waiting room kiosks, or smartphone).

Electronic PRO assessment can extend the spectrum of patients from whom PRO data can be collected by adapting collection methods to patient needs, such as by providing an audio option for individuals with low vision, limited computer experience, or low literacy.29 Touch screen entry eliminates typing and avoids mouse-based actions,27 which may be particularly important among older patients.30

Tailoring of PROs for Clinical Relevance

PRO measures can be constructed and evaluated using classical test theory or modern psychometric methods such as item response theory (IRT).31 Examples of PRO measures in clinical care settings include the Brief Pain Inventory32 and the Patient Health Questionnaire (PHQ-9),33–36 both validated and scored using classical test theory approaches. These measures provide scores for pain and depression. However, they entail two important constraints: (1) the inability to tailor content and (2) limitations on score interpretability. These measures ask the same items of each respondent, even if it does not provide relevant information. In clinical settings, providers need to choose between lengthy PRO measures that cover all possible health levels and brief measures that may apply to broad patient populations but may not be sensitive enough to identify small changes in clinical care settings.37 A new wave of PRO measures such as the Patient-Reported Outcomes Measurement Information System (PROMIS®) provide a framework for patient-level tailoring of content by using both computer adaptive tests (CAT) and static short forms that contain the most informative items, while using scores that are standardized against a general population. This affords greater measurement precision, eliminating common floor and ceiling effects for CAT administration,38 and increased accuracy for static forms,39 while allowing for score interpretation across large and varied patient populations.

Overcoming PRO Workflow Challenges

Adapting assessment systems to fit within clinic workflows remains a key barrier to clinical adoption. Patients need time to complete PRO assessments in-clinic, clinicians need to easily retrieve and review results, and results must be understandable and actionable. Furthermore, in-clinic connectivity comes with its own set of “hidden” workflow challenges such as a poor wireless signal which may slow or prevent PRO data submission. Below, we provide five case studies highlighting specific elements of electronic integration (Figure 1) and how they are leveraged to improve PRO integration into the clinic workflow.

Content Selection and Tailoring: Considering Clinic Workflow and Patient Burden

Systematic PRO collection and delivery to healthcare providers will not be fully embraced if it is perceived to disrupt or impede clinical workflow. Although electronic PRO measures can be administered rapidly, recommendations suggest limiting the number of PROs to reduce patient burden.40 Therefore, measure selection and tailoring to individual patients to ensure brevity and clinical relevance are an important first step.

The Cincinnati Children’s Hospital Medical Center has focused on system automation to minimize patient burden and reduce staff involvement. At check-in, patients receive an electronic tablet and a personal numeric code (linked to the specific visit in the electronic medical record) to enter. An algorithm selects PROs tailored to an individual patient according to pre-specified variables (e.g., visit type, diagnosis, age). PROs administered include the PROMIS Pediatric Pain Interference scale, Childhood Health Assessment Questionnaire (physical function), and Pediatric Quality of Life Inventory Generic and Rheumatology modules (v4.0 and 3.0, respectively).

Content selection and tailoring ensure that an appropriately minimal number of relevant PROs are administered at staggered intervals. Given the pediatric focus of this institution, patient age is a key consideration in tailoring PRO content and administration. In this system, patients receive age-appropriate PROs, and when tracked over time automatically “age-in” to new PRO content. Proxy (parent) use is automatically set up for young children, with other proxy options available for other patients if necessary because of developmental, cognitive, or other concerns.

The impact of tailored PRO selection is carefully considered relative to the clinic workflow. Quality improvement methods (e.g., process flow maps) have been deployed specifically to avoid lengthening patient visits. However, in clinical settings where a lengthy instrument is required (e.g., psychological batteries), the visit scheduling accounts for this, providing a delay to allow for the patient to complete the assessment. Patients are informed prior to their visit to arrive early and the reason why extra time is necessary.

Reporting Clinically-Relevant PRO Content: The My GI Health Project

Electronic PRO collection and reporting allows for detailed visual reports that facilitate interpretation of a patient’s PRO score. Little attention has been paid to how PRO information is best reported,15 and systems do not uniformly report clinically relevant information such as reference values and identification of meaningful change.41

The My GI Health Project (myGIhealth.org) is a website-based PRO system developed by the University of California, Los Angeles and the University of Michigan. My GI Health evaluates the severity of patient-reported gastrointestinal symptoms (PROMIS GI Symptoms) for clinical use. Reports present information on symptom severity in eight areas: reflux, gas/bloating, abdominal pain, diarrhea, incontinence, constipation, bloating, and nausea/vomiting (Figure 2). Prior to implementation, clinicians evaluated several report designs, indicating preferences for presenting symptom information against the U.S. population (rather than against the clinic’s patient population), and the use of percentile scores (instead of the PROMIS t-score). Providers also indicated a preference for grouping symptom intensity into quartiles.

Figure 2.

Figure 2

Open in a new tab

Patient-Reported Outcome Report from My GI Health

My GI Health illustrates how electronic reports can be implemented into the clinical workflow using alerts and a report format that allow clinicians to manually set their own point of clinical relevance. This is a departure from other methods for identifying this information, such as identifying thresholds numerically.19 Moving forward, both methods should be evaluated and compared side-by-side with respect to clinical relevance and effectiveness.

Data Collection: Routine Symptom and Psychosocial Screening in Cancer Care

Many electronic PRO collection programs incorporate centralized Web-based data submission and storage. Web-based PRO administration can remove PRO collection from the workflow for those patients who complete Web-based assessments. This functionality presents the following challenges:(1) reaching and engaging patients without at-home Internet access; (2) determining the appropriate response for PRO scores that indicate problems warranting immediate clinical attention; and (3) implementing an automated response across clinic systems. This case study focuses on these challenges.

In the Robert H. Lurie Comprehensive Cancer Center (RHLCCC) of Northwestern University, all patients receiving chemotherapy for gynecological cancers complete PRO CAT assessments (PROMIS Domains: Pain Interference, Fatigue, Physical Function, Depression, Anxiety) and a psychosocial needs assessment on a secure website up to 3 days prior to a clinic visit. Patients who are unable to complete the assessment online prior to their visit are asked when they check in to complete a survey using a tablet computer.

PROs are automatically scored and saved in the patient’s EHR. Scores that exceed a pre-determined and validated threshold for severity33, 34, 42 are flagged within the EHR and generate an automated message to appropriate staff (e.g., oncologist, dietician pool, or social worker pool) through the EHR messaging system. These automated referrals generate messaging in response to a patient-identified need (e.g., assistance with advance directives; personal finances).

Since implementation, approximately half of patients notified have completed the assessment prior to their visit.43This suggests that while pre-visit assessments may be feasible for some patients, supplemental in-clinic data capture is necessary to capture data from the full patient population. RHLCCC has considered how to identify and administer PROs to patient non-responders by developing and testing in-clinic interfaces that can address specific patient needs (e.g., visual impairment, low literacy, lack of fluency in English).

Demonstrating Value: Effective Evaluation of PRO Use

Because clinical use of PRO data is still limited, examples evaluating effectiveness and impact are not common, and vary in methodological rigor.3, 44 Internally, the use of evaluation and quality improvement techniques can be used to illustrate how PRO implementation adds value in a clinical care encounter. Demonstrating value to broader audiences requires additional outcomes, including the user experience (patient and provider attitudes, satisfaction with the system, doctor-patient communication), health services outcomes (clinical actions taken, referrals), and patient outcomes. This case study examines how value can be demonstrated while minimizing effects on the limitations of the clinical workflow.

The University of Washington (UW) has a large, multi-disciplinary HIV outpatient clinic that has successfully integrated PROs into routine clinical care.45 Ten PRO measures are administered, including the PHQ-9 (depression), the PHQ-5 (anxiety), the Alcohol Use Disorders Identification Test (AUDIT, alcohol use), and the HIV Symptoms Index, illustrating the broad range of general and disease-specific information collected. Internal evaluation has used Plan-Do-Study-Act cycles and a quality improvement framework to identify and address issues related to PRO implementation and clinic flow, electronic technology, scheduling, and delivery of assessment results.46 Qualitative interviews with patients, providers, and clinic staff were used to identify what content and features were most valuable (e.g., automated suicidal ideation alert).47

The approach has been adopted by other clinics in the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS), such that more than 30,000 PRO assessments of over 7000 HIV-infected patients have been completed as part of routine clinical care visits, providing data for clinical care, quality improvement, and clinical research efforts.

Integrating PROs in EHRs

Incorporating electronic PRO data directly into the EHR allows for PRO review at the point of care and tracking over time a long side other clinical information. Because of logistical issues, PROs have often been made available independently from the EHR system.9 While clinicians have reported their preference for PRO integration with EHR lab results,48 only a few systems provide this feature, meaning that PROs are often not presented in a useful way to clinicians.

Epic Systems Corporation (Verona, WI), a commonly used EHR system, included PROMIS measures in its 2012 software release. Epic offers a library of PRO measures; users can also add their own measures. Institutions using Epic can program it to trigger the administration of PROs at predefined intervals or in response to certain “events,” such as an office visit, surgery, hospitalization, acute illness, or change in health status. While clinician entered data has been supported for years, PRO administration through Epic is intended to occur outside of the clinical encounter through a patient portal (MyChart application) or in the waiting room on a tablet, kiosk, or personal computer (Welcome application). These data are scored and stored in the EHR database, and are reported in tabular or graphical form like lab data. This allows clinicians to contrast PRO scores with other clinical variables..

Discussion and Future Directions

Overall, these case studies illustrate that workflow considerations are fundamental to the successful integration of PRO assessments into clinical care settings. The collection and reporting of PRO data must be done with minimum burden and maximum clinical relevance in order to have a meaningful impact on patient care. The systems described in these case studies exemplify elements of well-designed systems: response automation, the tailoring of item selection and reporting algorithms, flexibility of collection location, integration with patient data elements, and assessments that take less than 10 minutes to complete. These features can facilitate the integration and adoption of PROs in a wide range of clinical care settings. These case studies also demonstrate the feasibility of PRO data collection in a wide range of patient populations (e.g., children, elderly, veterans) who may have significant symptom burden (e.g., cancer, HIV/AIDS). Most systems restrict PRO collection to patient self-report collected outside of the clinic room. However, one system (My GI Health) allows a doctors to administer the “history-taker” (a PRO assessment) to their patient. Doctor-led PRO assessments provide an interesting alternative design to patient self-assessment. However, clinician guidelines and training are necessary to ensure patients understand the questions and are constantly providing independent responses.

These case studies also illustrate the importance of EHR integration and current design limitations of electronic PRO systems. For example, incorporating outside PRO data elements into EHRs (rather than collecting PROs within the EHR, as illustrated by Epic) can be complicated, requiring solutions ranging from allowing providers to cut and paste PRO information into the record (My GI Health) and a variety of customized options that have been developed to address this issue across the many different clinics involved with CNICS (UW). CAT algorithms are highly relevant for use in clinical settings, but are infrequently used and currently unavailable in EHR-based collection. IRT-derived fixed measures show excellent reliability and validity25, 49 and also allow customization, however research has not yet shown if the measurement sensitivity gained by IRT-calibration translates into a clinical relevance for fixed formats.

The ability to report PRO scores electronically along with clinically relevant interpretations is important in each case study. Each case study approached PRO reporting differently, although all solicited provider feedback and preferences. In general, PRO scores must be presented in a way that is easily understood, clear, and actionable. Because many symptoms are common to multiple clinical conditions, such as pain, fatigue, and sleep difficulty, a common framework for scoring and interpretation can facilitate comparisons across diseases, patient subgroups, and the general population. Naturally, there remain more targeted, condition-specific areas that continue to require supplemental measurement attention where comparisons to the general population are not necessarily useful in clinical settings. An important limitation identified by the case studies was that EHR score reporting formats are limited to clinician review only. Systems here which have developed score reports for patients (My GI-Health) generate them independently from the EHR system.

The major limitations of this review include the focus on individual case studies rather than a broader systematic review, and reliance on self-reports of features rather than system demonstrations. However, the use of case studies allowed for in-depth discussion with system developers about design, measures, electronic integration and the challenges they have encountered, which could not have been gleaned from documentation of system features alone.

Conclusion

Moving forward, the ability of systems and modern measurement to tailor PRO content to patient needs has the potential to guide clinical use. Current efforts to develop symptom-focused measures (such as the recently released PROMIS GI symptoms measure) that focus on broad score interpretability within patient groups suggest that greater flexibility to tailor content to disease-specific, clinically relevant symptoms is possible and can support clinician actions. While these new symptom-specific measures may not alone solve the challenge of efficiently providing both general and disease-specific information across the patient spectrum, they are an important step in the integration of PRO data into the clinical workflow on a long-term and widespread basis.

The case studies illustrate important steps in the integration of PRO data into the clinical workflow. Technological capabilities and features will continue to advance quickly creating efficient PRO collection tailored for clinical relevance. Research and measure development efforts should continue to focus on understanding the role of both technology and system integration on the validity and reliability of PRO measures.

Acknowledgments

Funding:

The project described above was supported by the following:

Award Number P30CA051008 from the National Cancer Institute. NIH NIMH RO1 grant (RO1 MH084759), the University of Washington Center for AIDS Research NIAID grant (P30 AI027757), and the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) grant (R24 AI067039). PROMIS II was funded by cooperative agreements with a Statistical Center (Northwestern University, PI: David Cella, PhD, 1U54AR057951), a Technology Center (Northwestern University, PI: Richard C. Gershon, PhD, 1U54AR057943), a Network Center (American Institutes for Research, PI: Susan (San) D. Keller, PhD, 1U54AR057926) and thirteen Primary Research Sites which may include more than one institution (State University of New York, Stony Brook, PIs: Joan E. Broderick, PhD and Arthur A. Stone, PhD, 1U01AR057948; University of Washington, Seattle, PIs: Heidi M. Crane, MD, MPH, Paul K. Crane, MD, MPH, and Donald L. Patrick, PhD, 1U01AR057954; University of Washington, Seattle, PIs: Dagmar Amtmann, PhD and Karon Cook, PhD, 1U01AR052171; University of North Carolina, Chapel Hill, PI: Darren A. DeWalt, MD, MPH, 2U01AR052181; Children’s Hospital of Philadelphia, PI: Christopher B. Forrest, MD, PhD, 1U01AR057956; Stanford University, PI: James F. Fries, MD, 2U01AR052158; Boston University, PIs: Stephen M. Haley, PhD and David Scott Tulsky, PhD (University of Michigan, Ann Arbor), 1U01AR057929; University of California, Los Angeles, PIs: Dinesh Khanna, MD and Brennan Spiegel, MD, MSHS, 1U01AR057936; University of Pittsburgh, PI: Paul A. Pilkonis, PhD, 2U01AR052155; Georgetown University, PIs: Carol. M. Moinpour, PhD (Fred Hutchinson Cancer Research Center, Seattle) and Arnold L. Potosky, PhD, U01AR057971; Children’s Hospital Medical Center, Cincinnati, PI: Esi M. Morgan DeWitt, MD, MSCE, 17 1U01AR057940; University of Maryland, Baltimore, PI: Lisa M. Shulman, MD, 1U01AR057967; and Duke University, PI: Kevin P. Weinfurt, PhD, 2U01AR052186). NIH Science Officers on this project have included Deborah Ader, PhD, Vanessa Ameen, MD, Susan Czajkowski, PhD, Basil Eldadah, MD, PhD, Lawrence Fine, MD, DrPH, Lawrence Fox, MD, PhD, Lynne Haverkos, MD, MPH, Thomas Hilton, PhD, Laura Lee Johnson, PhD, Michael Kozak, PhD, Peter Lyster, PhD, Donald Mattison, MD, Claudia Moy, PhD, Louis Quatrano, PhD, Bryce Reeve, PhD, William Riley, PhD, Ashley Wilder Smith, PhD, MPH, Susana Serrate-Sztein, MD, Ellen Werner, PhD and James Witter, MD, PhD.

The Patient-Reported Outcomes Measurement Information System (PROMIS) is an NIH Roadmap initiative to develop a computerized system measuring PROs in respondents with a wide range of chronic diseases and demographic characteristics. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the National Institutes of Health

Reference List

  • 1.Food and Drug Administration; U.S. Department of Health and Human Services. Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. [Accessed October 7, 2014];2009 Dec; Available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf.
  • 2.Conway PH, Mostashari F, Clancy C. The future of quality measurement for improvement and accountability. JAMA. 2013;309:2215–2216. doi: 10.1001/jama.2013.4929. [DOI] [PubMed] [Google Scholar]
  • 3.Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006;12:559–568. doi: 10.1111/j.1365-2753.2006.00650.x. [DOI] [PubMed] [Google Scholar]
  • 4.Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17:179–193. doi: 10.1007/s11136-007-9295-0. [DOI] [PubMed] [Google Scholar]
  • 5.Wagner EH, Austin BT, Von KM. Organizing care for patients with chronic illness. Milbank Q. 1996;74:511–544. [PubMed] [Google Scholar]
  • 6.Wagner EH, Austin BT, Davis C, et al. Improving chronic illness care: translating evidence into action. Health Aff (Millwood) 2001;20:64–78. doi: 10.1377/hlthaff.20.6.64. [DOI] [PubMed] [Google Scholar]
  • 7.Dobscha SK, Gerrity MS, Ward MF. Effectiveness of an intervention to improve primary care provider recognition of depression. Eff Clin Pract. 2001;4:163–171. [PubMed] [Google Scholar]
  • 8.Brown RF, Butow PN, Dunn SM, et al. Promoting patient participation and shortening cancer consultations: a randomised trial. Br J Cancer. 2001;85:1273–1279. doi: 10.1054/bjoc.2001.2073. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Detmar SB, Muller MJ, Schornagel JH, et al. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA. 2002;288:3027–3034. doi: 10.1001/jama.288.23.3027. [DOI] [PubMed] [Google Scholar]
  • 10.Taenzer P, Bultz BD, Carlson LE, et al. Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psychooncology. 2000;9:203–213. doi: 10.1002/1099-1611(200005/06)9:3<203::aid-pon453>3.0.co;2-y. [DOI] [PubMed] [Google Scholar]
  • 11.Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol. 2004;22:714–724. doi: 10.1200/JCO.2004.06.078. [DOI] [PubMed] [Google Scholar]
  • 12.Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res. 2009;18:115–123. doi: 10.1007/s11136-008-9430-6. [DOI] [PubMed] [Google Scholar]
  • 13.Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory? Soc Sci Med. 2005;60:833–843. doi: 10.1016/j.socscimed.2004.06.022. [DOI] [PubMed] [Google Scholar]
  • 14.Lohr KN, Zebrack BJ. Using patient-reported outcomes in clinical practice: challenges and opportunities. Qual Life Res. 2009;18:99–107. doi: 10.1007/s11136-008-9413-7. [DOI] [PubMed] [Google Scholar]
  • 15.Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21:1305–1314. doi: 10.1007/s11136-011-0054-x. [DOI] [PubMed] [Google Scholar]
  • 16.Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Qual Life Res. 2009;18:125–136. doi: 10.1007/s11136-008-9436-0. [DOI] [PubMed] [Google Scholar]
  • 17.Eton DT, Beebe TJ, Hagen PT, et al. Harmonizing and consolidating the measurement of patient-reported information at health care institutions: a position statement of the Mayo Clinic. Patient Relat Outcome Meas. 2014;5:7–15. doi: 10.2147/PROM.S55069. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307:1583–1584. doi: 10.1001/jama.2012.500. [DOI] [PubMed] [Google Scholar]
  • 19.Cella D, Hahn EA, Jensen SE, et al. Washington, DC: National Quality Forum; 2012. [Accessed May 21, 2014]. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Commissioned paper #1. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2%7C. [Google Scholar]
  • 20.Deutsch A, Smith L, Gage B, et al. Washington, DC: National Quality Forum; 2012. [Accessed May 21, 2014]. Patient-reported outcomes in performance measurement. PRO-based performance measures for healthcare accountable entities. Commissioned paper #2. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2%7C. [Google Scholar]
  • 21.Porter ME. What is value in health care? N Engl J Med. 2010;363:2477–2481. doi: 10.1056/NEJMp1011024. [DOI] [PubMed] [Google Scholar]
  • 22.White A, Danis M. Enhancing patient-centered communication and collaboration by using the electronic health record in the examination room. JAMA. 2013;309:2327–2328. doi: 10.1001/jama.2013.6030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Lee TH. Care redesign--a path forward for providers. N Engl J Med. 2012;367:466–472. doi: 10.1056/NEJMhpr1204386. [DOI] [PubMed] [Google Scholar]
  • 24.Wasson JH, Stukel TA, Weiss JE, et al. A randomized trial of the use of patient self-assessment data to improve community practices. Eff Clin Pract. 1999;2:1–10. [PubMed] [Google Scholar]
  • 25.Cella D, Yount S, Rothrock N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45:S3–S11. doi: 10.1097/01.mlr.0000258615.42478.55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Chapman CR, Donaldson GW, Davis JJ, et al. Improving individual measurement of postoperative pain: the pain trajectory. J Pain. 2011;12:257–262. doi: 10.1016/j.jpain.2010.08.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Williams CA, Templin T, Mosley-Williams AD. Usability of a computer-assisted interview system for the unaided self-entry of patient data in an urban rheumatology clinic. J Am Med Inform Assoc. 2004;11:249–259. doi: 10.1197/jamia.M1527. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Economics and Statistics Administration; National Telecommunications and Information Administration; U.S. Department of Commerce. Exploring the Digital Nation: Computer and Internet Use at Home. [Accessed May 21, 2014];2011 Nov; Available at: http://www.ntia.doc.gov/report/2011/exploring-digital-nation-computer-and-internet-use-home.
  • 29.Buxton J, White M, Osoba D. Patients' experiences using a computerized program with a touch-sensitive video monitor for the assessment of health-related quality of life. Qual Life Res. 1998;7:513–519. doi: 10.1023/a:1008826408328. [DOI] [PubMed] [Google Scholar]
  • 30.Thornberry J, Bhaskar B, Krulewitch CJ, et al. Audio computerized self-report interview use in prenatal clinics: audio computer-assisted self interview with touch screen to detect alcohol consumption in pregnant women: application of a new technology to an old problem. Comput Inform Nurs. 2002;20:46–52. doi: 10.1097/00024665-200203000-00009. [DOI] [PubMed] [Google Scholar]
  • 31.Hays RD, Lipscomb J. Next steps for use of item response theory in the assessment of health outcomes. Qual Life Res. 2007;16(Suppl 1):195–199. doi: 10.1007/s11136-007-9175-7. [DOI] [PubMed] [Google Scholar]
  • 32.McCann L, Maguire R, Miller M, et al. Patients' perceptions and experiences of using a mobile phone-based advanced symptom management system (ASyMS) to monitor and manage chemotherapy related toxicity. Eur J Cancer Care (Engl) 2009;18:156–164. doi: 10.1111/j.1365-2354.2008.00938.x. [DOI] [PubMed] [Google Scholar]
  • 33.Berry DL, Blumenstein BA, Halpenny B, et al. Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial. J Clin Oncol. 2011;29:1029–1035. doi: 10.1200/JCO.2010.30.3909. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. 1999;282:1737–1744. doi: 10.1001/jama.282.18.1737. [DOI] [PubMed] [Google Scholar]
  • 36.Spitzer RL, Williams JB, Kroenke K, et al. Validity and utility of the PRIME-MD patient health questionnaire in assessment of 3000 obstetric-gynecologic patients: the PRIME-MD Patient Health Questionnaire Obstetrics-Gynecology Study. Am J Obstet Gynecol. 2000;183:759–769. doi: 10.1067/mob.2000.106580. [DOI] [PubMed] [Google Scholar]
  • 37.Donaldson G. Patient-reported outcomes and the mandate of measurement. Qual Life Res. 2008;17:1303–1313. doi: 10.1007/s11136-008-9408-4. [DOI] [PubMed] [Google Scholar]
  • 38.Fries JF, Witter J, Rose M, et al. Item response theory, computerized adaptive testing, and PROMIS: assessment of physical function. J Rheumatol. 2014;41:153–158. doi: 10.3899/jrheum.130813. [DOI] [PubMed] [Google Scholar]
  • 39.Crane PK, Narasimhalu K, Gibbons LE, et al. Item response theory facilitated cocalibrating cognitive tests and reduced bias in estimated rates of decline. J Clin Epidemiol. 2008;61:1018–1027. doi: 10.1016/j.jclinepi.2007.11.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Sage JM, Ali A, Farrell J, et al. Moving into the electronic age: validation of rheumatology self-assessment questionnaires on tablet computers. Arthritis Rheum; ACR/ARHP Annual Meeting; November 9–14, 2012; Washington, DC. 2012. p. S1102. [Abstract #203]. Number 10. [Google Scholar]
  • 41.Jensen RE, Snyder CF, Abernethy AP, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract. 2013 Dec 3; doi: 10.1200/JOP.2013.001067. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Cella D, Choi S, Garcia S, et al. Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment. Qual Life Res. 2014 Jun 18; doi: 10.1007/s11136-014-0732-6. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Wagner LI, Spiegel D, Pearman T. Using the science of psychosocial care to implement the new american college of surgeons commission on cancer distress screening standard. J Natl Compr Canc Netw. 2013;11:214–221. doi: 10.6004/jnccn.2013.0028. [DOI] [PubMed] [Google Scholar]
  • 44.Kotronoulas G, Kearney N, Maguire R, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32:1480–1501. doi: 10.1200/JCO.2013.53.5948. [DOI] [PubMed] [Google Scholar]
  • 45.Kitahata MM, Rodriguez B, Haubrich R, et al. Cohort profile: the Centers for AIDS Research Network of Integrated Clinical Systems. Int J Epidemiol. 2008;37:948–955. doi: 10.1093/ije/dym231. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Fredericksen R, Crane PK, Tufano J, et al. Integrating a web-based, patient-administered assessment into primary care for HIV-infected adults. J AIDS HIV Res. 2012;4:47–55. doi: 10.5897/jahr11.046. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Lawrence ST, Willig JH, Crane HM, et al. Routine, self-administered, touch-screen, computer-based suicidal ideation assessment linked to automated response team notification in an HIV primary care setting. Clin Infect Dis. 2010;50:1165–1173. doi: 10.1086/651420. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Gutteling JJ, Darlington AS, Janssen HL, et al. Effectiveness of health-related quality-of-life measurement in clinical practice: a prospective, randomized controlled trial in patients with chronic liver disease and their physicians. Qual Life Res. 2008;17:195–205. doi: 10.1007/s11136-008-9308-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS) Med Care. 2007;45:S22–S31. doi: 10.1097/01.mlr.0000250483.85507.04. [DOI] [PubMed] [Google Scholar]

RESOURCES