Implementing patient-reported outcome surveys as part of routine care: lessons from an academic radiation oncology department

Abstract

Patient reported outcomes (PROs) are reports of health conditions that come directly from patients. Use of PROs has been associated with improved patient outcomes, enhanced quality of life, and reduced end-of-life spending. Yet there are still outstanding questions regarding the process of implementing PRO collection in routine practice. In this article, we describe the experience of selecting and implementing PROs in a multisite, multidisease academic medical center–based radiation oncology practice and demonstrate that such large-scale rollout is feasible. We establish that PROs can be implemented with minimal to no workflow delays, are generally seen as valuable by clinicians, and can enhance patient-doctor communication. We additionally detail the challenges involved in selecting clinically relevant PRO questionnaires and the centrality of physician buy-in, easy data access, and clear workflows to successful implementation.

INTRODUCTION

Patient reported outcomes (PROs), defined as reports of patient health emanating directly from patients, help facilitate communication between patients and providers. PRO use has been associated with improved patient outcomes,¹ quality of life,² and patient-clinician communication.^3,4 PROs help physicians recognize undetected symptoms⁵ and increase symptom-related actions taken by clinicians.⁶

Use of PROs has gained prominence in oncology, with 29% of clinical trials incorporating some form of PROs as of 2013.⁷ In radiation oncology, a significant body of work has used PROs to describe differences in patient experience across treatment modalities and regimens,^8–11 as well as the temporal progression of symptoms and their predictors.¹² A recent study demonstrated that when palliative experts were alerted to clinically significant changes in PROs reported by radiation oncology patients, average per-patient end-of-life care spending decreased by $7,317 and patients experienced fewer hospital deaths.¹³

However, outstanding questions remain regarding the process of collecting PROs in daily practice. Unresolved topics include time constraints imposed by collecting PROs and addressing issues they elicit, whether PROs truly add value, the feasibility of patients efficiently interacting with technology-based reporting systems in routine care, and how to optimally select PROs for clinical use.¹⁴ In particular, literature on the barriers to and facilitators of large-scale PRO implementation in radiation oncology is limited. We thus sought to describe the experience of selecting and implementing PROs in a multisite, multidisease academic medical center–based radiation oncology department as part of routine clinical care. We additionally aimed to characterize opinions around PRO utility and the workflow and technology features associated with successful implementation.

IMPLEMENTATION EXPERIENCE

Motivation

The initiative to routinely collect PROs was launched in the Dana Farber/Brigham and Women’s Cancer Center radiation oncology department as part of a broader Partners Healthcare movement to acquire patient outcomes. At the time of the radiation oncology implementation, PROs had already been rolled out in cardiology, orthopedics, and rheumatology clinics. The rollout of PROs in radiation oncology was driven by physician interest in enhancing patient care and the opportunity to collect data for quality improvement and research purposes. The department chief requested that we build the infrastructure to routinely collect PROs with the aim of having questionnaires appropriate to each patient at each clinical time point.

Departmental structure

The department of radiation oncology consists of 40 physicians practicing across 2 main campuses and 4 network locations. Physicians typically subspecialize in the treatment of 1 or 2 disease sites. Residents, fellows, nurses, nurse practitioners, and radiation therapists complete the clinical team. Approximately 350 patients are treated daily across the department.

Selection of PRO measures and software

We asked physician and nursing leaders from each disease site within the radiation oncology department to choose (1) validated questions appropriate for eliciting symptom burden and severity during treatment and (2) questionnaires validated to track long-term symptoms. We suggested that they select from existing validated questionnaires (such as the Functional Assessment of Chronic Illness Therapy), tailor questions to cancer type and location, and emphasize questions’ clinical utility over research utility. Questions spanned physical and mental well-being, with a focus on radiation-related symptoms. We planned for a shorter questionnaire to assess acute treatment side effects and a separate one to assess long-term complications. We made an effort to limit baseline/follow-up questionnaires to 20–30 questions and on-treatment instruments to fewer than 20 questions.

Chosen question combinations were reviewed by the project leaders and Partners Healthcare PROs team, and subsequently programmed in Tonic (Menlo Park, CA, USA) digital survey software. Skip logic was used minimally (eg, gender-dependent questions), while composite scores were calculated if available for established surveys.

Tonic was selected as a software vendor at an enterprise level at a time when Partners Healthcare used a proprietary medical record. Tonic provides an intuitive, patient-friendly iPad-based survey display that integrates with Partners’ scheduling and electronic medical record (EMR) systems. Completed surveys are available in the EMR as PDFs. Table 1 describes the questionnaires used for each disease site at baseline/follow-up and during treatment.

Table 1.

Sources of baseline and follow-up questionnaires, by disease site

Site	Baseline/follow-up	On treatment visit
Gastrointestinal
Esophageal	FACT-G + EORTC-OES18	Custom
Gastroesophageal junction	FACT-G + EROTC-OG25	Custom
Gastric	FACT-G + EORTC-STO22	Custom
Pancreas and liver	FACT-Hep	Custom
Colorectal	FACT-G + EORTC-CR29	Custom
Anal	FACT-G + EORTC-CR29	Custom
Breast	FACT-G + FACT-B	Custom
Central nervous system	MDASI-Brain Tumor	MDASI-Brain Tumor
Head and neck	MDASI-HN	Custom
Gynecologic
Endometrial	FACT-G + FACT-En	Custom
Cervix	FACT-G + FACT-Cx	Custom
Vulvar	FACT-G + FACT-V	Custom
Genitourinary
Bladder	Bladder Cancer Index	Custom
Prostate	EPIC-26	Custom
Penile	PROMIS-10	Custom
Seminoma	PROMIS-10	N/A
Skin	PROMIS-10	RISRAS
Thoracic	EORTC Q30 + LC13	Custom
Sarcoma	PROMIS-10	PROMIS-10
Lymphoma	FACT-G + FACT-Lym
Head and neck	N/A	Custom
Thoracic	N/A	Custom
Stomach	N/A	Custom
Pelvis	N/A	Custom
Palliative	MDASI	MDASI

EPIC: Expanded Prostate Cancer Index Composite; MDASI: MD Anderson Symptom Index; FACT: Functional Assessment of Cancer Therapy; EORTC: European Organization for Research and Treatment of Cancer; PROMIS-10: Patient Reported Outcomes Measurement Information System Global Health Scale

Rollout process and logistics

In December 2014, PROs were rolled out over several weeks at all clinic locations. To facilitate electronic survey completion, 18 iPads were strategically placed at nursing stations throughout clinic and treatment areas at Dana Farber/Brigham and Women’s Cancer Center. Prior to rollout, structured introductory presentations were made to each disease site’s medical assistants (MAs) and clinical team in order to describe the purpose of PROs and associated workflows. Special attention was paid to iPad security, including how to store iPads in locked locations and how to respond if one went missing. We also created simple flyers to be handed to patients at clinic check-in that explained the rationale for PRO collection.

As the workflow currently stands, after a patient signs in at the clinic, an MA (or practice coordinator or nurse at network sites) introduces PROs to the patient via a script (staff are encouraged to engage all adult patients). The MA then logs in to the iPad and Tonic application, chooses the patient from that day’s schedule, selects from a menu the disease site and visit type the patient is being seen for, and hands the patient the iPad with the disease site and time point–specific questionnaire. To facilitate this handoff, we developed an appointment scheduling workflow in which the MA schedule notes the disease sites patients are under treatment for.

The Tonic software provides patient instructions in both written and video form. Survey questions are presented one at a time, with the Likert scale enlarged and depicted in bright, easy-to-differentiate colors (Figure 1 ). All questionnaires are in English, aside from the Patient Reported Outcomes Measurement Information System-10, which is available in Spanish, but in-person translation assistance is available upon request. Patients can elect to not fill out PRO surveys, and allowing question skipping was prioritized over data completeness. Within 2 min of a patient’s finishing the questionnaire, a PDF with results appears in the EMR. Providers are able to see both individual answers and time-trended summary scores. Figure 2 shows a sample physician report. Data are also aggregated on a patient-specific basis in the Tonic database.

Figure 1.

Sample questionnaire screen in Tonic.

Figure 2.

Sample panels from physician report, as seen in the Epic electronic medical record.

ASSESSMENT

As of June 2016, 21 695 outcomes had been collected from 7640 patients; 9967 outcomes were from baseline or follow-up visits, while 11 639 were from on-treatment visits.

In addition to tracking data aggregation over time since PRO rollout, we used the Tonic system to track the average time it takes patients to complete 1 question and a whole survey. In addition, we conducted an anonymous electronic survey of providers, including residents, attendings, and nurse practitioners (n = 53, 60% response rate in June 2016; n = 36, 50% response rate in June 2015), regarding their use of and experience with PROs 3 months and 1.5 years post-rollout. Follow-up semistructured qualitative interviews were conducted in person with 8 participants, who reported either very positive or negative experiences at 1.5 years post-rollout, to identify predictors of successful implementation.

As of June 2016, PROs were regularly administered for all disease sites/types, aside from pediatrics and central nervous system. Each PRO question and common survey took on average ∼14 s and ∼10 min to complete, respectively. In all, 86% of providers gave PRO surveys to their patients, but only 34% reviewed the results in their daily practice vs 47% a year prior (see Figure 3 A), with the decrease possibly due to the additional steps required to review responses after Partners’ Epic rollout in June 2015. A total of 88% of providers who reviewed results did so in Epic; the rest received pass-off from clinical team members. Of the providers, 52% reported no or rare impact of PRO administration on their ability to see patients on time, while 38% and 8% reported minor or moderate delays, representing a notable decrease vs a year prior (Figure 3A).

Figure 3.

Results from provider survey regarding PRO perception and use. (A) Comparison of PRO use and perception metrics, spring 2015 vs spring 2016. (B) Reported technical barriers to PRO use.

Of the respondents, 70% felt that PRO data were valuable to their role as a provider all the time, usually, or some of the time (similar to 2015; Figure 3A), and 77% said that PRO results were always or usually consistent with results elicited from patients. Also, 26% of providers said that PRO collection had changed their discussion of symptoms with patients or their decision-making. A majority said they would be very to moderately likely to recommend PROs to a colleague in another department or institution, including 82% of those who reviewed PROs in daily practice and 47% of those who did not.

Providers said PROs often made them aware of additional symptoms that the patient-doctor dyad had not discussed or whose severity had been downplayed. Examples included nausea, fatigue, sexual dysfunction, depressed mood, and rectal bleeding. Additionally, providers noted that PROs gave them a starting point from which to elaborate on difficult-to-discuss symptoms, such as sexual dysfunction.

Key barriers to PRO use included difficulty opening reports in the EMR, bringing data into clinical notes and accessing data, and having too much data to sift through (see Figure 3B). Providers’ main suggestions for PRO improvement included (1) finding a better way to visualize PRO data longitudinally, (2) requiring fewer clicks to access data, and (3) using shorter surveys. They noted that exam room setup was sometimes not conducive to PRO review and that patients were frustrated if they took the time to fill out the questionnaire but providers did not have time to review the data or address their concerns.

Additionally, qualitative responses suggested that clearer workflows were needed around PROs, including whether PROs are reviewed prior to appointments and who reviews them. Both patients and clinical staff felt that some of the questionnaires could be streamlined to reduce redundancy between survey and physician questions.

Individual interviews with nurses and physicians suggested that, although workflows varied among disease sites (ie, in some cases, nurses reported PROs to team physicians, and in others, physicians reviewed PROs themselves), physician buy-in was key to successful PRO implementation. Implementation was least successful when physicians did not find PROs valuable or did not feel they had time for them. In those situations, nurses were hesitant to administer PROs if they would not be reviewed. Nurse practitioners who saw their own patient panels were more often able to integrate PROs into their workflow independent of physicians’ views.

DISCUSSION

We have described the feasibility of implementing routine PRO collection in a large, academic, multisite radiation oncology department. Our results demonstrate that electronic collection of PROs throughout the course of radiation treatment results in minimal workflow delays and is generally considered valuable by clinicians. PROs are consistent with reports elicited from patients and often bring to the forefront symptoms not previously discussed, with positive effects on patient-doctor communication and treatment.

This implementation was helped by strong departmental leadership support, clinical staff interest, and enterprise support with a technology platform integrated with scheduling and the EMR. Selection of clinically relevant questionnaires proved to be the biggest challenge. Validated PRO measures come from the research literature and are not always ideal for routine clinical collection. Concise disease-site and context-specific questionnaires have been developed to fill this gap.^15,16

Our results suggest that ease of use of technology interfaces and physician buy-in are key predictors of successful PRO implementation. Accordingly, building PRO platforms that are accessible via limited clicks, display data longitudinally, and rapidly convey key data should be a priority for teams exploring PRO use. Additionally, helping physicians find the value of PROs in their practice (ie, via feedback-responsive question modification or demonstration of PRO integration into workflows) may be important for shaping PRO acceptance beyond initial training and launch.

Our study is consistent with previous smaller-scale work demonstrating the feasibility of technology-based PRO reporting interventions.^10,17,18 While its strengths include a description of a real-world implementation effort in a complex academic ecosystem, its drawbacks include the lack of a randomized design, lack of compliance measurement, and description of a work-in-progress system whose strengths and weaknesses are still being explored. Next steps include rollout of an electronic PRO platform with enhanced visualization capabilities and descriptions of threshold measures that can alert clinicians to significant changes in symptomatology.

In conclusion, we have demonstrated the utility of PRO rollout across a multisite radiation oncology institution while reemphasizing the importance of ease of data visualization and manipulation in adoption of nonmandated technologies with quality-enhancing potential.

Contributors

• LSR, NEM: Substantial contributions to the conception and design of the work; acquisition, analysis, and interpretation of data for the work; drafting the work or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

• AA, KO’N, AK, MK, CW, PFO III, NW: Substantial contributions to the conception and design of the work; acquisition and interpretation of data for the work; drafting the work or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

• BK: Substantial contributions to the conception and design of the work and interpretation of data for the work; drafting the work or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector.

Competing Interests

NEM is a paid consultant for Via Oncology.