Integrating Patient-Reported Outcome Measures into Routine Cancer Care: Cancer Patients’ and Clinicians’ Perceptions of Acceptability and Value

Abstract

Introduction:

Despite growing interest in integrating patient-reported outcome (PRO) measures of symptoms and functional status into routine cancer care, little attention has been paid to patients’ and clinicians’ perceptions of acceptability and value.

Methods:

A two-phase qualitative study was conducted to develop a web-based PRO screening system with 21 items assessing symptoms (e.g., nausea) and functional status. Phase 1 involved cognitive interviews with 35 cancer outpatients (n=9 breast chemotherapy, radiation for prostate (n=8) or head and neck cancer (n=10), and n=8 bone marrow transplant [BMT]). In Phase 2, we evaluated the acceptability and perceived value of reviewing a PRO measure during real-time clinical encounters with 39 additional outpatients (n=10 breast, n=9 head and neck, n=10 prostate, n=10 BMT) and 12 clinicians (n=3 breast, n=2 head and neck, n=4 prostate, n=3 BMT). At least 20% of patients were ≥60 years, African American, or ≤ high school.

Results:

Patients felt that their PRO summary of symptoms and functional status was helpful in discussing health issues with clinicians (92%), wanted to review their results with clinicians during future visits (82%), and would recommend it to other patients (87%). Clinicians found the PRO summary to be easy to interpret (83%), most helpful for documenting the Review of Symptoms (92%), and would recommend it to future patients (92%). Over 90% of clinicians reported that consultation time did not increase.

Conclusion:

Both cancer patients and clinicians reported that discussing a PRO summary of symptoms and functional status during an outpatient visit was useful, acceptable, and feasible.

Introduction

The Affordable Care Act emphasizes personalized medicine and health information technology (HIT) to improve quality of care and manage health care costs.1–3 An important aspect of personalized medicine is to integrate real-time, patient-reported outcome (PRO) measures into routine clinical visits. Current practice standards in oncology involve a clinician assessing a patient through open-ended questions, performing an exam, and then interpreting the patient’s responses about symptoms and functional impact. In many hospital systems and practices, standardized PRO measures are not administered or discussed with the patient during routine visits. However, the gold standard is to assess symptoms and functional status by patient report4 because providers tend to underdetect symptom onset,5–8 severity,9 and frequency of symptoms.8

Despite interest in using PRO measures during outpatient visits, little attention has been paid to clinicians’ and patients’ perceptions of acceptability and value. Reviewing PRO measures during clinical care provides an opportunity for the patient to clarify and elaborate on treatment side effects and other symptoms that may need to be managed.10 A systematic review of controlled trials showed that reviewing PRO measures with cancer patients increases discussion of symptoms such as pain and emotional distress.11 It may also increase providers’ detection and management of symptoms. For instance, oncology clinicians randomized to receive feedback from a PRO pain measure were more likely to make prescription changes for patients in the intervention group.12 This increase in communication and management of symptoms that occurs when PRO measures are reviewed may be why patients report having higher well-being13 and satisfaction with care.11

Less information is available about clinicians’ perceptions of value and acceptability. Three prior studies have examined oncologists’ perceptions of the advantages and disadvantages of discussing PRO measures with their patients during future clinical visits.14–16 The main advantages were the identifying and rectifying of problematic symptoms.14–16 Oncologists expressed concerns over potential patient burden, increasing consultation time, cost, inefficiency, staffing needs, and depersonalization of the physician-patient encounter.14–16 Clinicians in these studies were asked to imagine hypothetical scenarios, and thus no studies have been conducted during real-time testing to determine whether clinicians perceive it to be valuable and feasible to discuss a PRO measure during a routine visit.

Similarly, little research is available to inform best practices for integrating PRO questionnaires into routine care work flow and associated health information technology systems. Snyder et al. developed a user’s guide for implementing PROs in clinical practice.17 They noted a number of methodological and practical decisions that must be made, such as identifying the goals for collecting PROs; choosing questionnaires, timing of assessments, and mode of administration; developing processes for result reporting; and developing strategies for responding to issues identified by the PRO measures.17 This breadth of choices has led to heterogeneous implementation strategies across hospital systems. A recent systematic review concluded that there was substantial variation in the ways that 33 cancer hospital systems administered and integrated into clinical practice their web-based PRO measures.18 Few of the electronic systems appeared to be developed with input from either clinicians or patients.18 It was also unclear if any of the systems had been developed with both clinician and patient stakeholder input.18 Thus, there is a need to develop and evaluate a web-based screening system to elicit patient-reported symptoms and functional status that considers what cancer patients and clinicians need during clinical encounters. A web-based system was chosen because it permits immediate scoring and the creation of summaries highlighting a patient’s most problematic symptoms, and it can be immediately uploaded into an EHR.

Our primary objectives were the following: (1) solicit feedback from cancer clinicians and patients to develop a web-based screening system for securely administering and summarizing PRO measures for use during routine cancer care; and (2) pilot test the system during outpatient visits to evaluate cancer patients’ and clinicians’ perceptions of the acceptability and value of discussing PRO measures during clinical care.

Methods

We conducted a two-phase qualitative study to design the web-based Patient-Reported Symptom Monitoring (PRSM) system with stakeholder input from patients and clinicians. Our process was informed by the Consolidated Framework for Implementation Research (CFIR) in health services research.19 We used four CFIR domains to design a rigorous qualitative study: (1) patient characteristics; (2) patient needs; (3) clinician needs; and (4) evaluation of perceived acceptability, feasibility, and value.

In Phase 1, we evaluated a brief PRO screener for use across patients with different cancer types and a web-based system that collects patient responses in real time and summarizes responses to highlight the patient’s most problematic needs. In Phase 2, the PRO summaries were implemented into outpatient visits for usability testing. Both patients and clinicians reported on feasibility and perceived value. The two phases of development and evaluation are summarized in Figure 1.

Figure 1.

Overview of Two Research Phases for Development and Usability Testing

Note: Patient-Reported Symptom Monitoring (PRSM) system

Purposive Sampling of Cancer Patients

Three cancer clinics at a large academic medical center were selected because they treat patients who experience a range of symptoms resulting from treatment: the Bone Marrow and Stem Cell Transplant Clinic, Breast Medical Oncology Clinic (chemotherapy), and Radiation Oncology Clinic for head and neck, and prostate cancer patients. All patients were undergoing active treatment.

Purposive sampling is a qualitative research technique involving strategic choices about which individuals to include in a sample (typically not probability-based).20 A purposive sampling strategy was implemented to recruit at least 20 percent of cancer patients who were ages 60 years and older, were African American, or had a high school education or less. Patients could fulfill recruitment categories for more than one group by meeting multiple criteria. We included age, race, ethnicity, and education in the sampling strategy to generalize to the academic medical center’s cancer population and to other hospital systems. Additionally, prior research has shown that these groups may have more difficulty understanding, and may respond in different ways to, health-related questionnaires.21–23 Cancer patients who are older, are in a racial or ethnic minority, or who have a low education level are also at risk of disparities in treatment outcomes.24–28 Ethnicity was coded as “Hispanic” or “non-Hispanic” based on the demographic questionnaire, regardless of race.

Institutional Review Board (IRB) approval was obtained from the University of North Carolina at Chapel Hill. Patients in each phase were given an incentive of $25. Clinicians were not incentivized. Different cancer patients were recruited for Phases 1 and 2.

Selection of Symptoms and Questionnaires

First, our multidisciplinary research team decided which symptoms to include in the screener PRO measure. The symptoms needed to be applicable to patients across a variety of cancer types and treatments and the measure needed to be brief. We conducted a literature review,10–11,17,29–32 and the research team was unanimous in their decision to include the National Cancer Institute (NCI)’s list of symptoms to assess in clinical trials (fatigue, insomnia, pain, appetite loss, dyspnea, cognitive problems, anxiety, nausea, depression, sensory neuropathy, constipation, and diarrhea).33 The only exception was that constipation was not included in the screener. The study team also included headache, cough, rash, and urine leakage – given their moderate prevalence and effects on quality of life.34–35 Quality of life was included as a domain – given its relevance for prognosis36 and importance to patients. Finally, the research team felt it was important to give patients an opportunity to “set an agenda” for what they would like to discuss with their clinician to better personalize care.

Symptoms were operationalized with 16 items adapted from the PRO version of the NCI Common Terminology Criteria for Adverse Events (PROCTCAE);37 3 items from the Patient-Reported Outcomes Measurement Information System (PROMIS) Global Health scale assessing perceived health, general quality of life, and the ability to conduct daily activities;38 and 2 items written by the authors (21 total items: see Appendix A). The PRO-CTCAE was selected because it is a PRO developed to closely match clinicians’ toxicity ratings, and patients and clinicians were involved in its development.37 The study team wrote an item to assess “other symptoms not listed” for flexibility in reporting uncommon symptoms. The PROMIS Global Health scale was selected because it has satisfactory psychometric properties and national benchmarks for score comparisons,39 and it was developed with patient and clinician input.38 No measures were found to assess agenda setting, thus the research team wrote the following item: “What would you like to talk to your nurse or doctor about today?”

A seven-day time frame was selected because the interval is short enough for accurate recall yet allows for assessment of change.40 This time frame is also consistent with national initiatives such as PROCTCAE37 and PROMIS.38–39

Study Design

Phase 1: Cognitive Interviews

In Phase 1, we conducted in-depth cognitive interviews with cancer patients about the 21-item screener questionnaire. This was done to make sure the questions were comprehendible and measured what they were supposed to measure from the patient’s perspective (i.e., face validity), and that patients could provide valid responses reflective of their symptom and functional status. We conducted cognitive interviews in two rounds with a total of 35 patients. Trained interviewers held bachelor’s and master’s degrees and had at least two years of experience interviewing patient populations.

Patients in the first round of cognitive testing evaluated the 21 screening questions and a mockup of a report summarizing responses from a fictional patient. Substantially revised questions were reviewed with 10 additional patients. Items were defined as “substantially revised” if their revision involved the following: (1) adding or removing words that changed the meaning of a phrase; (2) word substitutions that in the judgment of the research team were more than a semantic simplification; or (3) changes to response options. The determination of which items qualified as “substantially revised” was a subjective judgment done by consensus. Coding for comprehensibility issues was done after interviews. We expected few problems to be noted for PRO-CTCAE and PROMIS questions because they had undergone cognitive testing at other academic centers.37–38

Phase 2: Patients’ and Clinicians’ Perceptions of Acceptability and the Value of Discussing PRO Measures During Clinical Encounters

LimeSurvey was chosen as the Web server-based software and was installed on hospital servers. LimeSurvey is an open source, online survey application written in PHP and distributed under the GNU General Public License (LimeSurvey.org).

Cancer patients were provided with an Internet-enabled tablet computer (XOOM or iPad) in the clinic waiting room and were logged into the survey by a research assistant. They were asked to complete the 21-item screener questionnaire before seeing their clinician. A summary of patient responses was automatically generated that listed symptoms, in order from most to least severe. A research assistant gave a hard copy summary to both the patient and clinician prior to the visit.

Immediately following the visit with the clinician, 39 cancer patients completed a questionnaire about acceptability, perceived value, and comprehension of the screening questions and summary sheet. Additionally, 20 of the 39 patients completed a semistructured interview after using the web-based system to elicit feedback on which aspects were useful or not useful and suggestions for revisions. These 20 patients were selected by inviting every other person in order of assessment date. If a patient was unavailable to complete the interview after the clinical visit, the next patient was selected.

Clinicians completed a questionnaire after each patient visit about the perceived value and utility of their patient’s PRO summary. We collected clinician’s gender, credentials (e.g., MD, nurse practitioner, etc.), and number of years in practice. In order to limit the burden, no information was collected about the patient’s disease stage. Treatment type was recorded based on the clinic the patient was attending (breast chemotherapy, radiation clinic for prostate or head and neck cancer, or bone marrow transplant clinic).

Results

Phase 1: Cognitive Interviews with Cancer Patients about the PRO Measure

Two rounds of cognitive interviews were conducted in order to ensure that questions were comprehensible to patients. Twenty-five cancer patients participated in round 1 and 10 additional patients participated in round 2; 66 percent were Caucasian, 29 percent had a high school education or less, and 49 percent were ages 60 and older (Table 1).

Table 1.

Demographic Characteristics of Cancer Patients in Phase 1 and Phase 2

	CANCER PATIENT COGNITIVE INTERVIEWS (n=35)* n (%)	CANCER PATIENT USABILITY TESTING IN CLINIC (n=39) n (%)
Clinic
Breast (Chemotherapy)	9 (26%)	10 (26%)
Head and Neck (Radiation)	10 (28%)	9 (23%)
Prostate (Radiation)	8 (23%)	10 (26%)
Bone Marrow (Transplant)	8 (23%)	10 (26%)
Age
18–40	1 (3%)	3 (8%)
41–50	4 (11%)	4 (10%)
51–60	13 (37%)	17 (44%)
61+	17 (49%)	15 (38%)
Gender
Female	17 (49%)	20 (51%)
Ethnicity
Hispanic	1 (3%)	1 (3%)
Race
Caucasian	23 (66%)	21 (54%)
African American	12 (34%)	15 (38%)
Other	0 (0%)	3 (8%)
Education
≤ High School	10 (29%)	16 (41%)
Some College	10 (29%)	10 (26%)
College	8 (23%)	8 (21%)
Postgraduate	7 (20%)	5 (13%)
Marital Status
Never Married	2 (6%)	3 (8%)
Married/Partner	21 (60%)	26 (66%)
Separated/Divorced	10 (29%)	9 (23%)
Widowed	2 (6%)	1 (3%)
Occupation
Employed	12 (34%)	15 (38%)
Retired	13 (37%)	10 (26%)
Disabled	9 (26%)	10 (26%)
Other	1 (3%)	4 (10%)
Income
≤$20,000	8 (23%)	13 (33%)
$20,001–$60,000	10 (28%)	13 (33%)
$60,001+	15 (43%)	12 (31%)
No response	2 (6%)	1 (3%)

Note:

^*25 patients participated in round 1 of cognitive interviews and 10 patients in round 2.

Three symptom terms were revised based on patient feedback (Appendix B). Several participants had difficulty understanding the phrase “loss of control of urine,” and it was changed to “loss of urine or leakage.” A qualifier was added for anxiety (worrying). “Fatigue, tiredness, or lack of energy was amended to “tiredness, lack of energy, or fatigue” because “fatigue” was difficult for several patients to read. We also revised the phrasing of questions measuring symptom severity (13 items) because 10 out of 25 patients had difficulty comprehending what was being assessed (Appendix B). The phrase, “In the last seven days, what was the severity of your [symptom] at its worst?” was revised to, “In the last 7 days, rate your worst [symptom].”In Round 2, patients reported that all rewritten questions were satisfactory (Appendix A).

Phase 2: Usability Testing During Clinical Encounters: Cancer Patients’ Perceptions of Acceptability and Value

In usability testing during clinical care, 82 percent of 39 additional cancer patients were Caucasian, 41 percent had a high school education or less, and 38 percent were ages 60 years and older (Table 1). Gender was balanced across the clinics.

Most cancer patients felt that reviewing a summary of their PRO measure with their clinician was helpful in discussing health issues (92 percent), wanted to review their PRO results with their clinician during future visits (82 percent), and would recommend it to other patients (87 percent). Over half of patients (64 percent) reported that the screening questions were helpful in discussing medical issues with their provider that might have been missed otherwise.

The majority (80 percent) chose to set an agenda for what they would like to discuss with their clinician during the visit. For instance, in Figure 2, an over 40-year-old African American woman receiving chemotherapy for breast cancer wanted to talk to her doctor about how long her numbness would last. She reported her numbness as being “mild,” but the agenda setting item permitted the patient to indicate it was important to discuss during the visit.

Figure 2.

PRO Measure Summary from a Patient

When asked about the length of the web-based PRO measure, 92 percent of cancer patients were willing to answer additional questions; of those, 28 percent were willing to complete 10 more questions, 15 percent to complete 20 more questions, and 15 percent to complete 30 or more questions. Over half (67 percent) were willing to complete the survey at home on a computer or smartphone and had Internet access, 15 percent were willing to complete the screening questions at home but did not have Internet access, and 8 percent did not want to complete the screening items at home (5 percent responded “I don’t know” and 5 percent did not provide an answer). See Table 2 for additional results with cancer patients. In Table 2, the response option “somewhat easy” was given its own column because it may have been a more socially acceptable way to indicate less enthusiasm than “difficult.”

Table 2.

Cancer Patient Impressions of Acceptability and Perceived Value (n=39)

	VERY EASY/EASY	SOMEWHAT EASY	NEITHER DIFFICULT NOR EASY	VERY DIFFICULT/DIFFICULT/SOMEWHAT DIFFICULT	NO ANSWER	NO SYMPTOMS IN LAST 7 DAYS
Seeing questions on computer	36 (92.2%)	1 (2.6%)	0	2 (5.2%)	0	0
Using tablet computer to answer questions	34 (87.1%)	3 (7.7%)	1 (2.6%)	1 (2.6%)	0	0
Answering questions about your symptoms and health	37 (94.8%)	0	0	2 (5.2%)	0	0
Understanding paper copy of your answers	31 (79.5%)	0	2 (5.1%)	0	0	6 (15.4%)
	VERY SATISFIED/SATISFIED	SOMEWHAT SATISFIED	NEITHER SATISFIED NOR UNSATISFIED	VERY DISSATISFIED/DISSATISFIED/SOMEWHAT DISSATISFIED	NO ANSWER	NO SYMPTOMS IN LAST 7 DAYS
In general, how satisfied were you with using the computer to report your symptoms?	37 (94.8%)	0	1 (2.6%)	0	1 (2.6%)	0
Overall, how satisfied are you with using the PRSM system?	32 (82.1%)	2 (5.1%)	0	0	5 (12.8%)	0
	VERY HELPFUL/HELPFUL	SOMEWHAT HELPFUL	NEITHER HELPFUL NOR UNHELPFUL	VERY UNHELPFUL/UNHELPFUL/SOMEWHAT UNHELPFUL	NO ANSWER	NO SYMPTOMS IN LAST 7 DAYS
How helpful was the symptom survey to remind you of symptoms you experienced in the last 7 days?	25 (64.1%)	2 (5.1%)	2 (5.1%)	0	4 (10.3%)	6 (15.4%)
How helpful was the symptom survey for talking to your doctor/ nurse about symptoms you experienced?	30 (76.9%)	6 (15.4%)	2 (5.1%)	0	0	1 (2.6%)

Note: PRSM = Patient-Reported Symptom Monitoring (PRSM) System

We also conducted one-on-one interviews with 20 out of 39 patients immediately after their clinical visit to solicit feedback. Cancer patients reported high satisfaction with the web-based screening tool and PRO summary, and had few recommendations for revisions. Positive comments included finding the tablet computer easier and faster to use than paper-and-pencil questionnaires. Negative comments included needing a slightly longer time to use the on-screen keyboard to respond to open-ended questions (<10 percent).

Clinicians’ Perceptions of Acceptability and Utility

Twelve clinicians were predominantly MDs (67 percent), male (58 percent), and had been practicing for an average of 10 years. Three clinicians treated breast cancer: two MDs (one male, one female) and one female Nurse Practitioner (NP). Two clinicians treated head and neck cancer: one male MD and one female NP. Four clinicians treated prostate cancer: three MDs (all male) and one female NP. Three clinicians treated bone marrow transplant patients: two MDs (both male) and one female NP.

The average number of patients seen by each clinician for this study was 2.6 (SD=1.9). In general, clinicians who saw multiple study patients reported similar satisfaction scores across patients, and thus we averaged across each clinician. In other words, each clinician contributed one averaged score in Table 3. As a sensitivity analysis, Table 3 presents results separately for seven clinicians unaffiliated with our study team and five clinicians who were part of our study team. Scores were similar whether from clinicians who were affiliated or unaffiliated with our study team, and thus we report in the text on the total (12 clinicians). In Table 3, the response option “somewhat helpful” was given its own column because it may have been a more socially acceptable way to indicate less enthusiasm than “unhelpful.”

Table 3.

Clinician Impressions of PRO Summary (n=12)

ITEM	AFFILIATED OR UNAFFILIATED CLINICIANS	VERY HELPFUL/HELPFUL	SOMEWHAT HELPFUL	NEITHER HELPFUL NOR UNHELPFUL	VERY UNHELPFUL/UNHELPFUL/SOMEWHAT UNHELPFUL
Overall, how helpful was summary of patient’s answers to prepare for visit?	Seven Unaffiliated Clinicians	5 (71.4%)	1 (14.3%)	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	3 (60.0%)	2 (40.0%)	0	0
How helpful or unhelpful: General health question	Seven Unaffiliated Clinicians	4 (57.1%)	1 (14.3%)	1 (14.3%)	1 (14.3%)
	Five Clinicians Affiliated with Our Study Team	4 (80.0%)	1 (20.0%)	0	0
How helpful or unhelpful: General quality of life question	Seven Unaffiliated Clinicians	6 (85.7%)	0	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	4 (80.0%)	1 (20.0%)	0	0
How helpful or unhelpful: Carry out daily activities question	Seven Unaffiliated Clinicians	4 (57.1%)	2 (28.6%)	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	3 (60.0%)	2 (40.0%)	0	0
How helpful or unhelpful: Symptom questions	Seven Unaffiliated Clinicians	6 (85.7%)	0	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	4 (80.0%)	1 (20.0%)	0	0
How helpful or unhelpful: Agenda setting (Issues that patient listed to talk about during appointment)	Seven Unaffiliated Clinicians	5 (71.4%)	0	2 (28.6%)	0
	Five Clinicians Affiliated with Our Study Team	3 (60.0%)	2 (40.0%)	0	0
How helpful was PRSM for talking to your patients about symptoms and health conditions they were experiencing?	Seven Unaffiliated Clinicians	3 (42.9%)	3 (42.9%)	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	3 (60.0%)	2 (40.0%)	0	0
How helpful was PRSM for documenting patient’s review of systems?	Seven Unaffiliated Clinicians	6 (85.7%)	0	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	5 (100.0%)	0	0	0
ITEM	AFFILIATED OR UNAFFILIATED CLINICIANS	VERY EASY/EASY	SOMEWHAT EASY	NEITHER DIFFICULT NOR EASY	VERY DIFFICULT/DIFFICULT/SOMEWHAT DIFFICULT
Easy or difficult to understand paper copy of patient’s answers?	Seven Unaffiliated Clinicians	6 (85.7%)	0	0	1 (14.3%)
	Five Clinicians Affiliated with Our Study Team	4 (80.0%)	1 (20.0%)	0	0
ITEM	AFFILIATED OR UNAFFILIATED CLINICIANS	VERY SATISFIED/SATISFIED	SOMEWHAT SATISFIED	NEITHER SATISFIED NOR UNSATISFIED	VERY DISSATISFIED/DISSATISFIED/SOMEWHAT DISSATISFIED
Satisfaction with PRSM System	Seven Unaffiliated Clinicians	5 (71.4%)	1 (14.3%)	1 (14.3%)	0
	Five Clinicians Affiliated with Our Study Team	3 (60.0%)	2 (40.0%)	0	0

Note: PRSM = Patient-Reported Symptom Monitoring (PRSM) System

Out of 39 total patients seen, there were 7 instances where the clinician did not receive the symptom summary before the patient visit (3 breast patients, 2 head and neck, and 2 bone marrow transplant patients), and thus they did not complete an acceptability questionnaire. In one case the printer was not working, and in the other 6 cases the patient did not complete the questionnaire before the clinician was ready to see them. These 7 instances are not included in Table 3.

Most clinicians found the PRO summary to be easy to interpret (83 percent), helpful for communicating with patients (67 percent), and would recommend it to future patients (92 percent) (Table 3). The PRO summary was perceived by clinicians to be most useful for documenting the Review of Symptoms (92 percent). Five clinicians adjusted the treatment plan and, of these, four (80 percent) felt that the PRO summary was “very helpful” or “helpful” in changing the treatment plan (the fifth rated it as “neither helpful nor unhelpful”).

Importantly, most clinicians (92 percent) reported that discussing the PRO summary with their patient during the clinical visit did not change the amount of consultation time (one reported a five-minute decrease and one reported a five-minute increase). Clinicians reported that 83 percent of patients talked with them about symptoms reported on the PRO measure (13 percent did not experience symptoms in the last seven days, and thus symptoms were not relevant to discuss). This left 4 percent (n=2) patient-clinicians dyads who received a hard copy of the PRO summary but did not directly discuss it during the visit.

The seven clinicians unaffiliated with our study team provided several comments: (1) “This would be great for full review of symptoms; add questionnaires such as IPSS [International Prostate Symptom Score Questionnaire], and medication/allergies”; (2) “May take time to familiarize with better”; (3) “I like that patient lists their questions for visit” [agenda setting]; and (4) “My patient put the important symptom in ‘other’ [field] and wrote ‘nothing’ in agenda field. I didn’t notice ‘other’ [field] until after she left, but we had talked about it anyway.”

Comments from the five clinicians who were part of our study team had similar themes: (1) “patient really liked survey”; (2) “needs to be ready and in the door for my review prior to seeing pt.”; and (3) “providers should get primer on the process before starting.”

Discussion

We conducted a two-phase qualitative study to develop a web-based PRO system to collect, store, and summarize cancer patients’ self-reported symptoms and functional status for use during routine cancer care. Our process involved patient and clinician feedback and testing during clinical encounters. This is the first study, to our knowledge, that assesses clinicians’ and patients’ perceptions in real time about the acceptability and utility of discussing PRO measures during routine care. Both cancer patients and clinicians reported that discussing a PRO measure during an outpatient visit was useful, acceptable, and feasible.

Cancer patients reported that it was easy to complete 21 items on a tablet computer in the waiting room. The majority felt that discussing a PRO measure with their clinician during an outpatient visit was helpful in reviewing health issues and would recommend it to other patients. Most patients also chose to set an “agenda” for topics they would like to discuss with their clinician during the visit. Discussing PRO measures with patients may promote patients’ involvement in their own care by communicating their health experiences in real time, and thus may improve care.11,32

Clinicians reported that reviewing a PRO summary with patients did not lengthen consultation time and was most useful for documenting the Review of Symptoms, indicating that PRO measures may help clinicians focus attention on their patients’ most problematic symptoms. Future testing will be necessary to determine when the PRO screener is sufficient for identifying problematic symptoms and when additional modules are necessary that are specific to types of cancer or treatments.

We observed slightly lower satisfaction among clinicians with the PRO system than among patients. Lower clinician satisfaction may have been due to the following reasons: (1) being unfamiliar with how to interpret PRO measures; (2) multiple suggestions for adding disease- and treatment-specific symptoms and better functionality of the system; or (3) for four clinicians who reported that the PRO summary was “somewhat helpful” or “neither helpful nor unhelpful,” they may have had less interest in the project prior to their involvement than had those who indicated it was more useful. We believe these suggestions for improvement indicate moderate to strong interest among approximately two-thirds of the clinicians in our study in continuing to use and improve PRO measures during clinical care. Future research should consider examining trends of perceived acceptability and usefulness as clinicians become more familiar with PRO systems over time. The Diffusion of Innovation Theory41 may be useful for this purpose. Given our small sample size and cross-sectional data collection, we were not able to determine if clinicians reported greater satisfaction over multiple uses.

Enhancing functionality of web-based PRO systems to optimally meet clinicians’ needs is a systematic barrier occurring across health systems. Less than half of web-based PRO systems in cancer care that were identified in a recent systematic review are linked to electronic health records (EHRs), and only 15 percent are linked to billing procedures.18 Therefore, the field would benefit from guidelines for incorporating PRO questionnaires into clinical workflow and decision-making, EHRs, and administrative systems.

PRO systems integrated into routine cancer care may also serve as a foundation for a “learning health care system” where symptoms, treatments, and outcomes from all cancer patients are utilized in improving care rather than from just the few cancer patients who enter a clinical trial.42–43 A state-of-the-art learning health care system would continually assemble clinical data from multiple sources (e.g., all patients attending clinics, registries, EHRs, and clinical trials) to provide the most up-to-date evidence for clinicians and researchers.42–43

Feasibility and Sustainability

Feasibility and sustainability issues will need to be addressed in future iterations. For the purposes of this study, the research assistant logged the patient into the tablet computer with a study ID number, printed out a hard copy of the PRO summary, and then gave copies to the clinician and patient. In future iterations, a front desk staff person may be able to enter the patient’s medical record number and help with “how-to” questions such as how to access the on-screen keyboard. Stukenborg et al.44 also found that palliative care cancer patients requested assistance with a tablet computer. Future research should keep this in mind when designing interfaces and plans to pretest PRO systems with diverse patients.

If patients are given the option to complete PRO measures on a home computer or smartphone prior to the visit, additional programming features and clinical procedures will be needed. For instance, programming flags will need to be in place to ensure that the questions have been completed prior to seeing the clinician and have valid responses, and that missing data is minimized.44 Additionally, procedures for alerting clinicians to immediate issues and for following up with the patient prior to appointments will need to be in place.

This approach requires that patients have access to online patient portals or interactive voice response systems to complete PRO measures prior to visits. Concerns have been raised that web-based PRO systems may not be feasible for underserved populations.46 However, the Pew Internet and American Life annual surveys indicate that 74 percent of adults go online and Internet use among adults ages 60 years and older – and those with a high school education or less – has been rising.47 Mobile technology, in particular, is increasing among underserved populations.47 Over 60 percent of the diverse cancer patients in our study reported having access to the Internet and would be willing to complete PRO measures at home prior to appointments.

Moving forward, as PRO systems become increasingly complex, training clinicians on using and interpreting PRO measures will be critical for future success. For instance, the PROMIS global health measures can be scored by transforming raw scores into T-scores (mean of 50 and standard deviation of 10).39 T-scores may not be intuitive, and training is advised so that clinicians can help patients understand their scores.

Limitations

This study was limited to one large, comprehensive cancer center with small samples of patients and clinicians, although the academic medical center treats patients who are diverse in terms of age, education level, and race (African American). We used purposive sampling within each of these demographic groups to maximize diversity, but further research is needed to determine if results generalize to other academic centers, community hospitals, and patients of other races or Hispanic ethnicity.

Future testing should also expand purposive sampling to include a greater variety of disease stages and treatment types (e.g., mastectomy, nerve-sparing robotic-assisted prostatectomy, hormone therapy). Symptoms are likely to vary across disease stages and treatment groups, and it will be important to determine whether the PRO screening questions have adequate symptom coverage for different groups.

Future testing is also needed to evaluate the psychometric properties of questions adapted from the PRO-CTCAE based on patient feedback. Testing at other medical centers and community practices is warranted to determine the validity and reliability of these PRO items to serve as screeners for problematic symptoms during cancer care. In addition, feasibility testing is encouraged to minimize disruptions to the clinic workflow.

Our study had several strengths, including methodological rigor for qualitative research and PRO evaluation. In Phase I, our multidisciplinary team chose the most informative PRO items for a brief screener, then conducted cognitive testing with cancer patients who were diverse in terms of clinical and demographic characteristics. In Phase II, we assessed both cancer patients’ and clinicians’ perceptions during real-time clinical encounters.

Conclusion

Both cancer patients and clinicians reported during real-time testing that discussing a PRO measure during an outpatient visit was useful, acceptable, and feasible. Real-time symptom reports involve cancer patients in their own care by communicating their health experiences and providing the opportunity to convey what is most important to them to discuss during their visits. Providing health care practitioners with real-time, patient-specific summaries of symptoms may allow clinicians to respond to concerns rapidly and effectively. The current study contributes to best practices for integrating real-time, patient-reported symptom and functional status data into routine cancer care.

Appendix A. Questions Reviewed During Cognitive Interviews and Final Screener Items

DOMAIN	21 ITEMS THAT UNDERWENT COGNITIVE INTERVIEWING	FINAL VERSION OF THE 21-ITEM SCREENER APPLICABLE ACROSS ALL CANCER AND TREATMENT TYPES
Agenda Setting for Visit	“What would you like to talk to nurse or doctor about today?”3	What would you like to talk to nurse or doctor about today?
Insomnia	What was the SEVERITY of your INSOMNIA (INCLUDING DIFFICULTY FALLING ASLEEP, STAYING ASLEEP, OR WAKING UP EARLY) at its WORST?1	Rate your worst insomnia (trouble sleeping).
Decreased Appetite	What was the SEVERITY of your DECREASED APPETITE?1	Rate your worst decreased appetite.
Nausea	What was the SEVERITY of your NAUSEA at its WORST?1	Rate your worst upset stomach.
Pain Frequency	How OFTEN did you have PAIN?1	How often did you have pain?
Pain Intensity	What was the SEVERITY of your PAIN at its WORST?1	Rate your worst pain.
Depression	What was the SEVERITY of your SAD or UNHAPPY FEELINGS at their WORST?1	Rate your worst sad or unhappy feelings.
Fatigue	What was the SEVERITY of your FATIGUE, TIREDNESS, OR LACK OF ENERGY at its WORST?1	Rate your worst tiredness, lack of energy, fatigue.
Headache	What was the SEVERITY of your HEADACHE at its WORST?1	Rate your worst headache.
Anxiety	What was the SEVERITY of your ANXIETY at its WORST?1	Rate your worst anxiety (worrying).
Neuropathy	What was the SEVERITY of your NUMBNESS OR TINGLING IN YOUR HANDS OR FEET at its WORST?1	Rate your worst numbness or tingling in your hands or feet.
Dyspnea	What was the SEVERITY of your SHORTNESS OF BREATH?1	Rate your worst shortness of breath.
Cough	What was the SEVERITY of your COUGH at its WORST?1	Rate your worst cough.
Trouble Concentrating	What was the SEVERITY of your PROBLEMS WITH CONCENTRATION at their WORST?1	Rate your worst problems with concentration.
Diarrhea	How OFTEN did you have LOOSE OR WATERY STOOLS (DIARRHEA)?1	How often did you have loose or watery stools (diarrhea).
Loss of control of urine	How OFTEN did you have LOSS OF CONTROL OF URINE (LEAKAGE)? 1	How often did you have loss of control of urine (leakage).
Rash	Did you have any RASH?1	Did you have a rash? (yes/no)
Other Symptoms Not Listed	Did you have any other symptoms not listed?3	Did you have any other symptoms not listed?
Overall Health	In general, would you say your HEALTH is:2 Excellent, Very Good, Good, Fair, Poor?	In general, would you say your health is: Excellent, Very Good, Good, Fair, Poor?
Quality of Life	In general, would you say your QUALITY OF LIFE is:2 Excellent, Very Good, Good, Fair, Poor?	In general, would you say your quality of life is: Excellent, Very Good, Good, Fair, Poor?
Daily Activities	To what extent are you able to carry out your everyday physical activities such as walking, climbing stairs, carrying groceries, or moving a chair?2	To what extent are you able to carry out your everyday physical activities such as walking, climbing stairs, carrying groceries, or moving a chair?

Notes:

¹PRO-CTCAE,

²PROMIS Global Health Questionnaire,

³Question Written by Authors.

Appendix B. Cognitive Interview Results

DOMAIN	ORIGINAL WORDING OF QUESTION	NUMBER REPORTING DIFFICULTY ANSWERING QUESTION	COMMENTS FROM ROUND 1 PARTICIPANTS (n=25)	REVISED WORDING OF QUESTION	COMMENTS FROM ROUND 2 PARTICIPANTS (n=10 ADDITIONAL PATIENTS)
Severity Stem	What was the severity of (your) [symptom] at its worst?	10/25	“Do you want to know how bad it is or how often it occurred?”	Rate your worst [symptom].	Acceptable as rewritten
			“Difficult to compute ‘at its worst.’ It often starts one way and ends another. Do you mean on a specific day?”
			“Tough when you have a lot of one symptom, but it isn’t that bad.” “Also difficult when you don’t have a symptom to report.”
			“At its worst – as compared to what?”
			“I didn’t read far enough to get to the ‘at its worst’ part so I didn’t factor that into my answers.”
Fatigue	Fatigue, tiredness, or lack of energy	3/25	Difficulty reading or understanding the word “fatigue.”	Tiredness, lack of energy, or fatigue	Acceptable as rewritten
Anxiety	Anxiety	2/25	Difficulty reading word “anxiety” or trouble understanding concept	Anxiety (worrying)	Acceptable as rewritten
Bladder Control	Loss of control of urine (leakage)	4/25	Difficulty understanding what control of urine meant	Leak urine/ urine leakage	Acceptable as rewritten