Acceptability of Routine Evaluations Using Patient‐Reported Outcomes of Common Terminology Criteria for Adverse Events and Other Patient‐Reported Symptom Outcome Tools in Cancer Outpatients: Princess Margaret Cancer Centre Experience

regorafenib dosing in patients with metastatic or recurrent gastrointestinal stromal tumors after failure of imatinib and sunitinib.

Abstract

Background.

Recent studies have demonstrated improved outcomes with real‐time patient‐reported outcome questionnaires (PRO questionnaires) using questions adapted for patient use from the National Cancer Institute's Common Terminology Criteria for Adverse Events (CTCAE). Outside of the clinical trial setting, limited information exists on factors affecting the completion of PRO questionnaires in routine practice. The primary aim of this prospective cross‐sectional study was to evaluate patient willingness to complete PRO questionnaires on a regular basis and to better understand responder biases to improve patient feedback.

Materials and Methods.

Patients performing PRO‐CTCAE toxicity and symptom PRO questionnaires in oncology clinics at Princess Margaret Cancer Centre from 2013 to 2016 were assessed for their willingness to complete PRO questionnaires using a nine‐item, tablet‐based acceptability survey. Patient‐reported characteristics (i.e., age, sex, language, marital status, education, occupation, etc.), cancer type, treatment modalities, and health metrics (i.e., Eastern Cooperative Oncology Group) were also collected. Characteristics were evaluated by logistic regression (odds ratios [OR]) using the primary outcome with prespecified levels of significance for univariate (p ≤ .10), and additional multivariate (p ≤ .05) testing.

Results.

A total of 1,792 patients (median age 60 years; range 18–97) with various cancer diagnoses were assessed. A greater proportion of female (56%) and white (74%) respondents with an annual household income of <$100,000 (69%) participated. More than half (58%) of respondents were willing to complete PRO questionnaires at every clinic visit, and a high proportion (77%) found utility in reporting physical and emotional feelings to clinicians using PRO questionnaires. In general, patients did not find that PRO questionnaires made clinic visits more difficult (93%). In uni‐ and multivariable testing, patients were more willing to complete sleep‐ and fatigue‐related PRO questionnaires relative to chemotoxicity‐based PRO questionnaires (OR 1.52; p = .012). Patients aged 40–65 versus 18–40 years were also more likely to report high PRO questionnaire acceptability (OR 1.49; p = .025). Additional patient characteristics such as white ethnicity (OR 1.76), Canada as country of birth (OR 1.66), and English language (OR 2.15) relative to other had higher acceptability on uni‐ (p < .001) and multivariable (p < .001) analyses. Patients reporting treatment intent as palliative (OR 0.69; p = .0013) or hematological (OR 0.73; p = .027) were less likely to report high PRO questionnaire acceptability on univariable analysis; however, only palliative patients (OR 0.72) maintained this effect on multivariable testing (p = .012). Patients reporting higher health utility scores (per change in .05) also had significantly increased PRO questionnaire acceptability in uni‐ (OR 1.06; p < .001) and multivariable (OR 1.05; p = .008) analyses. No significant differences in PRO questionnaire acceptability were seen between cancer types, education level, household income, employment status, or treatment modality.

Conclusion.

Routine assessment using PRO questionnaires is associated with moderate acceptability by patients with cancer. Specific patient characteristics are associated with higher completion willingness. Additional research is necessary to identify factors associated with low acceptability of PRO questionnaires and to develop site‐, ethnicity‐, and treatment‐specific instruments to assess the value of PRO questionnaires for symptom monitoring in clinical practice.

Implications for Practice.

This study will help to identify the clinical, demographic, and survey characteristics associated with willingness to complete patient‐reported outcome questionnaires regularly in the cancer outpatient setting.

Introduction

Patient‐reported outcomes are real‐time evaluations of patient well‐being that capture personal health metrics such as wellness, quality of life, and functional status while on treatment(s) or in follow‐up [1]. Significant efforts have been put forth in developing valid and accurate patient‐reported outcome questionnaires (PRO questionnaires) and incorporating these tools into both the clinical and research setting(s) [2]. PRO questionnaires have shown prognostic capacity beyond standard measures in randomized studies [3] and have demonstrated the ability to increase diagnostic accuracy and reduce bias in clinician‐reported prognostic models [4]. Studies have shown discordance between medical records and patients’ self‐reported outcomes [5]. For example, unidimensional reporting of patient toxicity by health care providers may underestimate the incidence, severity, and burden of patients’ symptoms [6] with lower sensitivity and specificity compared with patient self‐reporting [5], [7]. In addition, sensitive topics such as sexual health are commonly underreported by physicians [8], [9] and are more likely to be elicited by electronic PRO questionnaire assessments [10]. Inter‐reporter variability between physicians using the Common Terminology Criteria for Adverse Events version 4.0 (CTCAE) classification system can also introduce bias into toxicity reporting [11].

Integration of patient symptom self‐reporting in cancer care has shown distinct advantages including reduced resource use, increased treatment adherence, improved quality of life, and survival benefits [12], [13], [14]. However, these outcomes have only been achieved through high PRO questionnaire completion rates, making barriers to PRO questionnaire completion an active area of investigation. Historical evidence has shown that variances in survey length is a main contributor to responder fatigue and leads to lower survey completion rates [15]. Moreover, PRO questionnaire completion is dependent on patient‐specific factors including survey acceptability, completion willingness, applicability of results, assessed health metrics, and practicality of administration, in addition to many others. Patients with cancer may have a different degree of motivation with respect to PRO questionnaires compared with other patient groups, potentially being influenced by the desire for optimal treatment and clinician communication when faced with a potentially life‐limiting disease. Whether PRO questionnaire completion willingness is predominantly a patient‐, questionnaire‐, or a codependent phenomenon needs further clarification. This study aims to identify the clinical, demographic, and survey characteristics associated with willingness to complete PRO questionnaires regularly in the cancer outpatient setting.

Materials and Methods

This prospective, cross‐sectional study evaluated oncology outpatients at Princess Margaret (PM) Cancer Centre from July 2013 to August 2016 who completed one of six different validated PRO questionnaires in addition to an institutionally developed acceptability questionnaire. Patients ≥18 years of age with a confirmed cancer diagnosis (excluding central nervous system malignancies) who were able to communicate in English and cognitively well enough to consent to the study were eligible. Research coordinators approached patients by convenience sampling in oncology outpatient waiting rooms for voluntary participation and written consent. The process also included consent to access personal electronic medical records to collect relevant clinical information. Patients typically completed PRO questionnaires and study surveys after completing the Edmonton Symptom Assessment Score (ESAS) administered as part of routine clinical care at PM. This study was approved by the University Health Network research ethics board.

Between July 2013 and August 2016, patients completed one of six different validated PRO surveys. The specific questionnaire depended on the clinic and time of year of their clinic visit. The total number of questions each patient answered ranged from 43 to 86 questions (including acceptability questions) depending on the specific survey. Surveys were administered in the outpatient clinic waiting rooms at the PM (Breast, Gastrointestinal, Genitourinary, Gynecological, Head and Neck, Skin, Thoracic, and Hematological malignancies), except for the chemotoxicity survey, which was administered in the Chemotherapy Day care unit. The time period during which each PRO measure survey was administered is as follows: Sleep and Fatigue Survey 1: July 2013 to March 2014; Sleep and Fatigue Survey 2: June to August 2014; Chemotoxicity Survey: April 2014 to February 2015; Pain and Physical Function Survey: March to April 2015; Pain Survey: April to July 2015; Physical Function Survey: April 2015 to August 2016.

One of six different validated PRO questionnaires (supplemental online Appendix 1) were administered to eligible patients: (a) chemotherapy toxicity (PRO‐CTCAE version 1.0 [68 questions]), (b) sleep and fatigue 1 (insomnia severity index [ISI] and Fatigue Scale [FACIT‐Fatigue] [43 questions] + health utility score [HUS] questions, EQ‐5D‐3L [7 questions]), (c) sleep and fatigue 2 (ISI plus FACIT‐Fatigue [36 questions] with no EQ‐5D‐3L), (d) pain (Brief Pain Inventory [BPI] + EQ‐5D‐3L [47 questions total]), (e) physical function (World Health Organization Disability Assessment Schedule [WHODAS] + Health Assessment Questionnaire, Disability Index [HAQ‐DI] + PRO‐Eastern Cooperative Oncology Group [ECOG] + EQ‐5D‐3L [73 questions total]), and a (f) combination of pain and physical function (BPI + WHODAS + HAQ‐DI + EQ‐5D‐3L [86 questions total]). All surveys included unique PRO questionnaire elements along with common, overlapping questions including demographic variables such as sex, age, marital status, education, occupation, employment status, language, ethnicity, and country of birth. Some surveys included additional questions related to performance status (PS/ECOG) and an evaluation of individual health over the last month (five‐level scale from poor to excellent). Patient health was further evaluated using the seven‐question EQ‐5D‐3L questionnaire (five domain/functional status questions plus two global health questions scored on a visual analog scale [VAS; 0–100]) generating a health utility score, calculated from the EQ‐5D‐3L descriptive system using Canadian preference weighting [16]. Clinical data were collected from electronic patient records and included tumor type, treatment intent, and type of treatment.

A 10‐item acceptability questionnaire was administered at the end of each PRO questionnaire, including the primary outcome question “Would you be willing to complete such [PRO questionnaire] surveys at every clinical visit?” Secondary outcomes evaluated the utility of conveying patient physical and emotional well‐being to treating clinicians, or PRO effects on increasing difficulty of clinic visits. Six of the remaining questions were treated as exploratory outcomes (Fig. 1). One question was used as a response validation check to another question (redundancy) and was removed from the analysis because of collinearity. Responses were recorded in a five‐point Likert scale format, anchored by either strong disagreement or agreement with the statement, and a neutral/unsure category in the middle. If patients aborted their PRO questionnaires early, they were still encouraged to complete the 10‐item acceptability questionnaire. All six PRO questionnaires were administered under a uniform protocol for eligibility, approach, consenting, PRO questionnaire administration, and data collection.

Figure 1.

Acceptability questionnaire responses.

Statistical Analysis

All responses (including those with incomplete surveys) were analyzed if at least one acceptability question had been answered. Clinicodemographic information and PRO questionnaire characteristics were independently assessed. Acceptability question outcomes using five‐point Likert scale responses were dichotomized to collapse “agree/strongly, agree” and “disagree/strongly, disagree” into a binary Agree/Disagree value. The “unsure” response category was excluded from agree or disagree response calculations. Descriptive statistics were used to characterize the sample cohort. Nonparametric Kruskal‐Wallis tests were applied for continuous variables’ group comparisons, and Fisher's exact tests were used for categorical variable comparisons. Logistic regression (odds ratios [OR]) were used for willingness to complete PRO questionnaires modelling. Variables with a significance level p ≤ .10 in univariable models were selected for a multivariable logistic regression model using backward selection. A significance level of p ≤ .05 was used to retain variables for the final model. Analyses were performed using SAS 9.4 (SAS Institute, Cary, NC) and R 3.1.2. Correlation between variables was evaluated using Spearman rank correlation tests.

Results

Patient Characteristics and Demographics

Of 2,214 patients approached, 1,792 (81%) completed PRO surveys. The median age was 59.9 years (range 18–97). A higher proportion of patients reported as being female (56%), of white ethnicity (74%), being married (67%), speaking English as their primary language (83%), and having a performance status/ECOG of 0–1 (81%; Table 1). Most patients were treated with curative intent (50%) relative to palliative intent (32%). Treatment for hematologic malignancies made up a smaller proportion (18%). Patient classification by cancer diagnoses were similar, with overall ratings of health (VAS, HUS, self‐described) being generally high (Table 2). A majority of patients (96%) reported at least high school education or higher as well as an annual household income of $ < 100,000 (69%). Employment status showed a similar number of patients continued to work (47%) as to not work (50%). Average survey completion time was 12.4 minutes (range 0.5–56.6), with a higher proportion completing pain and/or physical function PRO questionnaires (48%) compared with chemotoxicity (24%) or sleep and fatigue (28%).

Table 1. Characteristics of study population.

Abbreviation: ECOG, Eastern Cooperative Oncology Group.

Table 2. Patient‐reported outcome surveys.

Abbreviation: HUS, health utility score.

Primary and Secondary Outcomes: Completion Willingness and Perception

Evaluation of the primary outcome showed a majority of patients were willing to complete PRO surveys at every clinic visit (58%). Secondary outcome questions showed that most patients found utility in telling clinicians how they felt physically and emotionally (77%) and did not feel that survey completion made clinic visits more difficult (93%; Fig. 1). Similarly, a majority of people did not find PRO surveys upsetting or distressful (96%) or time consuming (56%); however, half (50%) of patients found PRO survey questions irrelevant to their personal situation. With regard to electronic tablet‐based administration, patients were generally satisfied (89%) completing surveys by this method. Although just over half (51%) felt that survey information should be kept in their personal notes, the majority (72%) did not want to see a printout of their survey results.

Logistic Regression

In uni‐ and multivariable analyses (Table 3), factors associated with the primary outcome (willingness to complete PRO questionnaire regularly) were evaluated. PRO surveys evaluating sleep and fatigue relative to chemotoxicity had the highest statistically significant acceptability rate (OR 1.47) in both uni‐ (p = .004) and multivariable (p = .012) analyses. No correlation was seen with length of time taken to complete PRO questionnaires and completion willingness (OR 0.99; p = .20). Acceptability by age group showed that higher proportions of patients aged 40–65 years (OR 1.45) relative to 18–40‐years were willing to complete PRO surveys. This was significant in both uni‐ (p = .012) and multivariable (OR 1.49; p = .025) analyses. Linked variables such as being Canadian born (OR 1.66), speaking English as a primary language (OR 2.15), and being of white ethnicity (OR 1.73) relative to other were associated with higher willingness to complete PRO questionnaires (p < .001). Acceptability to complete the PRO questionnaires across tumor sites was similar; there were no statistical differences between hematological and nonhematological malignancies. Among nonhematological malignancies, surprisingly, patients with genitourinary cancers showed lower willingness to complete the PRO questionnaires (OR 1.08; p = .68). Patients with genitourinary cancer also self‐reported poor health, 18% versus only 4% in patients with other malignancies. Only 3% reported excellent health, versus 11% in other malignancies. As ethnicity, country of birth, and language are highly correlated, only one (ethnicity; OR 1.76; p < .001) was selected for entry into the multivariable regression model (Table 2). In further exploratory analyses, no differences in acceptability were seen among the Asian, black/African Canadian, and other nonwhite ethnic subgroups (data not shown).

Table 3. Uni‐ and multivariate analysis of factors associated with willingness to complete patient‐reported outcome surveys at every clinic visit.

Bolded p values are statistically significant.

Abbreviations: CI, confidence interval; ECOG PS, Eastern Cooperative Oncology Group performance status; HUS, health utility score; OR, odds ratio; VAS, visual analog scale.

Evaluation of PRO questionnaire acceptability relative to curative treatment intent showed that patients reporting hematological (OR 0.73; p = .027) or palliative (OR 0.69; p = .0013) disease management were less willing to complete PRO surveys, although only palliative‐intent treatment (OR 0.72; p = .012) remained statistically significant in multivariate analyses (Table 3). Evaluation of patient health via linked variables (ECOG, HUS, VAS, Health Q) generally demonstrated lower acceptability with declining health, which was further evaluated through stratified ECOG (3 relative to 0, OR 0.31; p < .001) and Health question score from 5 (OR 3.43) to 1 (reference) with a progressive trend for Health Q (global p = .0018; Table 3; Fig. 2). People with a .05 change in HUS (OR 1.04; p < .001) or five‐point change in health utility VAS (OR 1.04; p < .001) also had higher PRO questionnaire acceptability. HUS‐acceptability correlation (OR 1.05; p < .008) was maintained in multivariate analyses. Evaluation of sex, marital status, education, household income, employment status, cancer diagnoses, and treatment type with the primary outcome did not reach statistical significance.

Figure 2.

Subgroups with proportions of high acceptability for completing surveys at every visit (proportion and 95% confidence interval).

Abbreviation: PROQ, patient‐reported outcome questionnaire.

Discussion

Patient‐reported outcomes are increasingly being incorporated into oncology clinical trial designs [17] and routine clinical care [2] after their demonstrated efficacy in improving clinical outcomes [12], [13], [14]. This study examined patients’ acceptability to complete PRO questionnaires at each oncology visit. Patient preference for the shorter sleep and fatigue PRO questionnaires (average 40 questions; p = .004) rather than the longer pain and physical functioning (86 questions) PRO questionnaires (p = .3) is consistent with that seen in a prior meta‐analysis by Rolstad et al. [15]. Interestingly, time taken (minutes) to complete the PRO questionnaires was not significantly correlated with acceptability, suggesting that question content or subject relevance may be more important than total time taken to complete surveys.

Middle age (40–65 years) and ethnocultural differences including white ethnicity, being Canadian born, and speaking English as a primary language were also shown to have high rates of PRO questionnaire acceptability relative to other (each p < .001). In a large retrospective study by Hutchings et al., young age, nonwhite demographics, and poor socioeconomic status were identified as factors associated with poor response rates to PRO questionnaires in the setting of joint replacement surgery in the U.K. [18]. Ethnocultural influences on willingness to complete PRO questionnaires in this study were not adequately explained by linguistic barriers alone. All patients had enough English language skills to read the consent form and complete surveys, although unaccounted cultural values may have additionally influenced responses. For example, patients of non‐English‐speaking background may have been reluctant to participate in health surveys that were not conducted in their primary language. This highlights the importance of a transparent translation process to education levels reflective of the naturalized country/culture [19]. Moreover, any translated materials require the re‐establishment of face, content, and construct validity in addition to reliability.

Patients with different cancer types have a similar rate of willingness to complete the PRO questionnaires, except patients at genitourinary malignancies’ clinic. They were less willing to complete the PRO questionnaires, compared with other tumor sites. It is not clear why, and it has never been reported in previous studies. A high proportion (32%) of respondents in this study received palliative‐intent treatment. This may be a source of selection bias, as patients treated with palliative intent in this study were less willing to complete PRO surveys regularly (OR 0.6). Interestingly, a study by Pessin et al. [20] did not find completion of patient‐reported surveys as a significant burden (75%) at the end of life, but they did identify that question content—such as those discussing end‐of‐life issues or worsening of symptoms—was more likely to cause patient distress. Whereas question distress by treatment intent (palliative, curative, hematological) was not specifically evaluated in this study, overall rates of distress on acceptability‐PRO questionnaire were low (4%). Nonresponse to PRO tools has also been associated with worse clinical outcome [21]; however, this again may reflect patient health or performance status. Educating patients regarding the importance of self‐reported outcomes may alter their perception of utility and improve willingness to complete questionnaires.

Limitations

This study possesses limitations. First, PROQ questionnaires were primarily presented as a research study, rather than a tool to inform clinicians of patients’ experience. As survey utility is likely a significant motivator in patient willingness to completing PRO questionnaire surveys [22], this may have resulted in less willingness to engage with PRO questionnaires than if it was presented as a routine clinical tool (i.e., ESAS). Second, patient fatigue may have also contributed to survey completeness or responder accuracy, as PRO questionnaires were administered after PM standard‐of‐care ESAS evaluations. Third, responder and sampling bias may also limit the generalizability of these results, as only patients treated at PM (the largest tertiary oncology center in Canada) who were willing to complete PRO tools have been measured. This may represent a patient group who were more motivated or treatment seeking and thus more willing to complete PROs than those in community‐based oncology practices. Given the requirements of the research ethics board, we understandably have minimal information on patients who refused participation in studies. Most patients who declined participation (422 people) have self‐reported poorer health outcomes. We cannot trace ESAS (DART) data completion rates for those patients in order to confirm correlation with willingness to complete the PRO questionnaires, which may have resulted in an overestimation of willingness to complete regular surveys.

Conclusion

Overall, routine assessment of patient well‐being using PRO questionnaires is associated with moderate acceptability by patients with cancer. Future directions should attempt to identify factors associated with low acceptability of PRO questionnaires and to develop site‐, ethnicity‐, and treatment‐specific instruments to assess the value of PRO questionnaires for symptom monitoring in clinical practice.

Author Contributions

Conception/design: Hamzeh Albaba, Tristan A. Barnes, M. Catherine Brown, Nicole Mittmann, Wei Xu, Doris Howell, Geoffrey Liu

Provision of study material or patients: M. Catherine Brown, Hiten Naik, Mindy Liang, Andrea Perez‐Cosio, Lawson Eng, Wei Xu

Collection and/or assembly of data: M. Catherine Brown, Hiten Naik, Mindy Liang, Andrea Perez‐Cosio, Lawson Eng, Wei Xu

Data analysis and interpretation: Hamzeh Albaba, Tristan A. Barnes, Zachary Veitch, Sharara Shakik, Susie Su, Wei Xu

Manuscript writing: Hamzeh Albaba, Tristan A. Barnes, Zachary Veitch, Doris Howell, Geoffrey Liu

Final approval of manuscript: Hamzeh Albaba, Tristan A. Barnes, Zachary Veitch, M. Catherine Brown, Sharara Shakik, Susie Su, Hiten Naik, Tian Wang, Mindy Liang, Andrea Perez‐Cosio, Lawson Eng, Nicole Mittmann, Wei Xu, Doris Howell, Geoffrey Liu

Disclosures

The authors indicated no financial relationships.