Feasibility of Implementing the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events in a Multicenter Trial: NCCTG N1048

Abstract

Purpose

The US National Cancer Institute (NCI) Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) was developed to enable patient reporting of symptomatic adverse events in oncology clinical research. This study was designed to assess the feasibility and resource requirements associated with implementing PRO-CTCAE in a multicenter trial.

Methods

Patients with locally advanced rectal cancer enrolled in the National Cancer Institute–sponsored North Central Cancer Treatment Group (Alliance) Preoperative Radiation or Selective Preoperative Radiation and Evaluation before Chemotherapy and Total Mesorectal Excision trial were asked to self-report 30 PRO-CTCAE items weekly from home during preoperative therapy, and every 6 months after surgery, via either the Web or an automated telephone system. If participants did not self-report within 3 days, a central coordinator called them to complete the items. Compliance was defined as the proportion of participants who completed PRO-CTCAE assessments at expected time points.

Results

The prespecified PRO-CTCAE analysis was conducted after the 500th patient completed the 6-month follow-up (median age, 56 years; 33% female; 12% nonwhite; 43% high school education or less; 5% Spanish speaking), across 165 sites. PRO-CTCAE was reported by participants at 4,491 of 4,882 expected preoperative time points (92.0% compliance), of which 3,771 (77.2%) were self-reported by participants and 720 (14.7%) were collected via central coordinator backup. Compliance at 6-month post-treatment follow-up was 333 of 468 (71.2%), with 122 (26.1%) via backup. Site research associates spent a median of 15 minutes on PRO-CTCAE work for each patient visit. Work by a central coordinator required a 50% time commitment.

Conclusion

Home-based reporting of PRO-CTCAE in a multicenter trial is feasible, with high patient compliance and low site administrative requirements. PRO-CTCAE data capture is improved through centralized backup calls.

INTRODUCTION

The US National Cancer Institute (NCI) Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) was developed to enable patient self-reporting of symptomatic toxicities in oncology clinical trials.¹ The PRO-CTCAE is a companion to the CTCAE,² which is the longstanding lexicon of adverse events (AEs) widely used in cancer clinical trials for AE documentation.

PRO-CTCAE consists of a library of 124 items that measure 78 symptomatic AEs and is publicly available from the NCI.³ For each AE, up to three individual items evaluate its frequency, severity, and interference with daily activities (Appendix Table A1, online only). PRO-CTCAE items salient in a given clinical trial are selected from the item library and assembled by investigators into a custom survey.^4,5 Measurement properties of PRO-CTCAE items including validity, test-retest reliability, sensitivity to change over time, and equivalence of responses captured using different modes of administration (Web, paper, or automated telephone system) have been demonstrated previously.^6-8

Unlike traditional quality-of-life assessments in clinical trials, in which surveys are administered at a limited number of time points, PRO-CTCAE is designed to be administered more frequently to capture the full symptomatic AE profile of a treatment under investigation (eg, weekly during active treatment and periodically after treatment). Previously, we evaluated the feasibility of weekly completion of PRO-CTCAE in the NRG Oncology 1012 lung cancer chemoradiation trial⁴ and found that 86% of patients were willing and able to self-report PRO-CTCAE in clinic waiting rooms at expected time points during active treatment. Patients completed PRO-CTCAE questions on tablet computers brought to them by site staff at treatment visits. A limitation of this study was the dependence on patients being present in clinic to complete surveys. Such an approach is feasible when patients are frequently returning to clinic such as during chemoradiation, but is difficult to implement with patient populations that are seen in clinic infrequently; in-clinic reporting requires access to tablet computers in clinic waiting areas as well as dedicated site support staff. Indeed, a common reason for missed surveys in NRG 1012 was that site staff forgot to bring tablets to patients during clinic visits. Thus, a system that enables patients to self-report electronically between visits is desirable. Additional advantages include the fact that participants can receive electronic reminders to complete surveys and that dependence on site staff may be reduced. The aim of the current study was to examine the feasibility and resource requirements of weekly remote self-reporting using PRO-CTCAE by patients enrolled in a multicenter clinical trial.

METHODS

Participants

Patients with locally advanced rectal cancer enrolling in the US National Clinical Trials Network phase II/III multicenter randomized trial, North Central Cancer Treatment Group N1048 (Preoperative Radiation or Selective Preoperative Radiation and Evaluation before Chemotherapy and Total Mesorectal Excision [PROSPECT]), were invited to participate in a correlative study to evaluate the feasibility, acceptability, and resources required to implement PRO-CTCAE in a multisite cancer therapy trial. The protocol for PROSPECT, including this correlative study, was approved by the institutional review boards of all participating institutions, and all participants provided written informed consent. The North Central Cancer Treatment Group is now part of the Alliance for Clinical Trials in Oncology.

Participants in PROSPECT were randomly assigned to one of two arms. In arm 1, patients received 5.5 weeks of radiation with chemotherapy (5-fluorouracil or capecitabine), followed by surgical excision and then postoperative chemotherapy. In arm 2, patients received 12 weeks of preoperative chemotherapy (5-fluorouracil, leucovorin, and oxaliplatin); those with tumor regression ≥ 20% on the basis of imaging and proctoscopy underwent surgical excision followed by adjuvant chemotherapy; those with lesser response underwent the arm 1 sequence of treatment including chemoradiation, excision, and postoperative chemotherapy. PROSPECT was designed as a seamless, noninferiority phase II/III trial, with the primary end point of phase II being pelvic R0 resection rate and time to local recurrence, and that of phase III being disease-free survival and time to local recurrence.

Accrual to PROSPECT is ongoing, with a target enrollment of 1,140 patients in the United States, Canada, and Switzerland. The current analysis of PRO-CTCAE was preplanned, with Data and Safety Monitoring Board approval, to occur when the 500th patient completed 6 months of post-treatment follow-up. A comparison of PRO-CTCAE reports between study arms will be conducted at the time of trial completion.

PRO-CTCAE Survey

All North American English- and Spanish-speaking participants in PROSPECT were asked to self-report 30 PRO-CTCAE items representing 15 discrete symptomatic AEs (Appendix Table A2, online only) at baseline and weekly from home during preoperative treatment, and then every 6 months after surgery for 3 years. These items were selected by PROSPECT investigators on the basis of expected symptomatic toxicities related to trial therapies and on previously identified prevalent symptoms among patients with cancer undergoing treatment. The study had a recall period for PRO-CTCAE of 7 days.⁴

PRO-CTCAE items were administered as an electronic survey using software hosted at the NCI. At baseline, participants chose to complete the survey in English or Spanish via the Web or via an automated telephone (interactive voice response) system. Participants selected a preferred time and day of the week to receive an e-mail or automated telephone reminder to complete their scheduled survey. If they did not complete the survey, they would receive follow-up automated reminders each of the two subsequent days. Then, if they had still not responded, a central coordinator would call the participant to administer the survey by phone as backup data collection. During preoperative therapy, when surveys were administered weekly, the central coordinator had until the next survey was scheduled to complete backup data collection to avoid overlapping assessment periods. During the postoperative period, when surveys were administered every 6 months, the coordinator initially had up to 1 month to complete backup data collection. This was lengthened to 3 months to increase compliance when a lower than expected compliance rate was observed in the first 306 patients.

Staff PRO-CTCAE Training and Technical Support

The central PRO-CTCAE coordinator was responsible for training clinical research associates (CRAs) at all participating sites. This entailed a standardized 35-minute Webinar that taught CRAs how to register patients into the PRO-CTCAE software system and how to educate patients to self-report AEs using the electronic system. The central coordinator offered refresher orientations as needed (eg, for changes in CRA personnel), monitored data completeness, and was available to address technical questions or problems experienced by sites or patients. The coordinator also directly contacted patients who did not complete PRO-CTCAE surveys within the 2-day response window.

Participant PRO-CTCAE Training

After informed consent and before the baseline survey completion, site CRAs provided participants with information about completing PRO-CTCAE items from home via their selected interface (Web or telephone). Patients were reminded not to rely on the PRO-CTCAE system as a mechanism to inform clinicians about their symptoms and were instructed to communicate directly with their nurse or treating physician about any symptoms or other issues of concern.

Staff Effort and Feedback

CRAs at sites that enrolled participants were surveyed to gauge the amount of time required to use the PRO-CTCAE system and to obtain feedback about their experiences with the system. Semistructured interviews were conducted by telephone with nine randomly selected site CRAs after they had had 6 months of experience with the system.

Statistical Analysis

Data between arms were pooled, per requirements of the Data and Safety and Monitoring Board, to preserve the integrity of primary efficacy in this ongoing clinical trial. PRO-CTCAE compliance was calculated as the proportion of participants who completed PRO-CTCAE assessments at expected time points. Overall compliance at the trial level was evaluated by pooling across all time points (ie, baseline, all weekly preoperative time points, and 6-month postoperative follow-up), as well as at each time point individually. Compliance was also calculated at the individual participant level. Reasons for missed PRO-CTCAE assessments were collected using a standardized form for those patients who were reachable by the central PRO-CTCAE coordinator. Baseline patient characteristics were described using medians and ranges for continuous variables, and frequencies and relative frequencies for categorical variables. The impact of baseline patient characteristics on individual compliance (< 85% v ≥ 85%) and choice of Web versus automated telephone was investigated using univariate logistic regression. To assess whether patients continued to provide self-reported AE data when experiencing severe toxicities, the timing of missed PRO-CTCAE assessments relative to expedited clinician-reported CTCAE grade 4 toxicities and relative to hospitalizations was assessed descriptively. Statistical analyses were conducted by the Alliance Statistics and Data Center.

RESULTS

Participants

Between August 2012 and May 2016, the 557 North American participants in PROSPECT were approached to participate in this correlative study, of which 49 withdrew consent or died before treatment initiation, and eight spoke a language other than English or Spanish, yielding a total of 500 evaluable participants for this analysis. Preferred languages for patients who were ineligible because of language included Bengali, Cantonese, Japanese, Mandarin, Russian, and Urdu. Characteristics at baseline for the 500 PRO-CTCAE subjects pooled across the two treatment arms are listed in Table 1. Participants were a median of 56 years of age (range, 19 to 84 years), 163 of 500 (32.6%) were female, 440 of 500 (88.0%) were white, and 214 of 500 (42.8%) had completed high school education or less. Twenty-four participants (4.8%) selected Spanish as their preferred language for PRO-CTCAE completion, and the remainder chose English.

Table 1.

Baseline Patient Characteristics (N = 500)

The study protocol was approved at 435 sites across the United States and Canada. CRAs from each site received training in the use of the PRO-CTCAE software system. Participants were enrolled at 165 of these sites. Participants completed PRO-CTCAE surveys using their own computing devices or telephones; no hardware was provided to sites or patients for this study.

Compliance

During preoperative therapy, participants were expected to complete a PRO-CTCAE assessment at 4,882 discrete time points (ie, time points at which patients were alive and receiving preoperative treatment). Participants completed a total of 4,491 PRO-CTCAE surveys, yielding an overall compliance rate of 92.0% (Fig 1). Notably, the rate of compliance by patient electronic self-report alone was 3,771 of 4,882 (77.2%), with coordinator backup calls recovering an additional 720 of 4,882 (14.7%). Therefore, of the 1,111 surveys that patients did not complete on their own, 720 (64.8%) were recoverable through coordinator backup data collection. Compliance rates were durable over time, with slightly lower compliance at weeks 11 to 12 when patients were coming off active treatment. Two thirds of participants (337 of 500) completed all their expected surveys, whereas 90% (450 of 500) completed at least 70% of expected surveys, and only two patients completed no expected surveys (Fig 2). Lower compliance was associated with worse Eastern Cooperative Oncology Group performance status (P = .03), lower educational level (P = .03), and Hispanic or Latino ethnicity (P = .01).

Fig 1.

Proportion of participants completing the expected Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) survey at baseline, and at each week of preoperative therapy, and at 6-month post-treatment follow-up. Postop, Postoperative.

Fig 2.

Proportion of participants competing various proportions of expected Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) surveys during treatment (N = 500). Two thirds of patients (337 of 500) completed all expected surveys, whereas 90% (450 of 500) completed > 70% of expected surveys, and only two completed no expected surveys.

Among the 391 instances when patient responses were not captured either through self-report or coordinator backup data collection before or during preoperative treatment, 23 (5.9%) occurred because participants did not want to talk with the PRO-CTCAE coordinator (hung up the phone); 16 (4.1%) were caused by technical error with the PRO-CTCAE system by staff; nine (2.3%) were because of patient hospitalization; seven (1.8%) were because the patient felt overwhelmed and withdrew from this correlative study; and seven (1.8%) were because the patient did not want to use cell phone minutes. The remaining 329 of 391 causes (84%) were unknown, because the patient was not reachable by telephone despite attempts by the central coordinator before the next scheduled survey.

At the 6-month postoperative follow-up time point, 468 patients were still alive and on study; surveys were completed by 333 of the 468 (71.2% overall compliance), with 211 of the 468 (45.1%) self-reported and 122 of the 468 (26.1%) recovered by coordinator backup calls (Fig 1). Compliance increased from 210 of 306 (68.6%) to 123 of 173 (75.9%) when the allowable window for backup data collection was lengthened from 1 to 3 months. Reasons for missing data among the 135 instances when patient responses were not captured included the following: eight participants (5.9%) did not want to talk with the PRO-CTCAE coordinator; five patients (3.7%) were hospitalized; one patient (0.7%) felt too ill to complete the survey; and one patient (0.7%) felt overwhelmed; 120 instances of missing data (88.9%) were for unknown reasons.

PRO-CTCAE Reporting Proximate to CTCAE Grade 4 Toxicities

In an analysis of patient self-reporting compliance proximate to the 26 expedited clinician-reported CTCAE grade 4 toxicities during preoperative treatment (Appendix Table A3, online only), patients were able to report PRO-CTCAE within 7 days before or 7 days after 18 (69.2%) of these toxicities (10 [55.5%] within 7 days before only, and eight [44.4%] within 7 days both before and after). There were 40 hospitalizations during preoperative treatment. Participants completed surveys within 7 days before or 7 days after 27 (67.5%) of these hospitalizations (15 [55.6%] within 7 days both before and after, nine [33.3%] within 7 days before only, and three [11.1%] within 7 days after only).

Mode of Survey Completion

Among the 500 participants, 316 (63.2%) elected at baseline to complete PRO-CTCAE surveys via the Web and 184 (36.8%) by automated telephone. Patients who selected the Web were significantly younger (mean age, 55.1 years v 59.7years, P < .001), more highly educated (196 of 309 [63.4%] v 78 of 179 [43.6%] with post–high school education, P < .001), and less likely to report Hispanic ethnicity (16 of 305 [5.3%] v 24 of 180 [13.3%], P = .002) than those who chose to complete surveys by automated telephone, although age and education level were highly associated in this sample. Patients did not differ by sex, disease stage, ethnicity, or performance status. Overall compliance during the preoperative period was slightly higher for participants completing surveys by the Web (2,898 of 3,103 [93.4%]) versus by automated telephone (1,593 of 1,779 [89.5%]); Appendix Figs A1 and A2, online only).

Site Staff Effort and Feedback

On the basis of a survey of the 141 participating site CRAs (118 of 141 [83.7%] response rate; some CRAs cover more than one site), the median duration required for CRAs to teach a patient how to provide their PRO-CTCAE data electronically was 15 minutes (range, 5 to 60 minutes). At clinic visits, administrative work for the PRO-CTCAE to communicate with patients about problems completing surveys took a median of 10 minutes (range, 0 to 60 minutes). Most CRAs indicated that patient contact between clinic visits for issues such as lost passwords and log-in issues occurred never or rarely (44 of 118 [37.3%]) or occasionally (49 of 118 [41.5%]). CRAs reported spending a median of 5 minutes for each participant between each visit on work related to the PRO-CTCAE (range, 0 to 60 minutes). Most research staff reported that the software was easy to use (87 of 118 [73.7%]) and that they experienced no obstacles to implementing the system at their site (84 of 118 [71.2%]). Nonetheless, 12 of 118 CRAs (10.2%) experienced some technical difficulties, most commonly attributed to slow Internet connectivity.

Interviews with nine randomly selected site CRAs identified that the PRO-CTCAE system was easy to use, required minimal effort, and was manageable within their expected overall study-related duties. CRAs recommended developing a mobile application, because many patients had smartphones. CRAs recommended the PRO-CTCAE be integrated with Medidata Rave (the electronic data capture system used in National Clinical Trials Network trials) to save time registering patients and setting up study calendars.

Central Coordinator Effort

Work by the central coordinator for this trial required 50% of a full-time effort. Duties included conducting training Webinars for the 435 sites that opened the protocol, offering refresher training whenever new site staff joined the study, and responding to requests for technical support from sites and patients. The coordinator also attempted to collect 1,368 missing surveys from patients via telephone, which required an average of five calls per patient. Each unsuccessful call took 5 minutes on average, whereas successful calls (during which the coordinator administered the survey over the phone) took 15 minutes on average.

DISCUSSION

To our knowledge, this is the first published study demonstrating the feasibility of collecting patient-reported symptomatic AEs from home using a Web or automated telephone system in a multicenter cancer clinical trial. Most participants were willing and able to report this information weekly during active treatment. Backup data collection by a central coordinator further increased survey completion rates.

Prior studies of electronic PRO-CTCAE collection have found similarly high rates of survey completion using tablet computers or paper surveys in clinic.^4,9 However, these studies did not include at-home self-reporting between visits or human backup data collection, innovations that these study results suggest should become standard in patient-reported outcome (PRO) data collection of symptomatic AEs. In such an approach, the effort of site staff to support PRO data collection is limited and easily incorporated into their overall clinical trial–related duties, although central coordinator effort for human backup telephone calls is required.

Interest is growing in integrating PROs into cancer care, with evidence that symptom monitoring with PROs improves symptom control, communication, patient satisfaction, emergency room and hospitalization rates, tolerability of treatment, quality of life, and overall survival.^10,11 This article focuses on the implementation of PROs in clinical research as opposed to routine care delivery, and holds the promise of improving understanding of the patient experience with treatments.

Several limitations should be considered. First, although we did estimate the effort required by central and site staff for data collection, this study was not designed to gauge the effort required to integrate and analyze PRO-CTCAE data within the overall dataset for a trial. We expect that as PRO-CTCAE becomes more commonly used in trials and more familiar to staff, the associated administrative efforts for training, troubleshooting, data management, and statistical analysis will decline. Second, limited information was recovered in this study about reasons for missing PRO-CTCAE data, because many patients who did not self-report also did not respond to calls from study staff to elicit reasons for missing information. Such information about reasons for missing data may be useful to identify in future evaluations, to determine the effects of missing data on bias and the generalizability of study conclusions. Finally, because this was a planned interim analysis to assess feasibility only, it did not allow for comparison of PRO-CTCAE scores between study arms, or for the measure of data completeness after the 6-month follow-up time point. Such analyses are planned for when the primary clinical trial analysis is undertaken.

In conclusion, the PRO-CTCAE was developed to enhance understanding of the patient experience with symptomatic toxicities in cancer clinical trials. This study provides compelling evidence of the feasibility of this approach, laying the groundwork for broader implementation of the PRO-CTCAE in cancer clinical trials.

Appendix

Fig A1.

Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) compliance rates by week for subset of patients selecting Web reporting (n = 316). Postop, Postoperative.

Fig A2.

Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) compliance rates by week for subset of patients selecting automated telephone reporting (n = 184). Postop, Postoperative.

Table A1.

Generic Structure of PRO-CTCAE Items and Response Options

Table A2.

List of PRO-CTCAE Items Included in N1048 PROSPECT

Table A3.

Expedited Clinician-Reported CTCAE Grade 4 Toxicities

AUTHOR CONTRIBUTIONS

Conception and design: Ethan Basch, Amylou C. Dueck, Lauren J. Rogak, Sandra A. Mitchell, Andrea M. Denicoff, Martin R. Weiser, Bryce B. Reeve, Deborah Watkins Bruner, Deborah Schrag

Financial support: Sandra A. Mitchell, Lori M. Minasian

Administrative support: Lauren J. Rogak, Jennifer K. Wind, Mary C. Shaw, Narre Heon, Qian Shi

Provision of study materials or patients: Garth D. Nelson, Jeffrey P. Meyers, George J. Chang, Harvey J. Mamon, Tatjana Kolevska

Collection and assembly of data: Ethan Basch, Lauren J. Rogak, Sandra A. Mitchell, Jennifer K. Wind, Narre Heon, Jeffrey P. Meyers, Deborah Schrag

Data analysis and interpretation: Ethan Basch, Amylou C. Dueck, Lauren J. Rogak, Sandra A. Mitchell, Lori M. Minasian, Mary C. Shaw, Narre Heon, Qian Shi, Brenda Ginos, Garth D. Nelson, George J. Chang, Harvey J. Mamon, Tatjana Kolevska, Bryce B. Reeve, Deborah Watkins Bruner,

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Feasibility of Implementing the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events in a Multicenter Trial: NCCTG N1048

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/jco/site/ifc.

Ethan Basch

Consulting or Advisory Role: Noona Healthcare

Amylou C. Dueck

No relationship to disclose

Lauren J. Rogak

No relationship to disclose

Sandra A. Mitchell

No relationship to disclose

Lori M. Minasian

No relationship to disclose

Andrea M. Denicoff

No relationship to disclose

Jennifer K. Wind

No relationship to disclose

Mary C. Shaw

No relationship to disclose

Narre Heon

No relationship to disclose

Qian Shi

No relationship to disclose

Brenda Ginos

No relationship to disclose

Garth D. Nelson

No relationship to disclose

Jeffrey P. Meyers

No relationship to disclose

George J. Chang

Consulting or Advisory Role: Johnson & Johnson, MORE Health

Harvey J. Mamon

Honoraria: UpToDate

Martin R. Weiser

No relationship to disclose

Tatjana Kolevska

No relationship to disclose

Bryce B. Reeve

No relationship to disclose

Deborah Watkins Bruner

No relationship to disclose

Deborah Schrag

Research Funding: Pfizer