Feasibility of Patient Reporting of Symptomatic Adverse Events via the PRO-CTCAE in a Chemoradiotherapy Cooperative Group Multicenter Clinical Trial

Abstract

Purpose

To assess the feasibility of measuring symptomatic adverse events (AEs) in a multicenter clinical trial using the National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE).

Methods and Materials

Patients enrolled in Trial XXXX (XXXX) were asked to self-report 53 PRO-CTCAE items representing 30 symptomatic AEs at 6 time points (baseline; weekly x4 during treatment; 12-weeks post-treatment). Reporting was conducted via wireless tablet computers in clinic waiting areas. Compliance was defined as the proportion of visits when an expected PRO-CTCAE assessment was completed.

Results

Among 226 study sites participating in Trial XXXX, 100% completed 35-minute PRO-CTCAE training for clinical research associates (CRAs); 80 sites enrolled patients of which 34 (43%) required tablet computers to be provided. All 152 patients in Trial XXXX agreed to self-report using the PRO-CTCAE (median age 66; 47% female; 84% white). Median time for CRAs to learn the system was 60 minutes (range 30–240), and median time for CRAs to teach a patient to self-report was 10 minutes (range 2–60). Compliance was high, particularly during active treatment when patients self-reported at 86% of expected time points, although compliance was lower post-treatment (72%). Common reasons for non-compliance were institutional errors such as forgetting to provide computers to participants; patients missing clinic visits; internet connectivity; and patients feeling “too sick”.

Conclusions

Most patients enrolled in a multicenter chemoradiotherapy trial were willing and able to self-report symptomatic adverse events at visits using tablet computers. Minimal effort was required by local site staff to support this system. The observed causes of missing data may be obviated by allowing patients to self-report electronically between-visits, and by employing central compliance monitoring. These approaches are being incorporated into ongoing studies.

BACKGROUND

Adverse events (AEs) are reported in cancer clinical trials using the U.S. National Cancer Institute’s Common Terminology Criteria for Adverse Events (CTCAE).¹ The CTCAE is a library of items representing 790 discrete AEs. Each AE is graded using a 5-point numerical scoring system, which is anchored to discrete clinical criteria.² Approximately 10% of AEs in the CTCAE are symptoms (e.g., nausea, sensory neuropathy), which in trials have historically been reported by clinical investigators.³ However, there is empiric evidence that collection of this information directly from patients improves the reliability and precision of symptomatic AE detection in trials.^4–7

Recently, the U.S. National Cancer Institute (NCI) developed a library of patient-reported outcome (PRO) items to supplement the CTCAE, called the PRO-CTCAE.⁸ The PRO-CTCAE was developed by systematically identifying AEs in the CTCAE that are amenable to patient self-report then creating PRO items for each of these using rigorous interdisciplinary methods;⁹ then by establishing the measurement properties of these items using qualitative¹⁰ and quantitative¹¹ psychometric methods.

For each AE in the PRO-CTCAE, between 1 and 3 items are included to assess the frequency, severity, and/or interference with activities related to that AE (Supplemental Table S1). PRO-CTCAE can be administered to patients electronically using software hosted at the NCI which has undergone usability testing and refinement.⁹

Although it is established that PRO-CTCAE items are well understood by patients and accurately represent symptomatic AEs, the feasibility of implementation of PRO-CTCAE items in multicenter cancer clinical trials is not established. Specifically, the level of staff effort required to teach and remind patients to self-report using the PRO-CTCAE software and to manage these data, staff acceptance, and patient willingness and ability to longitudinally self-report during treatment are unknown. This information is essential for determining if it is practical to employ the PRO-CTCAE in future trials, and for providing information about barriers and strategies towards more broadly integrating the PRO-CTCAE into clinical trial workflow.

METHODS

Patients enrolled in the U.S. National Clinical Trials Network multicenter trial, Trial XXXX (XXXX) (ClinicalTrials.gov: XXXX) were invited to participate in a correlative study to evaluate the feasibility of utilizing the PRO-CTCAE within a clinical trial. The Trial XXXX protocol, including the embedded PRO-CTCAE correlative study, was approved by the Institutional Review Boards of all participating institutions, and all patients underwent informed consent.

Participants in Trial XXXX were randomly assigned to receive either liquid Manuka honey, Manuka honey lozenge, or placebo daily during radiation treatment. The primary endpoint of Trial XXXX was to assess the effects of Manuka honey on dysphagia at 4 weeks based on a numerical rating scale,¹² and results of that analysis have been reported elsewhere.¹³

All participants in Trial XXXX were asked to self-report 53 PRO-CTCAE items representing 30 discrete toxicities (Supplemental Table S2) at baseline and weekly during the four weeks of active radiation treatment, and once post-treatment at week 12. These items were selected by the clinical trial investigators based on expected toxicities related to the trial therapy, as well as based on a set of previously identified symptoms that are prevalent among cancer patients undergoing treatment.¹⁴ These items were loaded into the web questionnaire platform for the PRO-CTCAE (Figure 1), which is hosted by the NCI. PRO-CTCAE items were available in English or Spanish.

Figure 1.

PRO-CTCAE patient questionnaire interface, used via iPad in waiting areas at study visits (software hosted at the NCI)

A central PRO-CTCAE data manager was responsible for training clinical research associates (CRAs) at all participating sites. This entailed a standardized 35-minute webinar which taught CRAs how to register patients into the PRO-CTCAE software system and how to educate patients to login and self-report adverse events using the system. The central data manager also offered refresher orientations as needed (e.g., for changes in CRA personnel), and was available for technical questions or problems experienced by sites.

Site CRAs educated participants in the clinical trial to complete PRO-CTCAE items via wireless tablet computers anytime between informed consent and the baseline visit. Then, at each specified PRO-CTCAE assessment time point, CRAs were instructed, per protocol, to approach participants at their clinic visits and provide the wireless tablet computer to complete the PRO-CTCAE items. A 72-hour window prior to the due date for each PRO-CTCAE assessment was allowed. If a participant did not complete the PRO-CTCAE within that time frame, an email alert was generated to the site CRA, who was instructed to contact the patient and attempt to obtain the PRO-CTCAE information and enter it into the system.

At each study visit, site investigators reported adverse events using criteria from the CTCAE via a standardized AE form, which is the typical approach used in cancer clinical trials¹⁵ and required by the trial protocol. PRO-CTCAE reports of adverse events were not shared with CRAs or site clinical investigators. Patients were educated not to rely on the PRO-CTCAE system as a mechanism to inform clinicians about their symptoms, and to communicate directly with their nurse or treating physician about symptoms of concern.

PRO-CTCAE compliance during active treatment was defined as the proportion of pre-specified PRO-CTCAE reporting time points (i.e., baseline visit and weekly ×4 visits) at which PRO-CTCAE assessments were completed by participants who were still alive and enrolled in the trial. Compliance was also evaluated at each pre-specified reporting time point individually, including at the post-treatment 12-week visit. Reasons for missed PRO-CTCAE assessments were collected using a standardized form.

A survey of site CRAs was conducted to understand the effort required to use the PRO-CTCAE system and to obtain feedback. In addition, 10 one-to-one interviews were conducted with randomly selected site CRAs after they had 6 months of experience with the system, to focus on issues identified in the surveys. Effort required of the central data manager was evaluated by tabulating time for site trainings and refreshers. A patient survey was added halfway through the trial to collect patient impressions of the PRO-CTCAE system.

The cumulative incidence of post-baseline investigator-reported CTCAE grades and patient-reported PRO-CTCAE scores for each measured adverse event were tabulated by treatment arm (supportive care arm versus the combined Manuka honey arms).

RESULTS

Between February 2012 and October 2013, 163 patients enrolled in Trial XXXX, of which 3 were ineligible, 4 withdrew consent prior to treatment, and 4 opted not to receive treatment in the trial and were excluded from all analyses, yielding a total of 152 participants. Baseline characteristics were similar across the study arms (Table 1) with an overall median age of 66 years (range 37–85), 47% female, and 84% white.

Table 1.

Characteristics of the participants (N=152)

	Supportive Care (n=48)	Liquid Honey (n=52)	Lozenge Honey (n=52)
Age (years)
Median	66	67	65
Range	45 – 85	37 – 83	47 – 83
Gender
Male	23 (48%)	29 (56%)	29 (56%)
Female	25 (52%)	23 (44%)	23 (44%)
Race
American Indian or Alaskan Native	0 (0%)	1 (2%)	1 (2%)
Asian	2 (4%)	1 (2%)	1 (2%)
Black or African American	7 (15%)	5 (10%)	7 (14%)
White	39 (81%)	45 (87%)	43 (83%)
Use of IMRT
No	20 (42%)	23 (44%)	29 (56%)
Yes	28 (58%)	29 (56%)	23 (44%)
Percentage of Esophagus in Radiation Field
< 30%	33 (63%)	33 (64%)	32 (62%)
≥ 30%	18(38%)	19(37%)	20 (39%)

IMRT, Intensity Modulated Radiation Therapy

The study protocol was approved at 226 sites across the United States. CRAs from each of these sites underwent PRO-CTCAE orientation via a 35-minute webinar. Patients were actively enrolled into the trial at 80 of these sites, of which 34 (42%) required provision of tablet computers for PRO-CTCAE completion, while 46 sites (58%) had available waiting room computers.

During the trial, there were 715 scheduled clinic visits during active treatment at which participants were expected to complete a PRO-CTCE assessment (i.e., visits at which patients were still alive and enrolled in the trial). Of these, PRO-CTCAE assessments were completed at 618 (compliance rate of 86%, Figure 2). Compliance was lower at the post-treatment week 12 visit (72%), when application of protocol procedures was less stringent. Therefore, including all expected visits during active treatment and follow-up, patients self-reported at 715/849 (84%) time points.

Figure 2.

Proportion of study participants completing PRO-CTCAE Adverse Event self-reports at successive study visits.

Among the 134 instances when patients did not self-report PRO-CTCAE at expected time points during treatment and follow up, 28 (21%) occurred because participants missed their clinic appointment; 28 (21%) due to staff errors including forgetting to provide computers to participants and lack of staff coverage during CRA vacations; and 11 (8%) occurred at a single site where staff misinterpreted the protocol and did not observe PRO-CTCAE procedures. There were 20 (15%) cases where technical problems prevented PRO-CTCAE completion (including computer malfunction and internet connectivity problems); 18 cases (13%) when patients were considered “too sick” to self-report; and 12 (9%) PRO-CTCAE reports that were provided by patients outside the required time frame for reporting at a given time point.

Based on a survey of the 70 site CRAs who covered accrual at the 80 sites during conduct of this trial (some CRAs cover more than one site), the median duration for CRAs to teach a patient how to report PRO-CTCAE data electronically was 10 minutes (range 2–60) (Table 2a). At each follow-up visit, administrative work for the PRO-CTCAE took an average of 10 minutes (0–60) while patient contact for the PRO-CTCAE took an average of 15 minutes (0–60). Most research staff found the software system easy to use (79%) and perceived no obstacles at their site for implementing the system (72%, Table 2b). Nonetheless, about one-third experienced some technical difficulties, most commonly attributed to slow internet connectivity (reported by 31% of CRAs, Table 2c).

TABLE 2.

Results of surveys of site clinical research associates (N=70) and patients (N=63)

a. Effort by site clinical research associates
Activity	Median	Mean	Range
Time for CRA to learn PRO-CTCAE software	60 min	70 min	30–240*
Time for CRA to teach PRO-CTCAE to one patient	10 min	16 min	2–60
Time per clinic visit for PRO-CTCAE administrative tasks	10 min	12 min	0–60
Time spent with each patient at clinic visits for PRO- CTCAE	15 min	17 min	0–60

b. Ease of use by site clinical research associates
After training, PRO-CTCAE software was:
- Easy to use	52/66 (79%)
- Moderate to use	13/66 (20%)
- Difficult to use	1/66 (2%)
Did you experience any of these obstacles to implementing PRO- CTCAE at your site:
- No obstacles	46/64 (72%)
- Staff resources inadequate	8/64 (13%)
- Patient resistance	12/64 (19%)
- Staff resistance	2/64 (3%)

c. Technical difficulties experienced by site clinical research associates
Proportion of CRAs who noted experiencing more than minimal technical difficulties	23/55 (35%)
Number of CRAs reporting any problems (non-mutually exclusive) with:
- Connectivity/network problems/slow	17/55 (31%)
- Lost passwords	8/55 (15%)
- Software errors	6/55 (11%)
- Other (firewall, hardware, screen problem)	5/55 (10%)

d. Patient impressions
	Agree	Disagree
PRO-CTCAE questions were easy to understand	55/60 (92%)	5/60 (8%)
PRO-CTCAE software was easy to use	51/60 (85%)	9/60 (15%)
PRO-CTCAE improved discussions with my doctor/nurse	45/59 (76%)	14/59 (24%)
I would recommend PRO-CTCAE to other patients	45/59 (76%)	14/59 (24%)
PRO-CTCAE made me feel more in control of my own care	40/58 (69%)	18/58 (31%)

Abbreviation: CRA, Clinical Research Assistant at individual study sites

^*1 CRA noted 240 min; 8 CRAs 120 min; all others <90 min

Abbreviation: CRA, Clinical Research Assistant at individual study sites

In depth one-on-one interviews with 10 randomly selected site CRAs identified slow network connectivity as the most substantial barrier to feasibility and survey compliance. Staff felt that PRO-CTCAE was more challenging for older, ill, and non-computer experienced participants, but was feasible in such patients with encouragement and adequate support.

Effort by the central data manager included 226 35-minute training webinars and 17 refresher training webinars, and 42 interactions with site CRAs to answer questions about PRO-CTCAE software and internet connectivity problems. Approximately 15% of full-time effort was dedicated to this role during the study.

A patient survey was distributed to 67 participants and completed by 63 (94%), with most reporting that questions were easy to understand, software was easy to use, and PRO-CTCAE use led to improved discussions with physicians and nurses (Table 2d).

The cumulative incidence of post-baseline investigator-reported CTCAE and patient-reported PRO-CTCAE ratings are shown in Table 3. Rate of symptomatic AEs based on CTCAE or PRO-CTCAE were similar between supportive care and Manuka honey arms, as Manuka honey does not appear to confer adverse symptoms. The incidence of AEs was higher with patient reporting than clinician reporting, but because this was a supportive care trial testing a relatively benign intervention, investigators were not oriented to report AEs related to chemoradiotherapy, while the PRO-CTCAE asks patients to report symptoms regardless of potential etiology. Patients reported different frequencies for different symptoms. Figure 3 shows longitudinal PRO-CTCAE trajectories during the trial of two common adverse events related to chemoradiotherapy in this population, dysphagia and radiation dermatitis, as examples of how PRO-CTCAE can elucidate the patient experience over time.

Table 3.

Cumulative incidence of adverse events post-baseline for patients enrolled in Trial XXXX, based on investigator-reported CTCAE and patient-reported PRO-CTCAE.

Adverse Event			Any Level (>0)		High-Level*
			Control	Manuka§	Control	Manuka§
Anorexia	CTCAE:		24%	21%	2%	1%
	PRO-CTCAE:	Severity Interference	76% 54%	89% 74%	26% 20%	27% 27%
Anxiety	CTCAE:		9%	6%	–	–
	PRO-CTCAE:	Frequency Severity Interference	74% 72% 50%	90% 89% 59%	22% 20% 15%	27% 20% 18%
Concentration impairment	CTCAE:		2%	2%	–	–
	PRO-CTCAE:	Severity Interference	67% 48%	70% 56%	9% 9%	4% 10%
Constipation	CTCAE:		28%	24%	–	1%
	PRO-CTCAE:	Severity	74%	81%	17%	30%
Cough	CTCAE:		30%	34%	–	1%
	PRO-CTCAE:	Severity Interference	93% 61%	94% 71%	26% 20%	18% 16%
Depression	CTCAE:		2%	4%	–	–
	PRO-CTCAE (sad feelings):	Frequency Severity Interference	74% 74% 48%	87% 84% 60%	11% 11% 11%	18% 13% 13%
	PRO-CTCAE (nothing can cheer you up):	Frequency Severity Interference	57% 54% 35%	70% 61% 45%	7% 9% 7%	13% 10% 10%
Radiation dermatitis	CTCAE:		26%	22%	2%	1%
	PRO-CTCAE:	Severity	54%	61%	9%	10%
Diarrhea	CTCAE:		20%	15%	–	1%
	PRO-CTCAE:	Frequency	61%	61%	15%	12%
Dry mouth	CTCAE:		7%	7%	–	–
	PRO-CTCAE:	Severity	65%	79%	11%	11%
Dry skin	CTCAE:		11%	2%	–	–
	PRO-CTCAE:	Severity	67%	73%	4%	9%
Dysgeusia	CTCAE:		13%	13%	–	–
	PRO-CTCAE:	Severity	80%	84%	17%	15%
Dyspepsia	CTCAE:		22%	9%	–	–
	PRO-CTCAE:	Frequency Severity	76% 76%	80% 80%	33% 28%	27% 18%
Dysphagia	CTCAE:		37%	38%	–	2%
	PRO-CTCAE:	Severity	76%	86%	33%	23%
Dyspnea	CTCAE:		20%	34%	2%	5%
	PRO-CTCAE:	Severity Interference	83% 65%	90% 78%	17% 15%	19% 18%
Fatigue	CTCAE:		61%	50%	7%	1%
	PRO-CTCAE:	Severity Interference	98% 93%	97% 93%	37% 33%	39% 44%
Headache	CTCAE:		11%	7%	–	–
	PRO-CTCAE:	Frequency Severity Interference	63% 61% 33%	68% 65% 40%	15% 13% 11%	11% 5% 5%
Hiccups	CTCAE:		2%	5%	–	–
	PRO-CTCAE:	Frequency Severity	63% 57%	65% 60%	24% 11%	13% 3%
Hoarseness	CTCAE:		7%	3%	–	–
	PRO-CTCAE:	Severity	67%	73%	16%	6%
Insomnia	CTCAE:		13%	7%	–	1%
	PRO-CTCAE:	Severity Interference	85% 67%	84% 74%	24% 13%	22% 22%
Memory impairment	CTCAE:		2%	3%	–	–
	PRO-CTCAE:	Severity Interference	61% 46%	65% 53%	2% 2%	4% 5%
Mucositis	CTCAE:		4%	7%	–	1%
	PRO-CTCAE:	Severity Interference	35% 26%	48% 33%	7% 7%	3% 3%
Nausea	CTCAE:		37%	34%	2%	1%
	PRO-CTCAE:	Frequency Severity	63% 63%	79% 77%	15% 7%	16% 18%
Pain	CTCAE:		26%	23%	2%	2%
	PRO-CTCAE:	Frequency Severity Interference	78% 78% 65%	87% 87% 71%	37% 30% 26%	37% 23% 28%
Pruritus	CTCAE:		11%	3%	–	–
	PRO-CTCAE:	Severity	54%	57%	9%	4%
Rash maculo-papular	CTCAE:		9%	2%	–	–
	PRO-CTCAE:	Presence	33%	35%	–	–
Skin hyperpigmentat ion	CTCAE:		2%	3%	–	–
	PRO-CTCAE:	Presence	28%	28%	–	–
Urticaria	CTCAE:		–	–	–	–
	PRO-CTCAE:	Presence	13%	18%	–	–
Voice alteration	CTCAE:		–	6%	–	–
	PRO-CTCAE:	Presence	50%	48%	–	–
Vomiting	CTCAE:		17%	10%	2%	–
	PRO-CTCAE:	Frequency Severity	35% 35%	51% 47%	9% 9%	4% 5%
Wheezing	CTCAE:		–	4%	–	–
	PRO-CTCAE:	Severity	54%	65%	9%	11%

^*High-level clinician-reported adverse events are defined as CTCAE grade 3 or 4. High-level patient-reported adverse events are defined as PRO-CTCAE scores for severity items as severe or very severe; for frequency items as frequently or almost constantly; and for interference items as quite a bit or very much.

^§Manuka honey liquid and lozenge arms are collapsed into a single Manuka group for this table.

Figure 3.

Longitudinal PRO-CTCAE trajectories for radiation dermatitis severity (Panel A) and dysphagia severity (Panel B) at successive visits during the clinical trial, showing the distribution of scores at each time point for all arms combined. Each score number reflects ascending severity of PRO-CTCAE severity response criteria (mild, moderate, severe, very severe).

DISCUSSION

Most patients enrolled in a multicenter chemoradiotherapy clinical trial for lung cancer were willing and able to self-report their own symptomatic adverse events at clinic visits using tablet computers. Minimal effort was required by local site staff and by a central data coordinator to support use of the PRO-CTCAE. Most missing data was attributable to patients not attending scheduled clinic visits, to staff errors, and to technical problems with internet connectivity.

Reasons for missing data in this study are informative for designing future PRO-CTCAE implementation strategies aimed at improving response rates. These findings also inform use of other types of electronic PRO data collection in clinical trials. First, because this trial depended on PRO-CTCAE reports being completed by patients at their clinic visits, if patients missed visits for any reason (e.g., illness, vacation, logistics), the PRO-CTCAE data could not be captured. An alternative strategy is to collect this information using an approach that does not depend on visit attendance, such as between-visit reporting via the web or an automated telephone system. This approach is currently being employed in follow-up PRO-CTCAE feasibility assessments. Second, the central data manager was not allowed to directly interact with patients in this trial because of the structure of the protocol. As a result, the central data manager could not contact patients who missed reports for backup data collection. Site CRAs were depended upon for this function without central monitoring; an approach that our results suggest yielded missing data. In ongoing follow-up PRO-CTCAE studies, an approach is being used in which the central data manager can contact patients directly for reminders and backup data collection. Third, the current PRO-CTCAE software depends on a continuous active internet connection throughout questionnaire completion. Internet connectivity was a common barrier and frustration for CRAs and patients in this study, suggesting the potential value of a downloadable application for the PRO-CTCAE software. Finally, in the analysis for this study we were not able to accurately assess the timing of PRO-CTCAE reports with treatment cycle timing and delays, because of the lack of a software interface between the PRO-CTCAE system and the trial’s clinical data management system. In the future, such interfaces would enhance the ability to analyze relationships between patient-reported toxicities and the timing of treatments.

Necessary additional resources to support use of the PRO-CTCAE in this study included software hosting, maintenance, secure data storage, and user technical support. These roles were performed by the NCI’s Center for Biomedical Informatics and Information Technology (CBIIT). However, in the future this role might be performed by entities that conduct trials such as cooperative groups, pharmaceutical companies, and their contracted technology vendors.

Limitations of this trial included assessment in a single disease, lung cancer, which historically has had lower levels of PRO questionnaire compliance than other cancer populations.^16–18 Notably, the PRO-CTCAE is currently being assessed in other disease contexts, and even given this limitation, levels of compliance were relatively high. An additional limitation was that there was not an imbalance of toxicities between arms in this trial because Manuka honey does not cause adverse effects. Therefore, the capacity of the PRO-CTCAE to delineate toxicities between treatment arms could not be assessed. This is the focus of other ongoing studies. However, the analysis of PRO-CTCAE in this trial demonstrates the value of PRO-CTCAE for describing the relative prevalence of different symptomatic adverse events, and longitudinal trajectories of adverse events.

In conclusion, this study describes an approach for collecting and reporting patient-reported adverse events in clinical research, provides initial evidence of feasibility, and lends insights about approaches to potentially optimize response rates.

SUMMARY.

The Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) was developed by the National Cancer Institute to enable patient-reporting of toxicities in clinical research. To assess feasibility of implementation, PRO-CTCAE was integrated into an NRG Oncology trial. During treatment, patients reported via tablet computers at 86% of visits. Reasons for missing reports included staff errors, missed appointments, and internet connectivity. Strategies to address these reasons are being assessed in ongoing studies.