Abstract
BACKGROUND:
Clinicians are the standard source for adverse event (AE) reporting in oncology trials, despite the subjective nature of symptomatic AEs. The authors designed a pediatric patient-reported outcome (PRO) instrument for symptomatic AEs to support the National Cancer Institute’s Common Terminology Criteria for Adverse Events (CTCAE) (the Pediatric PRO-CTCAE). The current study developed a standardized algorithm that maps all possible Pediatric PRO-CTCAE response patterns to recommended CTCAE grades to improve the accuracy of AE reporting in pediatric oncology trials.
METHODS:
Two rounds of surveys were administered to experienced cancer clinicians across 9 pediatric hospitals. In round 1, pediatric oncologists assigned CTCAE grades to all 101 possible Pediatric PRO-CTCAE response patterns. The authors evaluated clinician agreement of CTCAE grades across response patterns and categorized each response pattern as having high or low agreement. In round 2, a survey was sent to a larger clinician group to examine clinician agreement among a select set of Pediatric PRO-CTCAE response patterns, and the authors examined how clinical context influenced grade assignment.
RESULTS:
A total of 10 pediatric oncologists participated in round 1. Of the 101 possible patterns, 89 (88%) had high agreement. The Light weighted kappa was averaged across the 10 oncologists (Light kappa = 0.73; 95% CI, 0.66–0.81). A total of 139 clinicians participated in round 2. High clinician agreement remained for the majority of generic response patterns and the clinical context did not typically change grades but rather improved agreement.
CONCLUSIONS:
The current study provides a framework for integrating child self-reported symptom data directly into mandated AE reporting in oncology trials. Translating Pediatric PRO-CTCAE responses into clinically meaningful metrics will guide future cancer care and toxicity grading.
Keywords: adverse events, patient-reported outcomes, pediatric oncology, toxicity grading
INTRODUCTION
Up to 60% of children diagnosed with cancer participate in a clinical trial.1–3 The federal government mandates that all trials collect and report adverse events (AEs), defined by the National Cancer Institute as “Adverse Event (AE) is any unfavorable and unintended sign (including an abnormal laboratory finding), symptom, or disease temporally associated with the use of a medical treatment or procedure that may or may not be considered related to the medical treatment or procedure.”4 Thus, AE reporting is an essential activity to ensure patient safety and to provide data to sponsors, regulators, patients, caregivers, and clinicians regarding treatment effects. The standard practice in oncology trials is for clinicians to grade and report all AEs using the National Cancer Institute’s Common Terminology Criteria for Adverse Events (CTCAE).5
Evidence from multiple studies in pediatric cancer populations has demonstrated that clinicians and caregivers underreport the number and severity of symptoms, such as nausea and insomnia, compared with self-report from children.6–9 These studies have suggested that there is a need to better capture the symptom experiences through direct self-report by the child. More accurately measured symptom AEs will help clinicians to improve the AE grading required for clinical trials, as well as enhance patient care, by guiding treatment modifications and supportive care guidelines to reduce symptom burden and morbidity.
In response to this need, our team designed the Pediatric Patient-Reported Outcome version of the Common Terminology Criteria for Adverse Events (Pediatric PRO-CTCAE), which to the best of our knowledge is the first pediatric self-report symptom AE measure designed for use in pediatric oncology trials. As previously described, the Pediatric PRO-CTCAE item library can assess up to 62 symptom AEs that are relevant and content valid for children and adolescents receiving cancer treatment.10,11 However, for the Pediatric PRO-CTCAE measurement system to be adopted in pediatric oncology trials and cancer care settings, we must translate self-report data from the Pediatric PROCTCAE questions into meaningful units for clinicians. This approach may enhance clinicians’ understanding of the child’s symptom AE experience and guide appropriate toxicity grading and cancer care.
The goal of the current study was to develop a mapping algorithm to be used as a clinician decision aid that will convert all possible Pediatric PRO-CTCAE response patterns into recommended CTCAE grades that are interpretable to clinicians. This mapping process was performed through a consensus process with pediatric oncologists and other clinicians experienced in treating children and CTCAE grading in pediatric oncology trials. These recommended grades provide complementary information from the child and/or adolescent to facilitate more comprehensive clinician CTCAE grading and to enhance understanding of the patient’s symptoms in clinical care.
MATERIALS AND METHODS
Participants
The current study involved 2 survey rounds and built on our established collaborations with the 9 pediatric hospitals involved in the design of the Pediatric PRO-CTCAE system: Children’s Healthcare of Atlanta, Children’s Hospital Los Angeles, Children’s Hospital of Philadelphia, University of Pittsburgh Medical Center Children’s Hospital of Pittsburgh, Children’s National Health System, Dana-Farber Cancer Institute, Duke University (round 2 only), St. Jude Children’s Research Hospital, and the University of North Carolina at Chapel Hill. The University of North Carolina at Chapel Hill Office of Human Research deemed the study exempt research.
Measures
Pediatric PRO-CTCAE measurement system
The Pediatric PRO-CTCAE was designed for children and adolescents to self-report symptom AEs they experience while undergoing cancer treatment.10,11 The system consists of a library of 62 symptom AEs assessed by 130 items. Each AE (eg, fatigue) has up to 3 attributes that were selected by experienced pediatric oncology clinicians in a prior study.11 Attributes include symptom presence, frequency, severity, and/or interference with daily activities. The recall period for all items is “the past 7 days,” and each question within an attribute has up to 4 response options. Figure 1 depicts the Pediatric PRO-CTCAE questions and response categories for pain as an exemplar. An adult PRO-CTCAE exists with a different number of items, more response options, and different item wording; for these reasons, a different mapping algorithm to CTCAE will be made available.
Figure 1.
The Pediatric Patient-Reported Outcome version of the Common Terminology Criteria for Adverse Events (Pediatric PRO-CTCAE) questions for pain and corresponding response options presented alongside the CTCAE grades for pain. ADL indicates activities of daily living.
Common Terminology Criteria for Adverse Events
The CTCAE (version 5.0) is a lexicon for clinicians to grade severity and report up to 837 AEs experienced by patients receiving cancer treatment.4 Although each AE has its own specific grading criteria, in general the grades are characterized as grade 0 (no AE), grade 1 (mild symptoms), grade 2 (moderate symptoms limiting instrumental activities of daily living [ADL]), grade 3 (severe symptoms limiting self-care ADL), grade 4 (life threatening), and grade 5 (AE-related death). In the current study, we did not include CTCAE grades 4 or 5 because patients would not be able to self-report these grades (Fig. 1). Previously, this group of investigators reviewed the CTCAE (version 4.0) terms that were amenable to self-report by children aged ≥7 years, resulting in the 62 symptom AEs selected for inclusion in the Pediatric PRO-CTCAE.11
Study Design
Round 1 survey
The goal of the first round of surveys was to assess levels of clinician agreement regarding CTCAE grades across all possible Pediatric PRO-CTCAE response patterns to understand when there was consensus in grading. We created a mapping sheet that would enable pediatric oncologists to provide their expert opinion regarding translating generic Pediatric PRO-CTCAE response combinations (ie, without clinical information such as the type of symptom AE) into equivalent recommended CTCAE grades. The mapping sheet contained all possible combinations of responses to the Pediatric PRO-CTCAE and included corresponding blank spaces for oncologists to assign their recommendation for the comparable CTCAE grade (see Fig. 2 for example). In total, there were 101 possible combinations.
Figure 2.
A sample subsection of the mapping sheet that oncologists completed. Pediatric PRO-CTCAE indicates Pediatric Patient-Reported Outcome version of the Common Terminology Criteria for Adverse Events.
Participants were pediatric oncologists who were attending physicians with ≥5 years of experience grading the CTCAE and treating diverse pediatric oncologic diseases. Individual oncologists who assisted with the design of the Pediatric PRO-CTCAE were invited to participate in the first round of surveys. The mapping sheet was distributed by email to facilitate easy completion, and each oncologist was sent a $10 gift card once the mapping sheet was returned. Next, completed mapping sheets were summarized and CTCAE recommendations were tallied for each response combination. Each Pediatric PRO-CTCAE response pattern was categorized as high agreement or low agreement using the Cohen kappa statistic. Given multiple raters and ordinal ratings, we adopted the Light version12 (averaging Cohen kappa statistics across all possible pairs among the 10 oncologists) of weighted kappa13 (multiplying each cell of the observed contingency table by squared distances between 2 ratings). The influence of prevalence on kappa has been found to be negligible.14 For further assurance, we also calculated the intra-class correlation for quantifying agreement among raters (2-way random with single measure).15 High agreement was defined as an individual CTCAE grade (grade 0–3) with ≥70% agreement noted among the senior pediatric oncologists. Low agreement was defined as no single grade for a response pattern having a percentage >70%. Landis and Koch classified a 70% level of agreement as “substantial agreement.”16 Two calls were held with all participating oncologists once the votes were tallied to review the results. To reduce respondent burden for the subsequent survey, these senior oncologists voted on which low agreement response patterns should be passed forward to round 2 of the study to seek input from a broader range of clinicians. If one oncologist voted to seek additional input on a response pattern, then the response pattern was moved forward to round 2.
Round 2 survey
The second survey round had 4 goals. The first goal was to evaluate clinician agreement regarding CTCAE grading for generic Pediatric PRO-CTCAE response patterns with low levels of agreement from round 1 among a large, diverse group of pediatric oncology clinicians. We did not expect our larger sample to reach consensus, but rather we wanted to confirm which response combinations were challenging to grade as well as understand the distribution of recommended grades among these items. The second goal was to confirm that Pediatric PRO-CTCAE response patterns categorized as high agreement in round 1 continued to be categorized as high agreement among the larger group of clinicians in round 2. We included a subset of high agreement response patterns from round 1 only to reduce respondent burden. The third goal was to assess whether clinicians assigned different CTCAE grades to generic Pediatric PRO-CTCAE response patterns when compared with the same response patterns for which additional clinical information was provided (eg, cancer type, treatment modality, and AE). This evaluation addressed the variation in assigning a CTCAE grade to a generic Pediatric PRO-CTCAE response pattern based on the clinical characteristics of the child. Finally, the fourth goal was to evaluate whether the context changed clinician CTCAE grading when patient-level factors varied (eg, boy or girl, chemotherapy or radiotherapy) within clinical scenarios.
The round 2 survey included 2 parts. The first part asked the clinicians to assign CTCAE grades to generic Pediatric PRO-CTCAE response patterns with low agreement that were passed forward from round 1, as well as a subset of high agreement generic response patterns from round 1. To assess whether context influences clinician grading, investigators (including pediatric oncologists) created clinical vignettes using Pediatric PRO-CTCAE response patterns that varied AE information (eg, fatigue vs nausea) and patient characteristics (ie, age, sex, treatment modality, and cancer diagnosis). The second part of the survey included these clinical vignettes with added context to inform CTCAE grading decisions. Figure 3 presents an example scenario, including the level of detail provided. The majority of the clinical vignettes had a generic Pediatric PRO-CTCAE response pattern counterpart that appeared earlier in the survey, which allowed us to determine whether grading differed. In addition, the survey included demographic questions regarding clinician characteristics (eg, institution and years of practice).
Figure 3.
An example of the type of scenario presented to clinicians for the round 2 survey. ADL indicates activities of daily living; Pediatric PRO-CTCAE, Pediatric Patient-Reported Outcome version of the Common Terminology Criteria for Adverse Events.
Physicians, nurse practitioners, and physician assistants with at least 2 years of experience grading the CTCAE and treating diverse pediatric oncologic diseases were eligible to participate. A representative from each of our 9 sites recommended clinicians from their respective site and provided contact information for those individuals. In total, we received contact information for 238 clinicians. Investigators then drafted an email that explained the goals of the study and each site representative sent this email to their nominated clinician colleagues. Afterward, clinicians were sent an individual survey link through Qualtrics, and 2 additional reminders. As a show of gratitude, each clinician was given a $5 gift card.
Generic Pediatric PRO-CTCAE responses were categorized as low agreement and high agreement using the same thresholds as in round 1. Given a repeated-measures design and rank data, we used the Wilcoxon signed rank test to compare scenarios in which we varied one element (eg, age, diagnosis, or treatment) within the clinical vignettes. We also used the Wilcoxon signed rank test when comparing generic Pediatric PRO-CTCAE response patterns with response patterns containing clinical context.
RESULTS
Round 1 Survey
In round 1, a total of 10 senior pediatric oncologists, representing 8 institutions, were approached to participate and all 10 completed the mapping sheet. For overall levels of interrater agreement, Light’s weighted kappa for ordinal measures averaged 45 possible pairs across the 10 oncologists was 0.73 (95% CI, 0.66–0.81). As an alternative measure of rater agreement, the intraclass correlation was 0.729 (95% CI, 0.67–0.79). Both statistics indicate a substantial level of agreement and good clinical significance.17
Of the 101 possible Pediatric PRO-CTCAE response patterns, 69 (68%) had high agreement, with 23 response patterns having unanimous agreement. The remaining 32 response patterns were categorized as low agreement; of these, the clinician members of our team selected 19 patterns to be passed to round 2 of the survey.
Low-agreement patterns included unlikely scenarios, such as if a patient reported symptom frequency of “almost all the time,” symptom severity of “did not have any,” and symptom interference of “a whole lot.” In addition, clinicians disagreed with regard to which response combinations warranted moving to a higher recommended CTCAE grade. There was particular disagreement concerning when a response pattern should remain at grade 2 and when it should be moved to grade 3. Overall, we observed that oncologists were more likely to assign a higher CTCAE grade to a response combination if the AE interfered with daily activities. In other words, oncologists tended to rely more on the interference attribute than frequency or severity.
Round 2 Survey
A total of 27 generic Pediatric PRO-CTCAE response combinations (8 high agreement and 19 low agreement) and 25 clinical vignettes were included in the round 2 survey. In round 2, a total of 238 clinicians were sent the survey link and 139 participated (response rate, 58%) (see Supporting Table 1). For overall agreement across 139 clinicians, considering all 27 (generic) response patterns, the Light’s weighted kappa was 0.57 (95% CI, 0.48–0.63). The median amount of time needed to complete the survey was 16 minutes.
Of the 19 low-agreement response patterns included in round 2, 13 patterns (68%) remained low agreement among the 139 clinicians (see Supporting Table 2). Of the 8 high-agreement response patterns from round 1, 5 response patterns (63%) continued to have high agreement in round 2.
There were 14 clinical vignettes with a generic response pattern counterpart. For 8 of the 14 response patterns (57%), the median assigned CTCAE grade differed significantly when clinical context was provided, based on the Wilcoxon signed rank test (Table 1). Among those 8 pairs that differed significantly, the most recommended CTCAE grade remained the same across the generic response pattern and clinical vignette for 5 cases: pruritus, vomiting, fecal incontinence, dyspepsia, and constipation. For 2 response patterns, adding clinical context (ie, diarrhea, chills) resulted in the most recommended CTCAE grade to shift down from a grade 3 to a grade 2. The final significant response pattern changed from a grade 1 (without context) to a split between grades 1 and 2 when neuropathy was indicated for the response pattern. However, in general, clinician agreement regarding recommended CTCAE grade assignment improved when clinical information was added to the response patterns. This finding was consistent across all the response patterns we assessed in round 2, with the exceptions of neuropathy, abdominal pain, and chills.
TABLE 1.
Survey Round 2 Comparing Clinician Recommended CTCAE Grades Without and With Clinical Context Provided
| No Context Provided | Context Provided | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Pediatric PRO-CTCAE Response Pattern | CTCAE Grade | CTCAE Grade | |||||||||
| Frequency | Severity | Interference | 0 | 1 | 2 | 3 | Symptom | 0 | 1 | 2 | 3 |
| Bad (2) | Not at all (0) | 5% | 91% | 3% | 1% | Neuropathya | 1% | 41% | 58% | 0% | |
| Very bad (3) | Not at all (0) | 1% | 57% | 32% | 10% | Pruritusa | 0% | 72% | 26% | 2% | |
| Sometimes (1) | A whole lot (3) | 0% | 10% | 61% | 29% | Vomitinga | 0% | 33% | 67% | 0% | |
| Most of the time (2) | A lot (2) | 0% | 1% | 57% | 42% | Fecal incontinencea | 0% | 3% | 92% | 5% | |
| Almost all the time (3) | A lot (2) | 0% | 1% | 26% | 74% | Diarrheaa | 0% | 0% | 86% | 14% | |
| Most of the time (2) | Very bad (3) | 0% | 1% | 18% | 81% | Chillsa | 0% | 6% | 73% | 21% | |
| Almost all the time (3) | Bad (2) | 0% | 2% | 60% | 38% | Dyspepsiaa | 0% | 3% | 92% | 5% | |
| Sometimes (1) | Did not have (0) | A whole lot (3) | 9% | 24% | 48% | 19% | Headache | 0% | 17% | 81% | 2% |
| Sometimes (1) | Bad (2) | Some (1) | 0% | 41% | 57% | 2% | Headache | 0% | 24% | 76% | 0% |
| Most of the time (2) | Did not have (0) | A whole lot (3) | 8% | 11% | 49% | 32% | Arthralgia | 3% | 9% | 59% | 29% |
| Almost all the time (3) | Did not have (0) | A whole lot (3) | 7% | 9% | 38% | 46% | Mucositis oral | 1% | 9% | 63% | 27% |
| Almost all the time (3) | A little bad (1) | Some (1) | 0% | 42% | 56% | 2% | Constipation | 0% | 30% | 70% | 0% |
| Almost all the time (3) | Bad (2) | A lot (2) | 0% | 0% | 35% | 65% | Abdominal pain | 0% | 17% | 56% | 44% |
| Almost all the time (3) | Very bad (3) | Some (1) | 0% | 7% | 63% | 30% | Constipationa | 0% | 8% | 89% | 3% |
Abbreviations: CTCAE, Common Terminology Criteria for Adverse Events; PRO-CTCAE, Pediatric Patient-Reported Outcome version of the Common Terminology Criteria for Adverse Events.
Blank boxes indicate that this response pattern did not include the attribute. Response percentages ≥70% were used to indicate whether a single grade received a majority of clinician recommendations (shown in bold) or whether 2 grades were recommended (if the response percentage was <70%).
Indicates that the 2 response frequencies were statistically significant (α = .01) using a Wilcoxon signed rank test.
Within clinical vignettes, we paired scenarios and varied the treatment type (chemotherapy vs radiotherapy), type of symptom AE (nausea vs anxiety; edema vs fatigue), patient sex (boy vs girl), patient age (9 years vs 16 years), and type of cancer (Ewing sarcoma vs acute lymphoblastic leukemia). There were no statistically significant differences noted between any of the pairs in terms of the median clinician recommended CTCAE grade (P > .01).
DISCUSSION
We believe the current mapping study provides a framework for integrating child and adolescent self-report symptom AE data into mandated AE reporting in oncology trials. The overall study goal was to develop a standardized algorithm for translating Pediatric PRO-CTCAE response patterns into recommended CTCAE grades that would inform the clinicians’ AE grading. After 2 survey rounds with experienced pediatric clinicians, we herein have presented Pediatric PRO-CTCAE response patterns that are mapped to recommended CTCAE grades (see Supporting Table 3). Prior pediatric studies have shown a poor correlation between PROs and clinician perception of patient symptoms and side effects.8,18 As such, incorporating the patient’s voice via the Pediatric PROCTCAE could enhance clinician understanding of symptom AEs, and augment clinician-patient communication regarding the identification and treatment of therapy-related AEs.
In certain cases of low agreement, we recommended 2 possible grades, with the final decision to be made by the clinician after conversation with the patient. We believe this approach may be a strength in certain settings. For example, for a patient with neuropathy, there may be a subjective distinction if a patient presents with some symptoms that fall into grade 2 but some that fall into grade 3 (eg, tripping), and these decisions may be influenced by such things as the patient’s age or activity level. Our approach allows for more depth and flexibility for a given symptom AE, and the recommended CTCAE grade based on this algorithm should not be accepted blindly, but rather taken into consideration along with a thorough review with the child.
Clinicians for the most part were consistent with their grading, especially when clinical information was added to response combinations. The current study findings support the generalizability of the mapping algorithm. However, there were instances (Table 1) when clinician-assigned grades differed when context was provided compared with response patterns when no context was provided. In one case, adding the symptom of neuropathy to a generic response pattern resulted in a change in the most endorsed grade from grade 1 (91% endorsed with generic) to grade 2 (58% endorsed with neuropathy). Differences in grading are important because they may lead to dose reductions or delays in the provision of chemotherapy or other treatments. In certain cases, a difference of 1 grade could lead to the patient being removed from protocol therapy. As an example, grade 3 vincristine-related neuropathic pain may lead to dose modification of a drug that is an important component of curative therapy. Current use of CTCAE and the Balis scale both are subjective and difficult to assess in children.5,19 The current study finding that clinicians may be influenced in their grading of AEs by contextual information indicates that the addition of patient input via Pediatric PRO-CTCAE may better inform the clinician as to the true severity of the symptom AE and give guidance regarding the risk/benefit ratio in specific patients and their treatment. This information highlights how the Pediatric PRO-CTCAE system may enhance the accuracy of AE reporting in pediatric oncology trials. Presenting the child’s responses to the clinician has the potential to have a large impact, especially in trials for new novel agents with low-grade, chronic toxicities, or those for which the symptom pattern profile is unclear.
In addition, we found that clinicians tended to give greater weight to the symptom attribute of interference with daily activities compared with other attributes such as symptom severity or frequency. This finding suggests that clinicians perceive interference as an important factor when determining the tolerability of a drug, which is consistent with the CTCAE, which tends to shift grades higher if ADLs are impaired.
There were limitations to the current study. In the second round of surveys, we repeatedly asked about the same response patterns, but in different contexts, and it is possible that some clinicians may have remembered what grade they assigned the earlier pattern in the survey. However, we did not alert the clinicians that they would see the same response patterns repeatedly, and clinicians most likely were not focused on this survey as a memory test. In addition, although we made every effort to sample clinicians from different pediatric hospitals using different cancer treatment protocols, it is possible that the clinicians participating in the current survey may not be generalizable to the larger population of pediatric oncology clinicians. All these pediatric cancer hospitals also were academic cancer centers, which are the institutions at which the majority of children with cancer receive care, and at which most clinical trials take place.
After the current study, we will examine how this mapping algorithm performs using data we have collected from children and adolescents and their providers who have participated in our longitudinal study. We will evaluate how the recommended CTCAE grades from the current study correlate with actual CTCAE grades assigned by clinicians for children who are actively undergoing cancer treatment. In addition, we plan to evaluate the feasibility of the routine collection and use of the Pediatric PROCTCAE in pediatric oncology trials and clinical practice. Success will rely in part on how well the children’s reported data are presented to the clinicians. We recommend presenting both the child’s individual responses to the Pediatric PRO-CTCAE and the recommended CTCAE grade(s), based on this mapping algorithm, to the clinician to maximize understanding of the child’s experiences. We also encourage discussion between the clinician and the child to determine the final CTCAE grade.
Accurate AE monitoring is an essential component of trial safety, design, and outcomes because differences in AE grading can lead to dose modifications or removal of a patient from a trial. Equally important, better AE reporting will improve patient safety and enhance drug labeling accuracy. Although the CTCAE system was designed for use in oncology trials, it often is used in clinical care to guide treatment decisions, including drug dosing and supportive care interventions.20–22 The availability of both the child’s reported symptom AE experience from the Pediatric PRO-CTCAE system and the recommended CTCAE grade(s) will greatly promote consistent and valid data with which to inform decision making.
Supplementary Material
FUNDING SUPPORT
Supported by the National Cancer Institute of the National Institutes of Health under award number R01CA175759 and the University of North Carolina Lineberger Comprehensive Cancer Center’s Developmental Funding Program.
Footnotes
Additional supporting information may be found in the online version of this article.
CONFLICT OF INTEREST DISCLOSURES
Molly McFatrich, Pamela S. Hinds, Catriona Mowbray and Bryce B. Reeve were supported by grant R01CA175759 from the National Cancer Institute of the National Institutes of Health for work performed as part of the current study. Sharon M. Castellino has received research funding from Bristol-Myers Squibb for work performed outside of the current study. The other authors made no disclosures.
REFERENCES
- 1.Sateren WB, Trimble EL, Abrams J, et al. How sociodemographics, presence of oncology specialists, and hospital cancer programs affect accrual to cancer treatment trials. J Clin Oncol. 2002;20: 2109–2117. [DOI] [PubMed] [Google Scholar]
- 2.Bleyer A, Budd T, Montello M. Adolescents and young adults with cancer: the scope of the problem and criticality of clinical trials. Cancer. 2006;107(suppl 7):1645–1655. [DOI] [PubMed] [Google Scholar]
- 3.Hunger SP, Lu X, Devidas M, et al. Improved survival for children and adolescents with acute lymphoblastic leukemia between 1990 and 2005: a report from the Children’s Oncology Group. J Clin Oncol. 2012;30:1663–1669. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.National Cancer Institute. Common Terminology Criteria for Adverse Events (CTCAE) Version 5.0. Accessed October 8, 2018 https://ctep.cancer.gov/protocoldevelopment/electronic_applications/docs/CTCAE_v5_Quick_Reference_5x7.pdf
- 5.National Cancer Institute. Cancer Therapy Evaluation Program: Common Terminology Criteria for Adverse Events (CTCAE). National Cancer Institute; 2018. [Google Scholar]
- 6.Collins JJ, Devine TD, Dick GS, et al. The measurement of symptoms in young children with cancer: the validation of the Memorial Symptom Assessment Scale in children aged 7–12. J Pain Symptom Manage. 2002;23:10–16. [DOI] [PubMed] [Google Scholar]
- 7.Dupuis LL, Taddio A, Kerr EN, Kelly A, MacKeigan L. Development and validation of the pediatric nausea assessment tool for use in children receiving antineoplastic agents. Pharmacotherapy. 2006;26: 1221–1231. [DOI] [PubMed] [Google Scholar]
- 8.Hockenberry MJ, Hinds PS, Barrera P, et al. Three instruments to assess fatigue in children with cancer: the child, parent and staff perspectives. J Pain Symptom Manage. 2003;25:319–328. [DOI] [PubMed] [Google Scholar]
- 9.Zhukovsky DS, Rozmus CL, Robert RS, et al. Symptom profiles in children with advanced cancer: patient, family caregiver, and oncologist ratings. Cancer. 2015;121:4080–4087. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Reeve BB, McFatrich M, Pinheiro LC, et al. Eliciting the child’s voice in adverse event reporting in oncology trials: cognitive interview findings from the Pediatric Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events initiative. Pediatr Blood Cancer. 2017;64(3). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Reeve BB, Withycombe JS, Baker JN, et al. The first step to integrating the child’s voice in adverse event reporting in oncology trials: a content validation study among pediatric oncology clinicians. Pediatr Blood Cancer. 2013;60:1231–1236. [DOI] [PubMed] [Google Scholar]
- 12.Conger AJ. Integration and generalization of kappas for multiple raters. Psychol Bull. 1980;88:322–328. [Google Scholar]
- 13.Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213–220. [DOI] [PubMed] [Google Scholar]
- 14.Vach W. The dependence of Cohen’s kappa on the prevalence does not matter. J Clin Epidemiol. 2005;58:655–661. [DOI] [PubMed] [Google Scholar]
- 15.Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–428. [DOI] [PubMed] [Google Scholar]
- 16.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
- 17.Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6:284–290. [Google Scholar]
- 18.Parsons SK, Barlow SE, Levy SL, Supran SE, Kaplan SH. Health-related quality of life in pediatric bone marrow transplant survivors: according to whom? Int J Cancer Suppl. 1999;12:46–51. [DOI] [PubMed] [Google Scholar]
- 19.Postma TJ, Heimans JJ. Grading of chemotherapy-induced peripheral neuropathy. Ann Oncol. 2000;11:509–513. [DOI] [PubMed] [Google Scholar]
- 20.Zhang S, Liang F, Tannock I. Use and misuse of Common Terminology Criteria for Adverse Events in cancer clinical trials. BMC Cancer. 2016;16:392. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Kuderer NM, Wolff AC. Enhancing therapeutic decision making when options abound: toxicities matter. J Clin Oncol. 2014;32:1990–1993. [DOI] [PubMed] [Google Scholar]
- 22.Edgerly M, Fojo T. Is there room for improvement in adverse event reporting in the era of targeted therapies? J Natl Cancer Inst. 2008;100:240–242. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.