Eliciting the child’s voice in adverse event reporting in oncology trials: Cognitive interview findings from the Pediatric Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events initiative

Abstract

Background

Adverse event (AE) reporting in oncology trials is required, but current practice does not directly integrate the child’s voice. The Pediatric Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) is being developed to assess symptomatic AEs via child/adolescent self-report or proxy-report. This qualitative study evaluates the child’s/adolescent’s understanding and ability to provide valid responses to the PRO-CTCAE to inform questionnaire refinements and confirm content validity.

Procedure

From seven pediatric research hospitals, children/adolescents ages 7–15 years who were diagnosed with cancer and receiving treatment were eligible, along with their parent-proxies. The Pediatric PRO-CTCAE includes 130 questions that assess 62 symptomatic AEs capturing symptom frequency, severity, interference, or presence. Cognitive interviews with retrospective probing were completed with children in the age groups of 7–8, 9–12, and 13–15 years. The children/adolescents and proxies were interviewed independently.

Results

Two rounds of interviews involved 81 children and adolescents and 74 parent-proxies. Fifteen of the 62 AE terms were revised after Round 1, including refinements to the questions assessing symptom severity. Most participants rated the PRO-CTCAE AE items as “very easy” or “somewhat easy” and were able to read, understand, and provide valid responses to questions. A few AE items assessing rare events were challenging to understand.

Conclusions

The Pediatric and Proxy PRO-CTCAE performed well among children and adolescents and their proxies, supporting its content validity. Data from PRO-CTCAE may improve symptomatic AE reporting in clinical trials and enhance the quality of care that children receive.

1 | INTRODUCTION

In 2016, over 10,000 children under 15 years of age will be diagnosed with a new cancer in the United States.¹ Contrary to low clinical trial participation rates in adults, over 60% of children and adolescents with cancer participate in a trial.² Enrollment in clinical trials at initial diagnosis has become the standard of care in pediatric oncology in the United States.³

It is mandatory in clinical trials that adverse events (AEs) be collected and reported. In oncology, the lexicon for AE grading is the National Cancer Institute’s Common Terminology Criteria for Adverse Events (CTCAE), which consists of 790 AE terms.⁴ While many AEs are graded based on laboratory values or clinical measurements, 62 AEs are symptoms (e.g., pain, fatigue, depression) that are subjective in nature.⁵ Evidence from multiple studies suggests that clinicians’ and parents’ ratings of children’s symptoms do not reflect children’s self-reported experiences.^6–12 Specifically, previous studies have found poor agreement between what children and parents/clinicians report, with parents/clinicians more often underreporting the burden of cancer and treatment on the lives of the children and adolescents.^6,7,9,10 Even worse, AE grading is based on what is documented in patient charts and symptoms are thus more likely missed compared with clinical or laboratory results.¹³ Despite these important findings, clinicians continue to routinely grade symptomatic AEs in pediatric oncology trials.

Our overall study is designed and seeks to validate a system for children and adolescents to self-report on the symptomatic AEs they experience while undergoing cancer treatment. The endgoal is to enhance the precision of AE grading in pediatric oncology trials and improve the healthcare for the children. This study extends previous work initiated by the NCI to design the Patient-Reported Outcomes version of the CTCAE (PRO-CTCAE) for adults in oncology trials.^14–16 The PRO-CTCAE system is different from the standard outcome measures used in research, such as the Patient-Reported Outcomes Measurement Information System^® (PROMIS^®), in that it screens patients over a broad range of symptom toxicities with the goal of informing CTCAE grading. Compared with other PRO measures, the PRO-CTCAE uses a small set of questions to identify the worst severity, frequency, and interference of symptom toxicities on the daily lives of trial participants. Envisioned use of the PRO-CTCAE includes AE detection/screening, support of dose-finding work, and assessment of comparative tolerability for product labeling claims.¹⁶

Based on our previous study among 187 pediatric clinicians from seven pediatric cancer centers, the draft pediatric and proxy versions of the PRO-CTCAE were designed to assess up to 62 symptomatic AEs.⁵ The main goal of this cognitive interview study was to establish, evaluate, and refine the PRO-CTCAE measures to be comprehensible to children and their caregivers and relevant for capturing AEs. The second goal was to stratify children into different age groups that represent different developmental stages to determine if 7–8-year-old children can understand and respond to the Pediatric PRO-CTCAE and at what age can adolescents transition to the already developed Adult PRO-CTCAE measure. Cognitive interviewing is a necessary step in the design and validation of a PRO measure¹⁷ and recommended by the Food and Drug Administration.¹⁸

2 | METHODS

2.1 | Participants and setting

Seven pediatric research hospitals participated: Children’s Hospital Los Angeles, Children’s National Health System (Washington, District of Columbia), Hospital for Sick Children (Toronto, Ontario, Canada), Dana-Farber Cancer Institute and Boston Children’s Hospital, Palmetto Health Children’s Hospital (Columbia, South Carolina), St. Jude Children’s Research Hospital (Memphis, Tennessee), and the University of North Carolina (UNC). These sites provided access to a diverse population of children in terms of demographics, cancer types, and treatment modalities. The UNC was the coordinating center. All sites received approval from their institutional review boards.

Children and adolescents ages 7–20 years who were diagnosed with a cancer of any type and receiving treatment within or outside the context of a clinical trial were eligible to participate. The child’s parent or caregiver (from here on referred to as “proxy”) must be at least 18 years of age to participate. Also, both the children and proxies must be able to speak English and to report their/their children’s symptom AEs. Ideally, both the child and her/his proxy should participate in the study; however, there were a limited number of times when only the child or the proxy participated, based on their preference. Participants ages 18 years or older provided their own signed consent and children younger than 18 years provided the assent.

2.2 | Measures

The Pediatric and Proxy versions of the PRO-CTCAE consist of a library of items to assess up to 62 symptomatic AEs included in the CTCAE. Based on the research team’s expertise in survey design and review of the literature, CTCAE medical terminology was translated into child-friendly terms (e.g., epitaxis = nose bleeds). Questions were developed to capture the child’s symptom experience. For a given AE, one to three questions were created to reflect attributes of the symptom experience including frequency, worst severity, interference with daily activities, or presence. Table 3 provides examples of each of these question types. In total, 130 questions were drafted for the Pediatric PRO-CTCAE. The Proxy version mirrored the child self-report version except “you” was replaced with “your child.” The Adult PRO-CTCAE was tested among 13–15- and 16–20-year-olds in Round 1; findings of the Adult PRO-CTCAE are reported elsewhere.

TABLE 3.

Final wording of Pediatric PRO-CTCAE question stems

Attribute	Sample question	Response options
Frequency	In the past 7 days, how often did your head hurt     (headache)?	Never/sometimes/most of the time/almost all the time
Severity	In the past 7 days, how bad was your sore throat?	Did not have any/a little bad/bad/very bad
Interference	In the past 7 days, how much did your itchy skin keep     you from doing things you usually do?	Not at all/some/a lot/a whole lot
Presence	In the past 7 days, did you have any changes in your     voice?	Yes/no/I do not know

The selected reference period for the Pediatric PRO-CTCAE questions balanced the need to minimize both recall bias (from long periods of memory) and burden from the child having to complete numerous repeated assessments to ensure there was no missing time gap in a longitudinal study. “In the past 7 days” was selected as the reference period, consistent with other validated pediatric PRO measures.^19,20

2.3 | Cognitive interviewing procedures

One-on-one interviews with the child/adolescent and the proxy had two parts. The first part elicited concepts and terminology from participants in a free-form format (i.e., without viewing the questionnaire). Participants were asked to discuss symptoms and other health concerns they had experienced in the last 7 days. Results from the concept-elicitation phase are published elsewhere and the language used by the children to describe symptoms in the free-form format informed the refinement of the Pediatric PRO-CTCAE questions tested in Round 2 of interviews.²¹ The second part of the cognitive interviews evaluated the draft Pediatric and Proxy PRO-CTCAE questionnaires using semistructure interview probes. Probes included questions such as, “How would you describe <symptom>?” or “In your own words, what do you think this question is asking?” Through this process, we obtained the feedback on the wording of the items, response options, and reference period.

We stratified interviews by age group (7–8, 9–12, 13–15, and 16–20 years) to represent distinct developmental stages. We performed two rounds of cognitive interviews. In Round 1, participants aged 7–12 years and their proxies completed the Pediatric or Pediatric-Proxy PRO-CTCAE. Participants aged 13–20 years and their proxies completed the Adult or Adult-Proxy PRO-CTCAE. Because of challenges understanding the Adult PRO-CTCAE measure in Round 1, the 13–15-year-old group completed the Pediatric PRO-CTCAE measures in Round 2.

In Round 1, children and proxies evaluated the 62 PRO-CTCAE symptom AEs (130 questions). To reduce respondent burden, we divided the 62 AEs among four forms with the goal of having at least six participants in each age group complete each form. Sample sizes of six participants per item are consistent with the sample sizes recommended by the NIH’s PROMIS initiative²² and other guidelines for conducting cognitive interviews.^17,23

Round 2 aimed to review items that were substantially revised based on findings from Round 1. Items are defined as being “substantially revised” if their revision involved the following: (i) adding or removing a word(s) that changed the meaning of a phrase; (ii) word substitutions that in the judgment of the investigators were more than a semantic simplification; or (iii) significant changes to the response options. In Round 2, 21 AEs (48 questions) were evaluated using two questionnaires.

We employed a retrospective probing method for cognitive interviews,²³ in which participants first completed a paper questionnaire on their own and were then asked by trained interviewers to explain their responses to particular items (Fig. 1). When possible, children and proxies were interviewed in separate spaces so as to not influence the other’s responses. During questionnaire completion, participants were asked to mark items they found hard to understand. As participants completed the questionnaires, the interviewers noted in their field notes if help or clarification was given. Although participants were encouraged to complete the questionnaires individually, interviewers assisted children in reading the items, when necessary. Upon the completion of the interviews, each participant received a $25 gift card.

FIGURE 1.

Flow of cognitive interviews for participants

To standardize procedures and ensure consistency across sites, interviewers were trained during a 1-day, in-person workshop led by a cognitive interview methods expert. Prior to the initiation of Round 2, all interviewers received a refresher training. Interviewers and site lead investigators participated in weekly team calls to discuss interview experiences, findings, recommendations, and ongoing recruitment progress.

2.4 | Analytic approach

With consent from the proxy and child, each cognitive interview was digitally audio-recorded. Recordings were transcribed by a professional service. In addition, the interviewers wrote detailed field notes during and after the interviews to aid in the preparation of an interview summary. Following the completion of a cognitive interview, the interviewers entered the data into a REDCap database. In REDCap, the interviewers reported on child and proxy demographics, item-by-item responses to each question including items participants marked as hard to understand. Additional interviewer comments regarding participant’s body language, facial expressions, perceived attitude, problems raised, and interview duration were also entered into REDCap.

Interview notes and participant responses were organized by each AE item to summarize participants’ experiences with the PRO-CTCAE questionnaire. Upon completion of each round, representatives from participating sites, including interviewers, physicians, nurses, and PRO methodologists, reviewed the summarized the data by age group to evaluate the participants’ understanding of the AE items, question stems, response options, and instructions. This process included reviewing participant-marked hard to understand items and items that interviewers identified (through discussion) as hard to understand. Transcripts were further reviewed to inform findings. When assessing comprehensibility, more weight was given to items when more than one child had difficulties with the item; however, all questions were discussed. Overall summaries were created and approved by representatives from each site, including recommendations of modifications and approval of items for the final version of the questionnaire.

3 | RESULTS

3.1 | Participant characteristics

In Round 1, 54 children (7–12-year-olds) were approached, nine refused and 45 children and 42 proxies participated in the interviews. Twenty children were in the 7–8-year-old group and 25 were in the 9–12-year-old group (Table 1).

TABLE 1.

Characteristics of child and adolescent participants in Rounds 1 and 2 of cognitive interviews

	Round 1 (n = 45)				Round 2 (n = 36)
	7–8 Years		9–12 Years		7–8 Years		9–12 Years		13–15 Years
	n = 20	%	n = 25	%	n = 12	%	n = 12	%	n = 12	%
Female	9	45	17	68	5	42	5	42	5	42
Hispanic ethnicity	4	20	5	20	3	25	4	33	1	8
Race
White	13	65	15	60	5	42	5	42	7	58
Black	1	5	5	20	2	17	4	33	1	8
Asian	2	10	3	12	2	17	0	0	2	17
Other	4	20	2	8	3	25	3	25	2	17
Inpatient	10	50	15	60	4	33	7	58	6	50
Cancer type
Sarcoma	2	10	6	24	1	8	7	58	2	17
Leukemia	10	50	12	48	7	58	5	42	6	50
Lymphoma	4	20	4	16	4	33	0	0	3	25
Other solid tumor	4	20	2	8	0	0	0	0	1	8
Brain tumor	0	0	1	4	0	0	0	0	0	0

For Round 2, 51 children and adolescents were approached and 15 refused. Thirty-six children and adolescents and their proxies participated (12 in each age group 7–8, 9–12, and 13–15 years) to evaluate “substantially revised” items and the revised symptom severity phrasing. Smaller samples participated in Round 2, as fewer questions were evaluated; however, our minimum of six children reviewing each item was maintained.

3.2 | Assessment of symptom AE terms

Round 1 included 7–8- and 9–12-year-olds evaluating the Pediatric PRO-CTCAE. Of the 62 AE terms evaluated, at least one 7–8-year-old experienced difficulty with 20 AE terms; however, only six of the 20 had two or more children experiencing difficulties, such as knowing what the terms meant or accurately defining it in their own words. Among the 9–12-year-olds, at least one child had difficulties with 22 AE terms; 13 of the 22 identified items had two or more children having difficulty. Within the 9–12-year-old group, 9- and 10-year-olds had greater difficulties compared with 11- and 12-year-olds. Online Supplementary Table S1 identifies the AE terms from Round 1 that were revised and evaluated in Round 2.

Round 2 included 7–8-, 9–12-, and 13–15-year-olds evaluating the items revised or needed further review by more children. Of the 21 AE terms included in Round 2, 13 had at least one 7–8-year-old who experienced difficulty with the term. Nine of the 13 identified items had two or more children having difficulty. Among the 9–12-year-olds, at least one child had difficulties with nine AE terms, but two or more children had difficulty with two of these nine terms. Among the 13–15-year-olds, only three items were hard to understand by one or more adolescents, and two or more adolescents had difficulty with two of these three items. Final wording of the AE terms are given in Table 2.

TABLE 2.

Final wording of Pediatric PRO-CTCAE symptomatic adverse event terms

Arms and legs feel weak/weakness     in your arms and legs	Feeling tired	Pain in your mouth or throat	Sad or unhappy feelings
Bigger belly than usual	Food or drink taste different than     usual	Pain or burning when you pee	Flashes of light that were not there     when your eyes were open or     closed
Bruise easily (get black and blue     marks on your skin)	Hair fall out	Pee more than usual	See blurry (have blurry vision)
Burning feeling in your chest (heart     burn)	Head hurt (headache)	Pee yourself on accident	Shaking chills
Change in the color of your pee	Hiccups	Pimples (bumps on the face or chest)	Sneezing
Changes in your voice	Hoarse (scratchy) voice	Poop yourself on accident	Sore throat
Cough	Itchy red bumps on your skin	Problems breathing (shortness of     breath)	Stomach pain
Dizziness	Itchy skin	Problems remembering things	Sunburn more easily
Dry eyes	Muscles hurt	Problems sleeping (trouble falling or     staying asleep)	Sweat more than usual or sweat for     no reason
Dry mouth	Nose bleeds	Problems with not being able to     poop	Think about hurting yourself
Dry skin	Not being able to sit still	Problems with paying attention     (focusing on TV, reading, or school     work)	Throw up
Fall down	Not want to eat your meals	Problems with swallowing	Watery eyes (tearing)
Fart more than usual	Numbness or tingly feeling in your     hands or feet	Puffiness (swelling) in your arms,     hands, legs, or feet	Wheezing (a whistling noise in your     chest when you breathe)
Feel hot all of a sudden (hot flashes)	Open sores or red spots on your skin	Racing heart beat	Worried or nervous feelings
Feel like you could not wait to pee	Pain	Ringing or buzzing in your ears
Feeling sick to your stomach     (nausea)	Pain in any bendable part of your     body (knees, ankles, shoulders, or     fingers)	Runny or watery poop

3.3 | Assessment of question structure and response options

In Round 1, children and adolescents had no difficulty in reading and providing responses to questions related to AE frequency, presence, or interference. However, the severity stem (How bad was the worst <symptom>?) was challenging across both 7–8 and 9–12 age groups. Children had difficulty describing the concept of the “worst” symptom experience and preferred discussing symptom severity in terms of how “bad” the symptom was. Also, children in both age groups interchanged concepts of frequency and interference when describing severity and the corresponding response options. As such, we changed the severity stem in Round 2 to reflect the preference for “bad” to express severity. We also changed one response option (from “Alittle” to “Alittle bad”) to better flow with the severity stem change. In Round 2, the new severity stem and response options were well understood across the three age groups. Final wording of the question types is given in Table 3.

3.4 | Assessment of recall period

In general, among 9–12-year-olds (Rounds 1 and 2) and 13–15-year-olds (Round 2), children/adolescents did not have any challenges with the 7-day reference period. Although some children described longer and some described shorter than the specified recall periods, most of them accurately defined a 7-day period. Among the youngest age group (7–8-year-olds), the reference period was generally a difficult concept in both rounds, with many children not accurately defining the appropriate recall period.

3.5 | Children’s overall rating of survey

Generally, the 7–8- and 9–12-year-olds had an easy time completing the Pediatric PRO-CTCAE instrument in both Rounds 1 and 2. Of the 45 children aged 7–12 years who participated in Round 1, 23 described it as “very easy,” 18 as “somewhat easy,” and four as “somewhat hard” to answer most of the questions. The four children who marked it as “somewhat hard” were in the 7–8-year-old group. Of the 24 children who were 7–12 years old in Round 2, 15 marked the questionnaire as “very easy,” seven marked as “somewhat easy,” and two as “somewhat hard.” Both of the children who marked the questionnaire as “somewhat hard” were in the 7–8-year-old group. Of the 12 adolescents (13–15-year-olds) in Round 2, eight marked the questionnaire as “very easy” and four as “somewhat easy.”

3.6 | Proxy findings

3.6.1 | Round 1

Of the 42 proxies who completed the Proxy PRO-CTCAE measure, 31 described it as “very easy” and 11 as “somewhat easy” to answer most of the questions. Of the 63 AEs assessed in Round 1, 13 AEs were identified as difficult by at least one proxy. Of these 13 AEs, only three had at least two proxies experiencing difficulties with. Proxies found certain AEs including “flashing lights,” “swollen belly,” and “hot flashes” difficult to understand, which were also the three AEs that the largest number of 7–12-year-old participants reported having difficulty understanding.

Similar to their children, proxies had some difficulty with the severity stem, interchanging the concept with both frequency and interference. However, both the interference and frequency stems seemed to work well. Most proxies had no trouble describing the 7-day reference period.

3.6.2 | Round 2

In Round 2, 32 proxies participated in cognitive interviews, with 21 participants describing the questionnaire as “very easy” and 11 as “somewhat easy.” Of the 21 AEs evaluated in the second round, eight were identified as difficult to understand by at least one proxy participant. Of these eight items, four (wheezing, tinnitus, hot flashes, and flashing lights) had at least two participants experiencing difficulties with. Overall, the revised severity stem worked well in Round 2.

4 | DISCUSSION

This cognitive interview study represents a critical step in the refinement and content validation of the Pediatric and Proxy PRO-CTCAE.¹⁷ Most questions in the Pediatric and Proxy PRO-CTCAE library were well understood by the child and adolescent participants, and their proxies.

This study revealed important age-related findings relevant for the application of PRO measures in the clinical research. Younger children experienced greater difficulties for some types of questions. While the majority of 7-year-olds (nine of 13) and eight of the 19 8-year-olds needed assistance from the interviewer to read the questionnaire to them, older children did not need this help. Results may have differed for children having the questionnaire read to them; however, we believe they are still able to self-report their symptom experiences. Younger children also interpreted terms used in questions more literally than older children did, see AEs of abdominal distention and flashing lights (eye disorder) in online Supplementary Table S1 for examples.

Another age-related issue was the 7-day reference period. A majority of the 7–8-year-olds had difficulty describing what “In the past 7 days” meant. Older children typically counted off the days of the week, while many of the youngest would only list a few days. This finding is consistent with prior studies’ findings that declarative memory is less developed in 7–8-year-olds than in older children.^24,25 Future evaluation of the Pediatric PRO-CTCAE in younger age groups may consider shortening the reference period, as younger children cognitively may struggle with the concept of time.

There were some symptomatic AE terms that were difficult for participants (child or parent) to understand if they had not personally experienced the symptom. However, children and proxies who had experienced the symptom did not struggle to understand what concepts we were referring to and felt that the questions were worded correctly. For example, flashing lights was an AE that was not understood by many participants, but in Round 2 of cognitive interviews, we interviewed children who had experienced this symptom and approved of the wording. We relied on these children’s experiences when deciding on the final wording of the questions. For each question in the Pediatric PRO-CTCAE, we believe that if a patient has experienced the AE they would be able to accurately report it with the current wording. In addition, the library is structured so that participants would only be administered items in specific studies where the AE is expected.

This study had limitations. While the sample sizes are consistent with the recommended guidelines,^22,23 we lacked adequate representation in key subgroups including children with central nervous system tumors. We likely did not interview the sickest of the children, as in-depth interviewing would have been too difficult for the child to maintain his or her focus. In addition, because our sample was limited to children undergoing treatment, several children were ill and fatigued, which may have limited their full attention during the interviews. To reduce respondent’s burden and facilitate participation, we included breaks and provided all participants with a squeeze toy. Although children out of treatment may have been more attentive, we wanted participants that were similar to children who would complete the Pediatric PRO-CTCAE in the future (i.e., those in a clinical trial). In addition, these children in our study were better able to describe AE symptoms because they had recently undergone treatment and experienced symptoms as a result. Finally, some children preferred to have their parents in the room during the interview, which could have impacted their responses.

This study is one of the first to employ cognitive interviewing in such a large sample of children undergoing cancer treatment. We also stratified interviews by age group to ensure representation among all ages of children, and this design allowed us to interview a number of 7- and 8-year-olds. Participants were diverse in terms of race/ethnicity, geography, gender, cancer type, treatment protocol, and treatment setting. Additionally, all of the cognitive interview audio recordings were transcribed, which contributed to consistent and thorough data analysis.

As a result of two rounds of cognitive interviewing, the Pediatric and Proxy PRO-CTCAE performed well, especially among older children in our study. When a 9-year-old girl was asked about why she said the survey was “very easy,” she responded, “Because it was mainly stuff about myself and I know everything about myself.” Questions on prevalent symptoms such as fatigue, depression, pain, headache, vomiting, and cough were well understood, as well as those on the less-common ones like dry skin or dizziness. This study served as an exploration into the ability of children at different developmental stages to be able to read, understand, and report on symptoms included in the PRO measures. As such, this study will inform future self-reported symptom assessments of children in clinical trials. Our next steps are to assess the construct validity of the Pediatric and Proxy PRO-CTCAE measures in a longitudinal multisite study. This will allow us to evaluate the instrument’s ability to detect changes in symptom status over time and compare self-report AEs with relevant clinical anchors. The availability of the Pediatric PRO-CTCAE measures will improve the way symptomatic AEs are reported and graded in clinical trials and enhance the quality of care that children receive by making providers more aware of problematic symptoms children are experiencing.

Abbreviations

AE

adverse event

CTCAE

Common Terminology Criteria for Adverse Events

PRO-CTCAE

Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events

PROMIS

Patient-Reported Outcomes Measurement Information System

PRO

Patient-Reported Outcomes

UNC

University of North Carolina