Cognitive interviewing of the U.S. National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)

Abstract

Purpose

The National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) is a library of question items that enables patient reporting of adverse events (AEs) in clinical trials. This study contributes content validity evidence of the PRO-CTCAE by incorporating cancer patient input of the relevance and comprehensiveness of the item library.

Methods

Cognitive interviews were conducted among patients undergoing chemotherapy or radiation therapy at multiple sites to evaluate comprehension, memory retrieval, judgment, and response mapping related to AE terms (e.g., nausea); attribute terms (regarding frequency, severity, or interference); response options; and recall period. Three interview rounds were conducted with ≥20 patients completing each item per round. Items were modified and retested if ≥3 patients exhibited cognitive difficulties or if experienced by ≤25% patients.

Results

127 patients participated (35% ≤high school; 28% non-white; 59% female). Most AE terms (63/80) generated no cognitive difficulties. The remaining 17 were modified without further difficulties by Round 3. Terms were comprehended regardless of education level. Attribute terms and response options required no modifications. Patient adherence to recall period (7-days) was improved when the reference period was incorporated.

Conclusions

This study provides evidence confirming comprehension of the U.S. English language versions of items in the PRO-CTCAE library for measuring symptomatic AEs from the patient perspective within the context of cancer treatment. Several minor changes were made to the items to improve item clarity, comprehension, and ease of response judgment. This study helps establish the content validity of PRO-CTCAE items for patient reporting of AEs during cancer treatment.

INTRODUCTION

The Common Terminology Criteria for Adverse Events (CTCAE) is a library of discrete adverse event (AE) measures used for investigator reporting of AEs in cancer treatment trials [1,2]. CTCAE is maintained by the U.S. National Cancer Institute (NCI), and its use is mandated in all NCI-sponsored trials. The CTCAE is also used in industry-sponsored trials and drug labels for oncology products, and is commonly employed by practitioners for toxicity assessment.

The CTCAE version 4 includes 790 measures representing both observable AEs that are dependent on specialized equipment or clinical training for reporting (e.g., decreased neutrophil count or retinal tear) and symptomatic AEs that are based on patients’ subjective experiences (e.g., nausea or peripheral sensory neuropathy). Currently, all CTCAE measures are reported by clinical investigators.

However, there is evidence that clinician reporting of patients’ symptomatic AEs is unreliable and under-detects the prevalence and severity of these events compared to patient self-reporting, both at baseline and across the course of a trial [3–9]. Patient self-reporting of symptomatic AEs is feasible even among those with substantial disease burden and impaired performance status [10,11], and there is increasing interest in assessing patient-reported outcomes (PROs) to gauge therapeutic response and treatment toxicity in both clinical trials and observational studies [12–26].

Therefore, in 2008 the NCI contracted to create a measurement system based on 77 items initially identified as being amenable to patient self-report from the CTCAE, called the PRO-CTCAE [27]. The central difference is that the CTCAE was designed to be reported by clinicians, whereas the PRO-CTCAE was designed specifically for patient self-report. Up to three PRO items were developed for each AE to individually measure the attributes of frequency, severity, and/or interference with activities (to mirror inclusion of those attributes in the CTCAE itself). All items were assigned a 7-day recall period based on evidence that memory about symptom events degrades over time [28], and expert consensus. This initial phase of PRO-CTCAE measure development was overseen by a committee including NCI representatives and consisted of identifying lay terms for each identified AE and developing generic item structures. Patient input was provided at every step of measure development. U.S. Food and Drug Administration representatives participated on the study committee as observers. Reviews of published literature and existing datasets were used to identify terms and item structures previously demonstrated to be well understood and accepted by populations of diverse race/ethnicity and low literacy, and amenable to linguistic adaptation [29,30]. The development of PRO-CTCAE items predated this cognitive interviewing study, and is reported elsewhere [31].

This paper describes the subsequent phase of PRO-CTCAE measure development through cognitive interviews. This consists of an iterative process in which successive rounds of semi-structured patient interviews and item refinement are conducted [32]. This process ensures that the content of items captures the most important aspects of the concepts of interest and that respondents understand how to complete the items, how to reference the correct recall period, the meaning of the items, and how to use the response option scales [33].

Cognitive interviews provide essential evidence in support of the content validity of a PRO measure for a particular purpose [33]. Content validity is defined as evidence that an instrument measures a concept of interest relative to an intended context of use [12]. In the case of the PRO-CTCAE item library, the intended context of use is AE or toxicity reporting in cancer clinical trials across diagnoses, stages, and treatment types. Envisioned applications include general AE detection/screening; support of dose-finding; and assessment of comparative tolerability for product labeling purposes. Due to this broad context of potential use across cancer types, as well as the large number of items in the PRO-CTCAE item library compared to most PRO instruments, a multicenter cognitive interviewing study design was developed including more patients than commonly included in such evaluations, with enrichment for lower educational levels.

METHODS

Participants and setting

Study participants represented a sample of patients receiving cancer treatment at one of four different cancer centers and their affiliated community network sites: Dana-Farber Cancer Institute, Duke University Medical Center, M.D. Anderson Cancer Center, and Memorial Sloan-Kettering Cancer Center. The study (clinicaltrials.gov; NCT00909207) was reviewed and approved by the Institutional Review Boards at each study site and at the NCI. Patients provided written informed consent to participate in this study. A sample size was determined to allow each PRO-CTCAE item to be assessed by 20 or more participants per round, exceeding usual practice concerning sample sizes in most cognitive interviewing research [34,35].

Patients were eligible for the study if they were age 18 years or older, receiving chemotherapy or radiotherapy with curative or palliative intent for an advanced cancer at one of the study sites, and able to speak and understand English. Patients were excluded if their physician determined that they were unable to report on their symptoms from the last seven days due to cognitive impairment. A purposive sampling strategy was used to ensure the inclusion of individuals with a diverse range of cancer diagnoses, self-reported race/ethnicity, gender, and age. Recruitment of at least 25% of patients with lower levels of educational attainment (i.e., ≤ high school) was prioritized.

Cognitive interview goals and procedure

Interviewers at each of the four study sites were trained through face-to-face, full-day sessions led by the Survey and Data Management Core of the Dana-Farber/Harvard Cancer Center. Interviewers were Bachelor’s or Master’s level research associates with at least two years of field experience interviewing patients for qualitative medical research. All interviewers participated in biweekly conference calls to discuss study-wide interviewer issues and ongoing recruitment.

Semistructured cognitive interviews were conducted to evaluate patients’ comprehension of PRO-CTCAE items and the extent to which items correspond to concepts of interest. Verbal probes were designed to elicit feedback about the following four areas: 1) symptomatic AE terms; 2) attribute terms – i.e., regarding severity, frequency, interference; 3) 7-day recall period; and 4) response options.

For each area, we considered four categories related to cognitive processing: comprehension, memory retrieval, judgment, and response mapping. Comprehension includes processes involved in attending to the question and instructions, assigning an accurate meaning to the question, inferring the objective of the question, and linking the terms used to the relevant concepts needed to answer the question. Memory retrieval involves adoption of a retrieval strategy, generating specific retrieval cues to trigger recall, and recollecting memories relevant to formulating a response. Judgment involves the processes used to combine or supplement memory, such as judgments concerning whether memory retrieval is complete and accurate, the utilization of inferences that fill in gaps in recall, the integration of memories into an overall judgment, and utilization of estimates that adjust for omissions in retrieval. Response mapping involves the process by which these judgments are mapped onto the response categories provided [36–40]. See Appendix I (online materials) for a complete set of cognitive interviewing probes sorted by concept of interest.

The cognitive interviews employed retrospective probing, in which participants first completed a paper questionnaire in full and were then asked about their responses [32]. During completion of a questionnaire, patients could mark items they viewed as difficult to answer or understand. This approach allows for the usual flow of the response process followed by discussion of problematic items. As the patient completed the questionnaire, if help or clarification was requested for any item, the interviewer encouraged the patient to complete the item based on the written instructions and made a note to subsequently probe this item.

Three iterative rounds of cognitive interviews were planned, with allowance to add rounds if problematic items remained after Round 3. In Round 1, participants completed one of three distinct questionnaire scripts, each of which included a subset of 20–25 AEs, from the total PRO-CTCAE library (Table 1). AEs representing gender-specific genitourinary phenomena were included in a male-version and in a female-version of Script 3, respectively (e.g., erectile dysfunction for men only; irregular menses for women only), and were administered only to participants of the pertinent gender and by an interviewer of the same gender as the participant.

Table 1.

Adverse events for which items were assessed in each round of cognitive interviews, listed by PRO-CTCAE terms used in each respective round.

Round 1, Script 1 N = 22 (12 males, 10 females)	Round 1, Script 2 N = 22 (11 males, 11 females)	Round 1, Script 3 N = 16 (9 males, 7 females)
Aching joints	Acne	Change in the color of your fingernails or toenails
Aching muscles	Arm or leg swelling	Loss of fingernails or toenails
Blurred vision	Bloating of the abdomen (belly)	Problems with concentration
Changes in sense of taste or problems tasting food	Body odor more than usual	Rash
Constipation	Breast area enlargement	Ridges or bumps on your fingernails or toenails
Cough	Bruising easily	Ringing in the ears
Decreased appetite	Decreased sweating	Shaking chills/shivering
Decreased sexual interest	Difficulty swallowing	Shortness of breath
Diarrhea	Felt like nothing could cheer you up	Skin burn from radiation
Difficulty sleeping (insomnia)	Felt unhappy	Skin cracking at the corners of your mouth
Dizziness	Flashing lights in front of your eyes	Spots or lines (floaters) that drift in front of your eyes
Dry mouth	Hair loss	Stretch marks
Dry skin	Hand-foot syndrome (a rash of the hands or feet that can cause cracking, peeling, redness or pain)	Sweats (excessive sweating)
Fatigue (tiredness, lack of energy)	Hives	Tremor, shaking
Feeling anxiety	Hot flashes	Unusual darkening of the skin
Heartburn	Increased passing of gas (flatulence)	Urine color change
Hiccups	Increased skin sensitivity to sunlight	Voice changes (hoarseness, loss of voice)
Increased need to urinate	Nosebleeds	Vomiting
Itchy skin	Pain in the abdomen (belly area)	Watery eyes (tearing)
Loss of control of urine (leakage)	Pain or burning with urination	Wheezing
Loss of control of stool or bowels	Pain, swelling, or redness at a site of drug injection or IV	Difficulty getting or keeping an erection (male only)
Mouth or throat sores	Pounding or racing heartbeat	Ejaculation problems (male only)
Nausea	Pressure or bed sores	Absence of menstrual period (female only)
Numbness or tingling in hands or feet	Problems feeling orgasm or climax	Irregular menstrual period compared to usual (female only)
Pain	Problems with memory	Pain during vaginal sex (female only)
		Unusual vaginal discharge (female only)
		Vaginal dryness (female only)

Round 2, Scripts 1 and 2 (17 males, 30 females)
Aching joints (such as elbows, knees, shoulders)
Acne
Arm or leg swelling
Bed sores or pressure sores
Bloating of the abdomen (belly)
Blurred vision
Body odor
Breast area enlargement or tenderness
Bruising easily (black and blue marks)
Constipation
Decreased sweating
Diarrhea
Feeling anxiety or worry
Felt that nothing could cheer you up
Flashing lights in front of your eyes
Hand-foot syndrome (a rash of the hands or feet that can cause cracking, peeling, redness or pain)
Heartburn
Hives
Insomnia (including difficulty falling asleep, staying asleep, or waking up early)
Itchy skin
Mouth or throat sores
Nosebleeds
Problems having orgasm or climax
Rash
Sad or unhappy feelings
Skin burns from radiation
Spots or lines (floaters) that drift in front of your eyes
Stretch marks
Tremors
Unexpected or excessive sweating during the day or nighttime (not related to hot flashes)
Unusual darkening of the skin
Vomiting
Wheezing (whistling noise in the chest with breathing)
Ejaculation problems (male only)
Irregular menstrual periods (female only)
Pain during vaginal sex (female only)
Unusual vaginal discharge (female only)

Round 3, Scripts 1 and 2 (11 males, 9 females)
Acne or pimples on the face or chest
Bed sores
Blurry vision
Body odor
Breast area enlargement or tenderness
Constipation
Hand-foot syndrome (a rash of the hands or feet that can cause cracking, peeling, redness or pain)
Headache*
Hives (itchy red bumps on the skin)
Hoarse voice**
Loose or watery stools
Mouth or throat sores
Rash
Skin burns from radiation
Spots or lines (floaters) that drift in front of your eyes
Stretch marks
Took too long to have an orgasm or climax†
Tremors
Unable to have orgasm or climax†
Unexpected decrease in sweating
Unusual darkening of the skin
Urge to urinate all of a sudden‡
Urinate frequently‡
Voice changes**
Vomiting
Wheezing (whistling noise in the chest with breathing)
Ejaculation problems (male only)
Irregular menstrual periods (female only)
Pain during vaginal sex (female only)
Unusual vaginal discharge (female only)

^*Added per expert panel consensus

^**Split from Voice changes (hoarseness, loss of voice) item per expert decision

^†Split from “Problems having orgasm or climax” item per expert panel consensus

^‡Split from “Increased need to urinate: per expert panel consensus

Interview Rounds 2 and 3 included only AEs that either: 1) did not receive endorsement by at least 25% of patients in the prior round (i.e., patients who did not experience the AE); 2) were modified based on results of the prior interviewing round; or 3) were modified or added because of changes that occurred from version 3 to version 4 of the clinician-reported CTCAE during the course of this study [2].

Following completion of the questionnaire, interviewers initially probed on check-marked items, then probed on all endorsed items. Interviewers then chose non-endorsed AEs for probing and asked patients to explain in their own words what the AE’s items and responses meant to them. Probes interrogated AE terms, attribute terms (i.e., regarding frequency, severity, interference), response option details, recall period, and cognitive issues in each of the four predetermined concepts of interest related to cognitive processing (i.e., comprehension, memory retrieval, judgment, and response mapping, See Appendix I).

Interviewers were also trained to employ optional or conditional probes in reaction to any hesitations, spontaneous patient comments about questions or response categories, or facial expressions or body language that may indicate meaningful reactions [32]. These conditional probes were semi-scripted with interviewers having access to a library of flexible probes with scenario-based training on how to apply these probes during interviews. At the end of the interview, patients were asked demographic questions including age, educational attainment, and race/ethnicity. Health status data attained from the medical chart and clinical staff included cancer type, disease stage, and current treatment.

Analytic approach

Each cognitive interview was digitally audio-recorded. The recordings as well as detailed field notes written during and after interviews were used to prepare a structured report of each interview. Each report included 1) patient demographics (i.e., age, education level, race/ethnicity, cancer type and stage); 2) full item-by-item results; 3) interviewer comments about participant body language or questions raised; 4) interview duration; and 5) a checklist of problems related to each item [41,42]. Interview reports were developed by two independent reviewers based on the recordings and/or field notes, with arbitration by a third in cases of disagreement.

Item-level summaries were then created for each PRO-CTCAE item based on aggregated data from across the interview reports. These structured summaries documented the frequency of problems experienced by patients at the item level (i.e., pertaining to PRO-CTCAE term, item structure, recall period, and response options), as well as the specific nature of those problems (i.e., related to cognitive processing: comprehension, memory retrieval, judgment, and response mapping, See Appendix I), and the characteristics of patients experiencing those problems. Following each round of interviews, the summaries were reviewed by a multidisciplinary team composed of experts in qualitative data analysis, health literacy, psychometrics, and clinical trial conduct, as well as patient advocates and participants from the NCI (represented by the authors of this paper).

Any item for which three or more patients reported problems was reviewed by the committee with attention to the specific nature of the problems and characteristics of the patients. In addition, all other items in the PRO-CTCAE library were reviewed, regardless of the number of problems identified during interviews by patients or interviewers. This information was used to determine if any revisions were merited based on patient feedback, and to draft a revised version of the item for subsequent testing. This overall approach was chosen collaboratively based on guidance from Willis, page 170 [32] which asserts that even a single problem can sometimes dictate the need for PRO adaptation. Saturation, defined as the point at which no new information suggesting a need for further instrument modification emerges, was documented [33]. The study was considered complete when no further information warranting modification or additional testing was generated for any item.

RESULTS

Patient sample

A total of 127 patients participated in the cognitive interviews (60 in Round 1; 47 in Round 2; 20 in Round 3), with 35% recruited at a high school education level or lower, and 28% reporting a race/ethnicity other than White/Non-Hispanic (Table 2). Of those who had a high school diploma or less, three reported a less than seventh grade education, three a junior high school education, and seven partial high school. The sample included 59% female patients, and a range of cancer types. All patients were undergoing active treatment with chemotherapy and/or radiation therapy, or had completed such treatment within the last month. Each research interaction (i.e., informed consent and cognitive interview) was completed in less than 90 minutes.

Table 2.

Patient characteristics

	Total (%)
N	127
Education
≤High School	44 (35%)
> High School	83 (65%)
Gender
Male	52 (41%)
Female	75 (59%)
Racial/Ethnic Subgroup
White/Non-Hispanic	91 (72%)
African-American	24 (19%)
Hispanic	6 (5%)
Asian/Other	6 (5%)
Disease Site
Genitourinary Malignancies1	27 (21%)
Breast Cancers	26 (20%)
Gynecologic Malignancies 2	23 (18%)
zGastrointestinal Malignancies3	22 (17%)
Hematalogic Malginancies4	11 (9%)
Lung Cancers5	9 (7%)
Head/Neck Cancers	7 (6%)
Melanoma	2 (2%)

¹Includes patients with bladder, prostate, testicular, and kidney cancers.

²Includes patients with leiomyosarcoma, cervical, endometrial, and ovarian cancers.

³Includes patients with esophageal, gastric, pancreatic, adrenal, and colorectal cancers.

⁴Includes patients with leukemia, multiple myeloma, and lymphoma.

⁵Includes patients with non-small-cell lung cancer, small cell lung cancer, and thymoma.

Assessment of terms and wording in symptom questions and response options

Table 1 lists which PRO-CTCAE items were tested in each round, respectively, and Figure 1 shows the number of PRO-CTCAE terms evaluated and revised in each of the three rounds of interviews. Overall, the items performed well, with most (63/80) not generating cognitive difficulties in any round of testing (i.e., comprehension, memory retrieval, judgment, and response mapping).

Figure 1.

Flow diagram for each round of cognitive interviews showing the number of PRO-CTCAE adverse events (AEs) for which items were evaluated in each round, and changes made. Each AE maps to a term in the CTCAE and may be characterized by up to three items representing the attributes of frequency, severity, and/or interference with activities.

AE, Adverse Event; CTCAE, Common Terminology Criteria for Adverse Events; PRO-CTCAE, Patient-Reported Outcomes version of the CTCAE

Round 1 of interviews included three script versions as planned a priori. Round 2 included one script version with 37 AEs, encompassing items with cognitive difficulties identified in Round 1, as well as items with low endorsement rates in Round 1, and those for which response options were modified (described below). No further cognitive problems meeting the prespecified criteria were detected by the third round. Table 3 shows the 17 AEs that required modification and retesting, including the initial and final versions and the reasons for modification. Items were well comprehended by patients regardless of education level.

Table 3.

Summary tracking matrix showing PRO-CTCAE modified terms and reasons for modifications

Initial Version	Modified Version	Reason(s) for the Modification
Absence of menstrual period*	Miss an expected menstrual period	To clarify that the term is aimed towards women who are expecting menstrual periods.
Aching joints*	Aching joints (such as elbows, knees, shoulders)	To clarify that the term should encompass multiple joints.
Acne*†	Acne or pimples on the face or chest	To clarify that the term does not refer solely to facial rash experienced by adolescents, rather than adults.
Body odor more than usual*†	Body odor	The phrase “more than usual” caused confusion and did not add substantive meaning to the term.
Breast area enlargement*†	Breast area enlargement or tenderness	The term “tenderness” was included to provide additional context.
Changes in sense of taste or problems tasting food*	Problems with tasting food or drink	To clarify that liquids should be considered in responses.
Decreased sweating*†	Unexpected decrease in sweating	The word “unexpected” was included to provide additional context.
Diarrhea±	Loose or watery stools (diarrhea)	The term was revised per consensus decision by expert panel.
Difficulty sleeping (insomnia)*	Insomnia (Including difficulty falling asleep, staying asleep, or waking up early)	The parenthetical phrase was included to provide additional explanation of the symptom.
Felt Unhappy*	Sad or unhappy feelings	Patients felt the addition of the term “sad” would help to clarify meaning.
Hives†	Hives (itchy red bumps on the skin)	The parenthetical phrase was added to clarify the difference between hives and rash.
Irregular menstrual periods compared to usual*†	Irregular menstrual periods	The phrase “compared to usual” was removed to reduce confusion.
Loss of control of stool or bowels*	Loss of control of bowel movements	The word “stool” was removed to reduce confusion.
Pressure or bed sores†	Bed sores	The word “pressure” was removed to reduce confusion.
Shaking chills/shivering*	Shivering or shaking chills	The sentence structure was changed to reduce confusion.
Sweats (excessive sweating)* †	Unexpected or excessive sweating during the day or nighttime (not related to hot flashes)	Detail was included to add context and to distinguish sweating from hot flashes.
Wheezing*†	Wheezing (whistling noise in the chest with breathing)0	The parenthetical phrase was added to clarify the meaning of “wheezing”.

^*Modified version tested in Round 2

^†Modified version tested in Round 3

^±Modified version tested in Rounds 2 and 3 per expert decision

In three cases, PRO-CTCAE AE terms were split into two based on changes made by the NCI to the source CTCAE measures between CTCAE v.3 and v.4 [2], and/or based on patient interviews suggesting that the initial AE terms were overly broad and encompassed more than one adverse event (Figure 1). Specifically, increased need to urinate was split into urinate frequently and urge to urinate all of a sudden; voice changes (hoarseness, loss of voice) was split into hoarse voice and voice changes; and problems feeling orgasm or climax was split into unable to have orgasm or climax and took too long to have an orgasm or climax. The PRO-CTCAE version of the CTCAE concept tremor was eliminated due to consistently poor comprehension, with an ultimate committee conclusion that this concept is not optimally amenable to patient self-reporting. Headache was added to the PRO-CTCAE item library based on patient comments that the general pain question was not appropriate for capturing headache.

Protocol-specified probing of PRO-CTCAE symptom attribute terms (i.e., severity, often, interfere) revealed that patients generally understood the term severity across all study sites and education levels (Table 4). Patients defined severity as how bad, painful, hurtful, unbearable, or intense a symptom was.

Table 4.

Generic wording for PRO-CTCAE items by attribute, and response options for each

Attributes	Example Question	Response Options
Severity	What was the SEVERITY of your VOMITING at its WORST?*	None	Mild	Moderate	Severe	Very Severe
Frequency	How OFTEN did you have a HEADACHE?	Never	Rarely	Occasionally	Frequently	Almost Constantly
Interference	How much did ANXIETY INTERFERE with your usual or daily activities?†	Not at all	A Little Bit	Somewhat	Quite a Bit	Very Much
Amount	Did you have any UNUSUAL VAGINAL DISCHARGE?	None at all	A Little Bit	Somewhat	Quite a Bit	Very Much
Present/Not Present	Did you have any VOICE CHANGES?	No	Yes

^*The word “worst” was modified to ALL CAPS based on Round 1 results in order to increase emphasis.

^† Initial wording “…with your usual activities” was modified based on cognitive testing to be “…with your usual or daily activities.”

Notably, PRO-CTCAE severity items ask patients to consider the worst severity of the symptom during the recall period. On probing in Round 1, there was some variation in judgment processes used by respondents in this regard: although most reported considering the magnitude of severity during their worst episode, some also considered average severity. Some reported that the chronicity of a symptom contributed to what strategy they used to answer the question. For example, one participant thought about her worst episode for symptoms that were periodic (such as vomiting or diarrhea), but thought about average severity for symptoms that were more chronic or non-episodic (such as anxiety or aching joints). Some patients also considered the number of occurrences or number of days affected when determining magnitude of severity. Therefore, in Rounds 2 and 3, the word WORST was put in ALL CAPS and participants in subsequent rounds focused on worst rather than average severity.

There was good general understanding across patients of the term often as a representation of frequency. When evaluating how often a symptom occurred, patients generally considered both the specific number of occurrences and the number of days they experienced a symptom. For example, one participant explained that she determined the frequency of her arm or leg swelling by the specific number of occurrences, but the frequency of pain in the abdomen by the number of days it occurred because the pain usually lasted most or all of the day when she had it.

There was also good general understanding among patients of the term interfere. Patients generally considered how symptoms got in the way or limited them from doing “normal” or “regular” activities, or were “incapacitating.” In Round 1, two patients misinterpreted the context of use of this term, as one participant noted, “interference is when someone interrupts a conversation.” Therefore, questionnaire instructions were modified in Round 2 to clarify that “For some of the questions, you will be asked whether a symptom interferes with your daily activities. To interfere means “to make difficult.” In subsequent rounds there was no further misinterpretation of this term.

Notably, PRO-CTCAE interference items included in Round 1 asked if symptoms interfere with daily activities. The term daily activities drew diverse interpretations. Many patients only considered activities of the daytime hours while others considered a 24-hour time period. On probing to elicit what terms patient prefer for activities during a 24-hour period including work, household chores (cleaning, laundry), cooking, driving, volunteer work, social activities, errands, using the computer, reading, texting on phone, bathing, dressing, exercising, gardening, and sleeping, patients preferred the terms usual or typical in addition to daily. Accordingly, in Rounds 2 and 3, interference item wording was modified to specify usual or daily activities. In subsequent rounds, no further misinterpretation of this concept was observed.

Finally, the phrase did you have any, which precedes each symptom term in PRO-CTCAE questions, was probed with universally good understanding.

Terms used in response options (Table 4) were well comprehended. For example, response categories were not found to be overlapping and wording was generally well understood. Multiple patients in Round 1 suggested that a “not applicable” category should be added for symptoms associated with specific treatments (e.g., skin burns from radiation), and for symptoms that are conditional (e.g., missed menstrual period among postmenopausal women; difficulty getting or keeping an erection and unable to have an orgasm or climax among patients who are not sexually active). We had anticipated scoring these responses as “never,” “none,” “not at all,” or “no,” depending on the item response scale in question, but “not applicable” was added to pertinent items in Rounds 2 and 3 with good patient understanding.

Recall period

In Round 1, the 7-day default recall period was specified as an instructional header at the beginning of groups of AE terms. Multiple patients did not notice or remember the recall period when answering questions, and considered various alternative recall periods including “the past two weeks,” “since cancer treatment began,” and “since being diagnosed with cancer.” Therefore, in Rounds 2 and 3, the 7-day recall period was specified at the beginning of each AE term individually. This approach improved awareness of the desired recall period, although some variability remained in how patients considered this time period (e.g., the last week, the recent past).

The final wording for PRO-CTCAE v1.0 item structures and response options is shown in Table 4, and the final library of 80 AE terms represented by 124 attribute items is shown in Table 5.

Table 5.

PRO-CTCAE version 1.0 terms

Aching joints (such as elbows, knees, shoulders)	Headache	Problems with memory
Aching muscles	Heartburn	Problems with tasting food or drink
Acne or pimples on the face or chest	Hiccups	Rash
Anxiety	Hives (itchy red bumps on the skin)	Ridges or bumps on your fingernails or toenails
Arm or leg swelling	Hoarse voice	Ringing in your ears
Bed sores	Hot flashes	Sad or unhappy feelings
Blurred vision	Increased passing of gas (flatulence)	Shivering or shaking chills
Bloating of the abdomen (belly)	Increased skin sensitivity to sunlight	Shortness of breath
Body odor	Insomnia (including difficulty falling asleep, staying asleep, or waking up early)	Skin burns from radiation
Breast area enlargement or tenderness	Irregular menstrual periods	Skin cracking at the corners of your mouth
Bruising easily	Itchy skin	Spots or lines (floaters) that drift in front of your eyes
Change in the color of your fingernails or toenails	Loose or watery stools (diarrhea)	Stretch marks
Constipation	Lose any fingernails or toenails	Took too long to have an orgasm or climax
Cough	Loss of control of bowel movements	Unable to have orgasm or climax
Decreased appetite	Loss of control of urine (leakage)	Unexpected decrease in sweating
Decreased sexual interest	Miss an expected menstrual period	Unexpected or excessive sweating during the day or nighttime (not related to hot flashes)
Difficulty getting or keeping an erection	Mouth or throat sores	Unusual darkening of the skin
Difficulty swallowing	Nausea	Unusual vaginal discharge
Dizziness	Nosebleeds	Urinate frequently
Dry mouth	Numbness or tingling in your hands or feet	Urge to urinate all of a sudden
Dry skin	Pain	Urine color change
Ejaculation problems	Pain during vaginal sex	Vaginal dryness
Fatigue, tiredness or lack of energy	Pain in the abdomen (belly area)	Voice changes
Feelings like nothing could cheer you up	Pain or burning with urination	Vomiting
Flashing lights in front of your eyes	Pain, swelling, or redness at a site of drug injection or IV	Watery eyes (tearing)
Hair loss	Pounding or racing heartbeat (palpitations)	Wheezing (whistling noise in the chest with breathing)
Hand-foot syndrome (a rash of the hands or feet that can cause cracking, peeling, redness or pain)	Problems with concentration

DISCUSSION

Direct patient input is essential in the development of contemporary patient-reported outcome measures [33,43]. Cognitive interviewing research provides assurance that measures have been developed or adequately modified and that respondents understand how to complete items as intended [32].

The current study provides evidence contributing to the content validity of U.S. English language versions of items in the PRO-CTCAE item library for measuring symptomatic AEs from the patient perspective within the context of cancer treatment. Results did not differ based on education level. Several minor changes were made to the items to improve their clarity, comprehension, and ease of response judgment.

The size and diversity of our sample is a particular strength of this study, allowing us to surmount several methodological issues and achieve qualitative rigor. Given the large number of PRO items to be assessed (i.e., 124 items representing 80 AEs), and the need to adequately test the issues of male and female specific items, ensuring that all items were tested by an adequate number of respondents required a larger sample than is typically employed in qualitative research, and required carefully planned strategies to minimize participant burden. Simultaneously, the credibility of our results is enhanced by our including symptomatic participants currently or recently treated for advanced cancer for whom a majority of the symptomatic AE items would be highly salient, and by enriching the sample with respondents of lower educational attainment.

Envisioned uses of the PRO-CTCAE item library in cancer clinical trials include general adverse event detection/screening, support of dose-finding, and assessment of comparative tolerability for product labeling purposes. Comparative effectiveness research contexts including registries and safety surveillance programs are also anticipated. These uses apply across cancer treatment types (including systemic therapy, radiation therapy, surgery) both for evaluation of acute and late toxicities. It is anticipated that not all items in the PRO-CTCAE item library would be administered to all patients in a given study. Due to the number of AEs in the library, such an approach would be impractical, and not all AEs are relevant to a particular disease or treatment context. Rather, a subset of AEs is selected for a patient case report form customized for a given study based on prior evidence of which AEs are salient to that context. In addition, identification of “core” sets of AEs is underway to specify groupings appropriate to particular contexts based on prevalence and severity data, which investigators could use as a rationale for assembling these surveys [44]. Finally, it is anticipated that a mechanism will be employed for patients to add additional unsolicited AE information beyond the list in patient case report forms. The NCI-hosted the PRO-CTCAE software includes such a functionality which can map patient-entered symptoms back to the PRO-CTCAE and CTCAE libraries as well as to the Medical Dictionary for Regulatory Activities (MedDRA).

A limitation of the current study is that we did not develop verbatim transcripts of the interviews; however the recordings were available digitally on a secure site and were regularly accessed, as needed, to provide clarification and enrichment of the field notes. Furthermore, we did not employ stratified sampling to match cancer incidence rates as we did not hypothesize differences in item comprehension or item acceptability, based on disease site. The current study was limited to the English language version of PRO-CTCAE in a United States population of cancer patients. Assessments of content validity are therefore ongoing in other countries where English is spoken, as well as in other languages to which PRO-CTCAE items are being translated. Additional testing of PRO-CTCAE in individuals with low literacy and extremely low educational attainment is an important direction for future research. Quantitative assessment of measurement properties of PRO-CTCAE items, including construct validity, test-retest reliability, sensitivity to change over time, and appropriate recall period (e.g., 7-day vs daily recall), are beyond the scope of the current report, and are being reported elsewhere [45,46]. The current study was limited to paper administration; software hosted at the NCI has been developed that can administer the items to patients via tablet, web or automated telephone (interactive voice response) and these three modes of administration are currently being evaluated for measurement equivalence [47]. Finally, feasibility studies in multicenter clinical trials to evaluate the optimal approach for integrating PRO-CTCAE into existing clinical trial operations are also underway. Studies are also planned to gauge the utility of collecting symptomatic toxicities by self-report, determine the best approaches to incorporate PRO-CTCAE data into adverse event reporting and study-level interpretation of the toxicity profile of a given therapy.

Development of PRO-CTCAE is occurring within a broader context of patient centeredness in health care [48], and it is the hope of the investigators and the NCI that PRO-CTCAE will enhance understanding of the patient experience with cancer treatment, thereby improving safety, comfort, and patient-clinician communication.