Applying the Toxicity Index to Patient-Reported Symptom Data: An Example Using the EORTC Colorectal Cancer-specific Quality of Life Questionnaire

Abstract

Purpose:

The toxicity index (TI) is a summary index that accounts for toxicity grades associated with cancer symptoms that has been shown to be more sensitive than other toxicity systems to treatment differences. The TI can be used with patient-reported symptoms but requires that scores for different items represent equivalent severity. The purpose of this paper is to provide an example of scoring patient-reported symptom that satisfies the requirement of equivalent symptom severity.

Methods:

A sample of 1232 adults with rectal cancer from a Phase III clinical trial self-reported 18 symptoms on the European Organization for Research and Treatment of Cancer colorectal cancer measure using a 4-category response scale (Not at all; A little bit; Quite a bit; Very much). The sample was 22–85 years old (average age = 57), 30% female, 85% non-Hispanic white, 59% stage II and 41% stage III cancer. A recoded TI (TI*) was created using item response theory category thresholds.

Findings:

The TI* had larger rank-order correlations than the original TI with Karnofsky Performance Status, hemoglobin level, symptom bother and other aspects of health-related quality of life.

Implications:

Recoding items based on category thresholds yielded a more valid TI score that can be used to summarize adverse events.

Introduction

Toxicity data consist of treatment attributable adverse events (AEs) graded as 0, 1, 2, 3, 4, or 5, for each of the 790 AE terms, grouped in 26 system organ classes according to Common Terminology Criteria for Adverse Events (CTCAE) v4.0 [1]. Toxicity grading by clinicians is a standard component of cancer clinical trial data collection. Grade 0 adverse events represent the absence of toxicity. The toxicity index (TI) was inspired by hash functions and provides a summary of all n observed toxicity grades [2]. Each of the $n$ toxicity grades $X_i(i=1,\dots ,n)$ for an individual is represented in descending order: ${\mathit{X}}_1\ge {\mathit{X}}_2\ge \cdots \ge {\mathit{X}}_n$. An individual’s TI score is a function of the ordered toxicity grades:

$$\text{TI}=\sum _{i=1}^n\frac{X_i}{\prod _{j=1}^{i-1}\left(1+X_j\right)}$$

Any TI greater than or equal to 3 corresponds to a dose limiting toxicity and the maximum toxicity grade is the integer part of the final score. For example, a TI of 3.0 indicates a single grade 3 toxicity, whereas a TI of 3.5 means that the patient experienced at least one grade 3 toxicity plus additional toxicity. All toxicity grades are represented in the score, though lower grades contribute less to the final score than higher grades.

The TI has potential to be used with patient-reported symptom measures. But the TI assumes equal levels of impact for the item response categories for different symptoms. CTCAE grades are treated as equivalent across symptoms. It also may be acceptable for patient-reported symptoms measured using a response scale such as: Not bothered at all; A little bit bothered; Somewhat bothered; Bothered quite a bit; and Bothered very much. But the TI approach is not ideal for summarizing patient reports of symptoms when severity is not captured. For example, use of the TI with reports about frequency of symptoms or extent to which symptoms occur may be problematic because severity may differ (e.g., runny nose versus vomiting).

Category response curves provide information about item response options in multi-item scales that identifies where they fall on the underlying continuum. Item response theory can be used to get estimates of threshold parameters that represent the underlying trait level necessary to respond above each threshold with 0.50 probability [3]. These thresholds indicate the relative severity by item response options.

This paper presents a comparison of scoring the TI for a patient-reported symptom measure scored assuming equal distances between response categories versus scoring based on item thresholds in the National Surgical Adjuvant Breast and Bowel Project R-04 rectal cancer clinical trial.

Methods

Study Design and Sample

Eligible patients were diagnosed with surgically resectable stage II or III rectal adenocarcinoma. A total of 1608 patients participated in the Phase III clinical trial of rectal cancer (NCT00058474) between 2004 and 2010 [4–5]. All patients who spoke English, French, or Spanish were invited to complete a questionnaire at baseline prior to randomization to treatment. If the patient was not accessible in person, staff were encouraged to mail the questionnaire to the patient or collect responses by telephone.

The trial was approved by the local institutional review boards, and all patients provided written consent. The secondary analyses reported here were determined to be exempt by the Cedars Sinai and UCLA institutional review boards. The sample consisted of 1232 adults with complete data for 18 symptom items (see Measures below) analyzed: 22–85 years old (average age = 57), 30% female, 85% non-Hispanic white, 59% stage II and 41% stage III cancer (Table 1).

Table 1:

Sample Characteristics (n = 1232)

Age Mean (SD) [range]	57 (11) [22, 85]
Gender
Female	30%
Race/ethnicity
Hispanic	5%
non-Hispanic Black	5%
non-Hispanic White	85%
non-Hispanic other/unknown race	5%
Body Mass Index
<18.5 underweight	1%
<25 normal	26%
<30 overweight	36%
30+ obese	37%
Clinical Stage
Stage II	59%
Stage III	41%
Karnofsky Performance Status(a)
Fully active (K90–100)	85%
Restricted/ambulatory (K70–80) + (K50–60)	15%

(a) Fully active, able to carry on all pre-disease performance without restriction

(K70–80) Restricted in physically strenuous activity but ambulatory

(K50–60) Ambulatory and capable of all self-care but unable to carry out any work activities

Measures

The baseline patient-reported survey included 112 questions. The focus of the analyses are 18 symptoms (items 60–77 on the baseline survey) assessed in the European Organization for Research and Treatment of Cancer (EORTC) colorectal cancer-targeted quality of life questionnaire (QLQ-CR38) [6]. The QLQ-CR38 assesses the extent to which symptoms were experienced in the past week: Not at all; A little bit; Quite a bit; Very much (Table 2). A higher score indicates a greater extent of experiencing symptoms.

Table 2. EORTC colorectal cancer-specific quality of life (QLQ-CR38) item frequencies on baseline survey.

Patients sometimes report that they have the following symptoms or problems. Please indicate the extent to which you have experienced these symptoms or problems during the past week. Please answer by circling the number that best applies to you.

During the past week:	Not at all	A little bit	Quite a bit	Very Much
60. Did you urinate frequently during the day?	31%	38%	26%	5%
61. Did you urinate frequently during the night?	37%	47%	12%	3%
62. Did you have pain when you urinated?	93%	5%	1%	0.3%
63. Did you have a bloated feeling in your abdomen?	62%	28%	8%	3%
64. Did you have abdominal pain?	72%	22%	5%	1%
65. Did you have pain in your buttocks?	60%	22%	10%	9%
66. Were you bothered by gas (flatulence)?	38%	41%	16%	5%
67. Did you belch?	57%	36%	5%	2%
68. Have you lost weight?	57%	33%	7%	3%
69. Did you have a dry mouth?	73%	20%	4%	2%
70. Have you had thin or lifeless hair as a result of your disease?	96%	4%	0.4%	0.2%
71. Did food and drink taste different than usual?	90%	8%	2%	1%
72. Have you felt physically less attractive as a result of your disease or treatment?	80%	14%	3%	2%
73. Have you been feeling less feminine/masculine as a result of your disease or treatment?	81%	14%	3%	2%
74. Have you been dissatisfied with your body?	66%	26%	6%	2%
75. Were you worried about your health in the future?	19%	47%	22%	13%
During the past four weeks:	Not at all	A little bit	Quite a bit	Very Much
76. To what extent were you interested in sex?	27%	37%	23%	12%
77. To what extent were you sexually active (with or without intercourse)	39%	37%	19%	6%

Note: Questions 76 and 77 were reversed scored when creating the toxicity index so that a higher score represented greater frequently of symptoms.

Also included in the baseline survey was the Functional Assessment of Cancer Therapy (FACT) for patients with colorectal cancer Trial Outcomes Index (FACT-C TOI), the 13-item FACT neurotoxicity scale (FACT-GOG-NTX-13), the SF-36 v.2 vitality scale, and a 17-item symptom bother checklist used in the study (SCL-17) [7–10].

Clinical measures analyzed were the maximum adverse event (AE) grade, hemoglobin level, and the Karnofsky performance status (KPS) index.

Analysis

Our primary analysis used baseline survey data, but we looked at consistency of results with one-year post-surgery survey data. The standard coding of the QLQ-CR38 items is: 0 = Not at all, 1 = A little bit, 2 = Quite a bit; and 3 = Very much. We report internal consistency reliability [11] and item-scale correlations for the 18-item QLQ-CR38 symptom score using this scoring. We used categorical confirmatory factor analysis with diagonally weighted least squares to evaluate whether the items were sufficiently unidimensional to estimate response category thresholds using the item response theory graded response model. Because of content overlap (local dependency) among the QLQ-CR38 symptoms, we included six residual correlations (item pairs: 60–61, 63–64, 72–73, 72–74, 73–74, 76–77) We evaluated model fit using the comparative fit index (CFI) and the root mean square error of approximation (RMSEA). CFI values > 0.95 and RMSEA values < 0.06 are considered good fit [12].

Category thresholds for the 18 items (3 thresholds per item) were estimated from the graded response model [13]. Standard errors around the thresholds were used to create 95% confidence intervals. Overlapping confidence intervals of the three thresholds for each item were identified. Threshold estimates were used to adjust the scoring of item responses. The 0 for “Not at all” was preserved but the distance between scores assigned for other response options were shifted based on differences in item thresholds.

The TI is scored so that a higher score represents a greater toxicity. We hypothesized positive correlations with measures scored so that a higher score is worse (maximum AE grade, SCL-17, and worried about health in the future) and negative correlations with measures scored so that a higher score is better (KPS, hemoglobin, FACT-C TOI, FACT-GOG-NTX-13, and SF-36 v.2 vitalty scale). Spearman rank-order correlations of the TI and the TI with these variables were estimated.

Confirmatory factor analysis (CFA) was conducted using Mplus [14] and all other analyses with SAS version 9.4, TSIM3 (SAS Institute Inc., Cary, NC, USA).

Results

Internal consistency reliability for the 18 symptom-item scale was 0.79 and item-scale correlations (corrected for item overlap with the scale total) ranged from 0.26 to 0.49. “Sufficient” unidimensionality of the 18 QLQ-CR38 symptom items was supported by the fit of the one-factor CFA model: CFI = 0.962 and RMSEA = 0.054. The chi-square was 587.385 with 129 degrees of freedom (p<.0001).

The original scoring of the QLQ-CR38 symptom items is shown at the top of Table 3: 0 = Not at all, 1 = A little bit; 2 = Quite a bit; and 3 = Very much. Threshold estimates from the graded response model for each item are also shown. Thresholds within items that differed significantly are denoted by not sharing a superscript letter. For example, the threshold between quite a bit and very much for Item 60 (frequently urinate during the day) was 4.43 (SE = 0.41) while the threshold between not at all and a little bit was −1.28 (SE = 0.14), and these thresholds were significantly different.

Table 3. Original Scoring of Items Highlighted in First Row (Recoding of Each Item Show in Rows Below).

Items	0 (Not at all)		1 (A little bit)		2 (Quite a bit0		3 (Very much)
Item 60	0		1		2		5
Thresholds		−1.28 e		1.21 c,a		4.43 a
Item 61	0		1		3		5
Thresholds		−0.76 a		2.18 a		4.30 a,b
Item 62	0		2		4		5
Thresholds		2.41a		3.77 a		4.71a
Item 63	0		1		3		4
Thresholds		0.47c		1.98a		3.12b,c
Item 64	0		1		3		3
Thresholds		0.85b		2.28a		3.38a,b
Item 65	0		1		2		3
Thresholds		0.35c		1.50c		2.45c
Item 66	0		1		2		4
Thresholds		−0.62d		1.50c		3.18b
Item 67	0		1		4		5
Thresholds		0.35c		3.37a		5.31a
Item 68	0		1		4		5
Thresholds		0.31c		2.61a		4.02a,b
Item 69	0		2		4		5
Thresholds		1.00b		2.63a		3.59a,b
Item 70	0		2		4		5
Thresholds		2.84a		4.41a		5.21a
Item 71	0		2		4		5
Thresholds		1.68c*		2.87a		3.73a,b
Item 72	0		2		3		4
Thresholds		1.09b		2.15a		3.00b,c
Item 73	0		2		3		4
Thresholds		1.14b		2.18a		2.86b,c
Item 74	0		1		3		4
Thresholds		0.61c		2.26a		3.29a,b
Item 75	0		1		2		4
Thresholds		−2.03e		0.98c		2.80b,c
Item 76	0		1		1		3
Thresholds		−3.96f		−1.21e		1.95c
Item 77	0		1		1		2
Thresholds		−5.59f		−2.21e		0.97d

Note: See Table 1 for item wording. Thresholds in rows that share a superscript letter do not differ significantly from one another.

To determine how to modify the original scoring of the QLQ-CR38 symptoms for the TI summary measure, we compared thresholds across items. We used as many integers and no more than needed to reflect the variation in severity across symptoms indicated by the thresholds. We ended up needing to add two integers (4 and 5) to reflect variation in thresholds. For example, we scored responses of very much as 5 for Item 60 but 4 for Item 63 (bloated feeling in your abdomen) because the threshold between quite a bit and very much for the latter was smaller (3.12, SE = 0.20) than the former (4.43, SE = 0.41). The TI index was computed from the original scoring and then the revised TI (TI*) was scored based on item category thresholds.

The Spearman rank order correlations between the TI and TI* at baseline was 0.65. TI* was more strongly associated with other variables than was the TI (Table 4). TI* was significantly more highly associated than the TI with six variables: 1) Karnofsky Performance Status; 2) hemoglobin level; 3) SCL-17 scale; 4) FACT-C TOI; 5) FACT NTX score; and 6) SF-36 v2 vitality scale. We found similar results for survey data collected one-year post-surgery (Table 5).

Table 4.

Spearman correlations (p-value) at Baseline for Two Variants of Toxicity Index (TI) Created from the EORTC Colorectal Cancer-Specific Quality of Life Questionnaire (QLQ-CR38)

Variables	Original TI	TI*
Maximum Adverse Event (AE) Grade	0.06 (.0376)	0.08 (0.0036)
Karnofsky Performance status	−0.07 (.0091)	−0.13 (<.0001)
Hemoglobin level	−0.08 (.0049)	−0.15 (<.0001)
SCL-17 (17-item symptom bother scale)	−0.02 (.4814)	0.16 (<.0001)
FACT-C Trial Outcomes Index (TOI)	0.01 (.6021)	−0.17 (<.0001)
FACT neurotoxicity (GOG-NTX-13) score	−0.03 (.3840)	−0.18 (<.0001)
SF-36 v2 vitality scale (4 items)	−0.05 (.0918)	−0.21 (<.0001)

Note: TI* is based on recoding of item scores using graded response model threshold estimates. FACT = Functional Assessment of Cancer Therapy

Table 5.

Spearman correlations (p-value) at One-Year Post-Surgery for Two Variants of Toxicity Index (TI) Created from the EORTC Colorectal Cancer-Specific Quality of Life Questionnaire (QLQ-CR38)

Variables	Original TI	TI*
Maximum Adverse Event (AE) Grade	0.06 (.0608)	0.06 (0.1200)
Karnofsky Performance status	0.00 (.9492)	−0.12 (0.0008)
Hemoglobin level	−0.09 (.0113)	−0.14 (<.0001)
SCL-17 (17-item symptom bother scale)	−0.07 (.0395)	0.09 (0.0096)
FACT-C Trial Outcomes Index (TOI)	0.00 (.9499)	−0.14 (<.0001)
FACT neurotoxicity (GOG-NTX-13) score	0.04 (.2114)	−0.09 (0.0187)
SF-36 v2 vitality scale (4 items)	−0.05 (.1130)	−0.17 (<.0001)

Note: TI* is based on recoding of item scores using graded response model threshold estimates. FACT = Functional Assessment of Cancer Therapy

Discussion

The value of patient-reported symptoms has been documented for more than two decades (e.g., Justice et al [15]). The work reported here is consistent with the ongoing efforts to incorporate the patient’s voice into the assessment of adverse events in cancer clinical trials. For example, Smith et al. [16] noted that some of the adverse events in the CTCAE are best assessed by asking the patient.

The TI* was significantly correlated with several patient-reported measures (SCL-17, FACT TOI, FACT NTX score, SF-36 v.2 vitality scale), maximum AE grade, Karnofsky Performance Status, and hemoglobin level. The greater relative validity of the TI* compared to the TI was supported by consistently larger associations with other variables as hypothesized.

One limitation of the TI is that it requires rank-based analysis because it does not follow any well-known probability distribution such as the normal distribution. However, it contains more information than other toxicity analysis methods by accounting for both the multiplicity and severity of toxicities, without losing the natural interpretability of the maximum grade approach. This added information was shown to yield greater power in detected treatment differences than maximum grade and average toxicity approaches [17–18].

Conclusions

This paper provides a prototype of how the TI can be applied to patient-reported symptom measures and illustrates the value of adjusting item scoring to account for different levels of underlying symptom severity. The method used to adjust scores is not the only or necessarily the best approach. Future research and applications are needed to evaluate similar and different strategies to adjust category scoring of polytomous symptom items to satisfy the underlying assumption of equivalence across items implicit in the scoring of the TI.

Highlights.

• The toxicity index summarizes cancer symptom grades.

• The index is useful for assessing treatment differences.

• Use of item thresholds for patient-reported symptoms improves the validity of the index.