Validation of the German patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE™)

The Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE™) was developed by the US National Cancer Institute. In a multicentre study, a set of 31 items drawn from the German language version of the PRO-CTCAE were shown to be reliable and valid to capture symptomatic adverse events directly from patients undergoing cancer treatment.

Abstract

Background

Integrating the patient's perspective has become an increasingly important component of adverse event reporting. The National Cancer Institute has developed a Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE™). This instrument has been translated into German and linguistically validated; however, its quantitative measurement properties have not been evaluated.

Patients and methods

A German language survey that included 31 PRO-CTCAE items, as well as the EORTC QLQ-C30 and the Oral Mucositis Daily Questionnaire (OMDQ), was distributed at 10 cancer treatment settings in Germany and Austria. Item quality was assessed by analysis of acceptability and comprehensibility. Reliability was evaluated by using Cronbach's' alpha and validity by principal components analysis (PCA), multitrait-multimethod matrix (MTMM) and known groups validity techniques.

Results

Of 660 surveys distributed to the study centres, 271 were returned (return rate 41%), and data from 262 were available for analysis. Participants' median age was 59.7 years, and 69.5% of the patients were female. Analysis of item quality supported the comprehensibility of the 31 PRO-CTCAE items. Reliability was very good; Cronbach's' alpha correlation coefficients were >0.9 for almost all item clusters. Construct validity of the PRO-CTCAE core item set was shown by identifying 10 conceptually meaningful item clusters via PCA. Moreover, construct validity was confirmed by the MTMM: monotrait-heteromethod comparison showed 100% high correlation, whereas heterotrait-monomethod comparison indicated 0% high correlation. Known groups validity was supported; PRO-CTCAE scores were significantly lower for those with impaired versus preserved health-related quality of life.

Conclusion

A set of 31 items drawn from the German PRO-CTCAE item library demonstrated favourable measurement properties. These findings add to the body of evidence that PRO-CTCAE provides a rigorous method to capture patient self-reports of symptomatic toxicity for use in cancer clinical trials.

introduction

Adverse event (AE) monitoring is an essential aspect of any cancer clinical trial, ensuring that study participants are not harmed by treatment, and allowing for conclusions to be made about a treatment regimen's safety and tolerability profile. For systematically grading and reporting treatment-related toxicity in cancer clinical trials, AEs are assessed with the Common Terminology Criteria for Adverse Events (CTCAE). The latest version 4.03 of the CTCAE released by the National Cancer Institute (NCI) contains 790 items including laboratory tests, clinical events, and symptom evaluation [1]. Typically, the grading of symptoms in clinical trials is carried out by research staff. As an underestimation of AE severity by healthcare professionals has been repeatedly demonstrated [2], there has been expanding interest to incorporate the patient's perspective via patient-reported outcomes (PRO) [3]. Covering different aspects of AE reports, both clinician-reported and patient-reported approaches are complementary [1, 2].

In 2010, the NCI initiated development of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE™) [4] and has evaluated the PRO-CTCAE item library using both qualitative and quantitative techniques [5]. The PRO-CTCAE item library contains 124 items evaluating 78 symptomatic toxicities, and the English language version has demonstrated generally favourable measurement properties [5].

The PRO-CTCAE item library has been translated into German and linguistically validated in a sample of patients undergoing haematopoietic stem cell transplantation [6]; however, its quantitative measurement properties (item quality, reliability, and validity) have not been evaluated.

The aim of this study was to assess item quality, reliability, and validity of a subset of items from the PRO-CTCAE German item library, thereby supporting the continued implementation of PRO-CTCAE in cancer clinical trials in German-speaking countries.

patients and methods

setting and sample

This study was conducted at 10 cancer treatment centres in Germany and Austria between October 2012 and April 2013. Participants had to be able to read, write, and comprehend German. There were no eligibility limitations with respect to the type of treatment currently being received, disease site, or inpatient/outpatient setting. Participants who, in the opinion of the treating clinician, had clinically significant cognitive impairment were excluded from the study. The study received approval of the Ethics Committee of the University of Bonn.

A total of 660 questionnaires were distributed to the 10 participating centres. Totally, 271 participants were enrolled and provided their informed consent. They were provided with the study instruments and could complete study procedures independently by paper and pencil during their visit at the centre, or at home and return by surface mail.

study measures

Of the 124 items contained in the PRO-CTCAE German language item library, 31 were selected for validation and defined as a ‘core item set’. The selected items reflect 14 symptomatic toxicities and were selected a priori on the basis of their prevalence across cancer treatment types and disease sites [5, 7] and based on expert consultation. PRO-CTCAE attributes include frequency (F) (e.g. how often did you have nausea), severity (S) (e.g. what was the severity of your pain), and interference with daily activities (I) (e.g. how much did fatigue interfere with your usual or daily activities). Responses are provided on a five-point Likert scale. The recall period for PRO-CTCAE is the past 7 days. PRO-CTCAE symptomatic toxicities included in this study were those crossing disease sites and treatment modalities (shown in Table 1). Examples of the PRO-CTCAE items are shown in supplementary Table S1, available at Annals of Oncology online. For more information about PRO-CTCAE and for permission to use, visit http://healthcaredelivery.cancer.gov/pro-ctcae/

Table 1.

PRO-CTCAE item clusters

Item cluster	Number of items	Item dimensions
Anxiety and sadness	6	Frequency, severity, interference
Nausea and vomiting	4	Frequency, severity
Appetite loss	2	Severity, interference
Fatigue	2	Severity, interference
Pain	3	Frequency, severity, interference
Mucositis and xerostomia	4	Severity, interference
Dyspnoea	2	Severity, interference
Mental concentration	2	Severity, interference
Numbness and tingling	2	Severity, interference
Insomnia	2	Severity, interference
Constipation	1	Severity
Diarrhoea	1	Frequency

PRO-CTCAE, Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events.

Participants also completed the Quality of Life Questionnaire of the European Organisation for Research and Treatment of Cancer (EORTC QLQ-C30) and the Oral Mucositis Daily Questionnaire (OMDQ). The EORTC QLQ-C30 [8] includes 30 items summarised to a global health status subscale, functional subscales, and symptom subscales. The OMDQ [9] contains eight items covering the subscales general well-being, mouth and throat pain, and diarrhoea. In the present study, the recall period of the OMDQ was extended from 24 h to the past 7 days, allowing comparability across PRO-CTCAE, EORTC QLQ-C30, and the OMDQ.

For comparability and only for the purpose of the MTMM analysis, PRO-CTCAE, EORTC QLQ-C30, and OMDQ scores were calculated according to the following equations [10].

PRO-CTCAE item cluster scores (clusters indicated in Table 1) and EORTC QLQ-C30 subscale scores were calculated by averaging the component items of each of the clusters or subscales according to

$$\text{RS}=\frac{I_1+I_2+\cdots +I_n}{n}$$ (1)

where I_1,2,…,n are the item values and n is the number of items.

To ease the interpretation of MTMM models, two PRO-CTCAE item clusters (problems with mental concentration and anxiety/sadness) and the analogous EORTC QLQ-C30 functional constructs (cognitive function and emotional function) were linearly transformed to a 0–100 scale using the following formula:

$$\text{Score}=\left\{1-\frac{\text{RS}-1}{\text{Range}}\right\}\times 100$$ (2)

All of the other PRO-CTCAE item cluster scores and the EORTC QLQ-C30 symptom subscale scores and the global health status score were linearly transformed to a 0–100 scale so that higher scores indicate worse symptoms and better quality of life, using the following formula:

$$\text{Score}=\left\{\frac{\text{RS}-1}{\text{Range}}\right\}\times 100$$ (3)

In the above equations, range refers to the theoretical range of the response options. In the case of missing data, scores were calculated if a minimum of 50% of the values were present [10].

item quality

A raw data analysis was carried out. Acceptability and comprehensibility were assessed by inspecting for illogical endorsement patterns and missing values. To evaluate item redundancy, weighted kappa with a weighting scheme according to Agresti [11] and Bowker's test (P <0.05) were used [12].

reliability

Reliability reflects the consistency and stability of test scores and was determined by calculating Cronbach's alpha for each PRO-CTCAE item cluster indicated in Table 1.

validity

Validity refers to the degree to which an instrument measures what it is supposed to measure. Construct validity was evaluated using principal component analysis (PCA) with varimax rotation, multitrait-multimethod matrix (MTMM) methods, and known groups validity techniques to assess whether PRO-CTCAE items measure the same constructs as corresponding items from the EORTC QLQ-C30 and OMDQ. The MTMM correlation coefficients were calculated as rank correlation coefficients according to Spearman's hypothesising that different instruments measuring similar constructs would show high correlations (r > 0.6), whereas different instruments measuring different traits would not correlate strongly [13]. Known groups validity evaluated the criteria age (≤60 and >61 years), diagnosis (breast cancer and all other types of cancer), and quality of life (EORTC QLQ-C30 item 30 score ≤4 and >4 [8]), using the Mann–Whitney U test (P< 0.05).

Statistical analyses were carried out using SPSS^® and Microsoft Excel^®. Data were treated as interval-scaled data. P values were adjusted for multiplicity by Benjamini–Hochberg's step-up procedure (false discovery rate of 0.05) [14].

results

patients

The return rate was ∼41%. Totally, 271 patient questionnaires were returned, and 262 were available for psychometric analyses.

The median age of the sample was 60.0 years (mean age 59.7 years, range 24–91); a majority of the sample was female (69.5%). The native language of almost all respondents was German. In the sample, 56.7% had a school leaving certificate as the highest educational degree. A majority of the sample had breast cancer (42.1%) due to a high proportion of participating breast cancer units, and ∼80% had been treated on an outpatient basis. The EORTC QLQ-C30 item 30 score suggested that the sample had a generally moderate level of health-related quality of life (HRQL) of the population (mean 55.9, standard deviation 23.4).

item quality

Inspection of the item-level endorsement patterns indicated that PRO-CTCAE, EORTC QLQ-C30, and OMDQ responses were not distributed normally. The frequency distributions of responses for all items are shown in supplementary table S2, available at Annals of Oncology online. Illogical endorsement patterns of 0.71% and missing values of 0.78% suggest that PRO-CTCAE is acceptable and comprehensible to German speakers. Redundancy assessment according to weighted kappa [15] revealed that mucositis, dyspnoea, and mental concentration (severity versus interference), as well as vomiting and pain (frequency versus severity), exhibited almost perfect agreement. Only one item showed only moderate agreement: sadness (interference versus frequency). However, Bowker's test identified significant differences between the attributes for all items except nausea, vomiting, and pain (frequency and severity). Detailed results are shown in supplementary table S3, available at Annals of Oncology online.

reliability

In general, Cronbach's alpha should be >0.8, but values >0.7 are still acceptable [16]. All Cronbach's alpha values for the PRO-CTCAE item clusters were >0.9 indicating very good reliability, except for the item clusters nausea and vomiting as well as appetite loss (good reliability with values >0.8), and mucositis and xerostomia (values >0.7); see supplementary table S4, available at Annals of Oncology online.

validity

PCA with varimax rotation identified 10 components explaining 81.5% of the total variance (based on 231 questionnaires). Most of the items contributed strongly to only one of the extracted factors. Four PRO-CTCAE items, specifically fatigue (severity and interference), difficulty swallowing (severity) and dry mouth (severity), exhibited cross-loadings with other PRO-CTCAE items. Fatigue demonstrated significant cross-loadings with dyspnoea and difficulties with mental concentration, and difficulty swallowing and dry mouth cross-loaded with PRO-CTCAE symptom terms reflecting gastrointestinal symptoms (nausea and anorexia). Detailed PCA results are shown in supplementary figure S1, available at Annals of Oncology online.

The MTMM according to Campbell and Fiske (based on 210 questionnaires) confirmed construct validity [13]. To facilitate interpretation of the MTMM results, Table 2 presents a summary of the main results, showing the proportion of the correlation coefficients that were moderate (0.4–0.6) or high (>0.6), as well as the proportion of non-significant correlation coefficients. For full results, see supplementary figure S2, available at Annals of Oncology online. All monotrait-heteromethod (MTHM) correlation coefficients were ≥0.7. The mean of all MTHM correlation coefficients quantifies convergent validity. With a mean of 0.80 (median 0.83, range 0.70–0.86), convergent validity was high. Heterotrait-monomethod (HTMM) correlation coefficients, representing discriminant validity, were smaller compared with the MTHM. Of the PRO-CTCAE HTMM correlations and of the EORTC QLQ-C30 HTMM correlations 21% and 27%, respectively, were moderate, and none were high. The heterotrait-heteromethod (HTHM) correlation coefficients were the smallest. The proportion of non-significant coefficients was higher than in the HTMM.

Table 2.

Proportion of correlations in the multitrait-multimethod matrix

Correlation type	High correlation (%)	Moderate correlation (%)	Non- significant correlation (%)
Monotrait-heteromethod	100	0	0
Heterotrait- monomethod PRO-CTCAE	0	21	15
Heterotrait- monomethod EORTC QLQ- C30	0	27	20
Heterotrait-heteromethod	0	15	26

The assessment of known groups validity indicated that PRO-CTCAE scores were statistically significantly different in the two groups that also differed with respect to HRQL. Those respondents with better HRQL (EORTC QLQ-C30 item 30 score >4) showed statistically significantly lower PRO-CTCAE scores compared with those patients with lower HRQL (EORTC QLQ-C30 item 30 score ≤4). These differences in PRO-CTCAE scores were largest for fatigue (mean 1.41 versus 2.54) and least pronounced for diarrhoea (mean 0.51 versus 0.83) and mucositis (mean 0.45 versus 0.82), see Figure 1.

Figure 1.

Known groups validity for the criterion health-related quality of life (--- EORTC QLQ-C30 item 30 score ≤4, n = 131; ^..... EORTC QLQ-C30 item 30 score >4; n = 114), P value adjustment for multiplicity.

For the known groups analyses based on age (>60 versus ≤60 years of age) or disease site (breast cancer versus all other tumour sites), there were no statistically significant differences in PRO-CTCAE mean scores, see supplementary figure S3 and figure S4, available at Annals of Oncology online.

discussion

A German language core item set of the PRO-CTCAE was psychometrically tested using a variety of approaches, and the examined items demonstrated generally favourable measurement properties. These results are the first quantitative evidence that a subset of the PRO-CTCAE item library in the German language meets accepted criteria with respect to item quality, reliability, and validity for use as a patient-reported measure of symptomatic toxicity in cancer clinical trials.

The return rate of 41% was just acceptable since rates of 50%–70% are generally recommended. However, the sample size of 262 participants was sufficiently large to adequately power these analyses.

Inspection of the endorsement patterns did not reveal logically untenable response patterns (e.g. endorsing none for severity but a little for interference), and there were only few missing values for PRO-CTCAE. These observations suggest that PRO-CTCAE was well comprehended by German speakers.

While Bowker's test results suggested some redundancy between the frequency and severity attributes for three symptoms (nausea, vomiting, and pain), these observations should not be over-interpreted. Very few respondents were experiencing vomiting, and the low variability in this symptom may have produced an inflated level of concordance. The inclusion of multiple attributes is intended to improve the precision of PRO-CTCAE in capturing the latent construct (e.g. pain that is severe but infrequent). Replication of these results across multiple samples is needed in order to draw conclusions about whether these PRO-CTCAE items are understood by German speakers as reflecting distinct attributes of their symptom experience. Cronbach's alpha values revealed an excellent reliability of the PRO-CTCAE core item set.

Findings from the PCA, MTMM, and known groups analyses support the good validity of the German language PRO-CTCAE items and extend the linguistic validation of the PRO-CTCAE German as carried out by Kirsch et al. [6]. Validity is the most fundamental consideration in evaluating a new PRO measure [17]. The PCA yielded a set of 10 factors that were conceptually consistent with the intended dimensionality of the PRO-CTCAE item library [18]. For each PRO-CTCAE symptom term, the items reflecting attributes of frequency, severity, and interference generally loaded most strongly on a single factor. Interpretation of the very limited number of cross-loadings observed was conceptually consistent with the available knowledge of symptom clusters [19] in that fatigue severity and interference cross-loaded with problems with mental concentration and with dyspnoea, whereas the severity of swallowing problems and xerostomia each cross-loaded with other gastrointestinal symptoms.

Construct validity was also supported by the results of the MTMM analysis. Convergent validity as first criterion was fulfilled, with a mean MTHM correlation coefficient of 0.8 (median 0.83, range 0.70–0.86). Correlation coefficients approaching 1.0 indicate that a new measure may be too similar to the comparison measure and therefore redundant [20]. For the analysis of PRO-CTCAE, the mean correlation coefficient, however, clearly deviated from a perfect correlation so that redundancy is not given. Discriminant validity as second criterion was met since HTMM correlation coefficients were significantly smaller compared with the MTHM. The third criterion, HTHM correlation coefficients being the smallest, was also met. However, the fourth criterion, requiring similarity of the HTMM and HTHM pattern, could not be evaluated due to the complexity of the correlation matrix. In summary, the MTMM of PRO-CTCAE met three of the four criteria of Campbell and Fiske [13], supporting the construct validity of a PRO-CTCAE core item set in the German language. Additional evidence of the construct validity of PRO-CTCAE German was demonstrated via known groups validity technique, which revealed that there were statistically significant differences in PRO-CTCAE scores between those respondents with impaired and preserved HRQL.

Several caveats should be considered in interpreting these findings. First, as only a subset of the PRO-CTCAE German language item library was evaluated, results may not generalise to the entire German language PRO-CTCAE item library. Second, due to the selection of the study centres, there was over-representation of women with breast cancer in the study sample, and the sample mean age was somewhat younger than the average age (69 years) of cancer onset in Germany [21]. Lastly, clinical data about current receipt of treatment were not collected. Thus, the degree to which the sample is sufficiently representative of a diverse population of cancer patients and survivors is not known. Therefore, additional studies are needed to extend the findings of this study.

conclusion

In conclusion, a subset of items from the German language version of PRO-CTCAE was found to be reliable and valid suggesting that PRO-CTCAE German can be used for patient self-reporting of symptomatic AEs. Data derived from this instrument can supplement CTCAE toxicity evaluations in future clinical cancer trials in German-speaking countries.