COMPARISON OF PROVIDER-ASSESSED AND PATIENT-REPORTED OUTCOME MEASURES OF ACUTE SKIN TOXICITY DURING A PHASE III TRIAL OF MOMETASONE CREAM VERSUS PLACEBO DURING BREAST RADIOTHERAPY: THE NORTH CENTRAL CANCER TREATMENT GROUP (N06C4)

Abstract

Purpose

Considerable interobserver variability exists among providers and between providers and patients when measuring subjective symptoms. In the recently published Phase III N06C4 trial of mometasone cream vs. placebo to prevent radiation dermatitis, the primary provider–assessed (PA) endpoint, using the Common Toxicity Criteria for Adverse Events (CTCAE), was negative. However, prospectively planned secondary analyses of patient-reported outcomes (PROs), using the Skindex-16 and Skin Toxicity Assessment Tool (STAT), were positive. This study assesses the relationship between PA outcomes and PROs.

Methods and Materials

Pearson correlation coefficients were calculated to compare the three tools. Statistical correlations were defined as follows: <0.5, mild; 0.5–0.7, moderate; and >0.7, strong.

Results

CTCAE dermatitis moderately correlated with STATerythema, and CTCAE pruritus strongly correlated with STAT itching. CTCAE pruritus had a moderate correlation with Skindex-16 itching. Comparing the 2 PRO tools, Skindex-16 itching correlated moderately with STAT itching. Skindex-16 burning, hurting, irritation, and persistence all showed the strongest correlation with STAT burning; they showed moderate correlations with STAT itching and tenderness.

Conclusions

The PRO Skindex-16 correlated well with the PRO portions of STAT, but neither tool correlated well with CTCAE. PROs delineated a wider spectrum of toxicity than PA measures and provided more information on rash, redness, pruritus, and annoyance measures compared with CTCAE findings of rash and pruritus. PROs may provide a more complete measure of patient experience than single-symptom, PA endpoints in clinical trials assessing radiation skin toxicity.

INTRODUCTION

Accurate measurement of the effectiveness and adverse effects of agents tested in cancer control studies is important, but relevant endpoints such as pain, fatigue, and annoyance can be complex and subjective. These symptoms can lead to changes in quality of life and can be associated with patient distress and bother.

Historically, many trials that included measurement of adverse effects of treatment have used the Common Terminology Criteria for Adverse Events (CTCAE) to assess toxicity. CTCAE (version 3.0) is a standardized, provider-reported, objective scale that has been carefully crafted by panels of experts and updated over the years (1). It has been validated for many symptoms. However, provider-assessed (PA) outcomes may have drawbacks (2, 3). Although the CTCAE is designed to measure effects, its ability to summarize toxicities is limited. Additionally, this tool requires clinicians to interpret the patient’s experience and uses ambiguous wording for many of the more subjective categories. Finally, most provider-reported outcome tools do not measure patient bother, quality of life, or treatment satisfaction. For example, even if a provider-reported tool measures a symptom accurately, it may not measure how the symptom affects the way a patient feels or is able to function. A decline in patient function or well-being may be more likely to contribute to treatment breaks or decreased quality of life than a technically severe symptom that does not bother the patient.

Standardized, provider-reported toxicity assessments have good interobserver concordance for objective measures such as laboratory values. However, interobserver variability among providers is much higher when measuring subjective symptoms such as pain, nausea, fatigue, and anorexia (4, 5). Perhaps more important, there is considerable provider–patient discordance when assessing subjective outcomes, with patients reporting greater severity of symptoms than providers (6, 7).

Patient-reported outcomes (PROs) are defined as information that comes directly from the patient, without modification or interpretation by another observer (8). PROs are of particular interest in cancer symptom management trials because these trials specifically address symptoms and quality-of-life outcomes. PROs have been used for more than 30 years in certain subjective assessments such as pain (9). They have been used more recently in a wide range of trials, from stomatitis and nausea to fatigue, anorexia, and quality of life (10–13). PROs also have been correlated with more traditional endpoints such as overall survival (14). Most recently, groups such as the National Cancer Institute, the North Central Cancer Treatment Group, and the U.S. Food and Drug Administration have attempted to outline challenges, successes, and guidelines for PRO use (8, 15, 16). As with PA outcomes or any other measurement tool, PROs must have reliability and validity when measuring the selected outcome (17).

The North Central Cancer Treatment Group recently performed a Phase III randomized controlled trial of mometasone cream vs. placebo for preventing radiation-induced dermatitis in patients undergoing breast or chest wall radiotherapy (N06C4 trial). Symptoms were assessed with the CTCAE version 3.0. As noted in the recently published findings (18), the primary endpoint of PA radiation dermatitis was negative; specifically, the mometasone group had a mean maximum grade of 1.2, and the placebo group grade was 1.3. However, prospectively planned secondary analyses, using the Skindex-16 and Skin Toxicity Assessment Tool (STAT), showed positive results. Patients treated with mometasone reported significantly less itching, burning, irritation, persistence or recurrence of symptoms, and annoyance (18). The current article further assesses the relationship between PA outcomes and PROs in this subset of patients.

METHODS AND MATERIALS

Eligibility criteria for N06C4 have been reported previously (18). Briefly, eligible patients were aged ≥18 years. Patients had an Eastern Cooperative Oncology Group performance status of 0, 1, or 2 and histologic proof of a primary invasive breast carcinoma or ductal carcinoma in situ. Patients underwent a planned course of continuous, definitive, or adjuvant external beam radiotherapy to the whole breast or chest wall, with a minimum prescription dose of 50.0 Gy.

PROs for N06C4 were measured through completion of booklets, including the Linear Analogue Self Assessment questionnaire, the Skindex-16 toxicity measurement instrument, and a symptom experience diary. The booklets were completed at baseline, weekly during radiotherapy, and weekly for a 2-week observation period after radiotherapy. The STAT was used at baseline and weekly during radiotherapy. It included a PRO component of skin symptoms, in addition to a PA component. CTCAE version 3.0 dermatologic adverse effects were assessed by clinical providers at baseline and weekly during radiotherapy.

The magnitudes of the scoring scales were as follows: CTCAE version 3.0, 0–5; STAT, 0–5 (with the exception of erythema, graded on a scale of 0–2); and Skindex-16, 0–6. For all scales, 0 was the best grade. The Skindex-16 total score was the average of all 16 questions. The Skindex-16 instrument had several subscales: the symptom subscale averaged the response to five questions, the emotional subscale averaged the response to six questions, and the functional subscale averaged the response to five questions.

The current analysis compared the results of the Skindex-16, dermatologic CTCAE version 3.0, and patient-reported component of STAT. Pearson correlation coefficients were calculated for scores over all time points. Statistical correlations <0.5 were considered mild, correlations of 0.5–0.7 were moderate but distinct, and those >0.7 were strong, indicating redundancy in the test. Most comparisons involved more than 1,200 measurements. The average number of measurements per comparison was 1,218 (range, 891–1,274 measurements), which provided a high probability (>90%) of detecting a correlation coefficient >0.1. Any correlation coefficient should be accurate to within 0.06 with 95% confidence (19).

RESULTS

Data were collected from 166 patients for eight time points. Table 1 details the correlation coefficients over all time points between the provider-assessed CTCAE and the PRO portion of the STAT. Of note, the CTCAE dermatitis had a moderate correlation with the STAT erythema; also, the CTCAE pruritus correlated strongly with the STAT itching question. No other items appeared to have significant correlations.

Table 1.

Pearson correlation coefficients for provider-assessed CTCAE toxicities and patient-reported STAT scores^*,^†

	STAT
	Erythema		Burning		Itching		Pulling		Tenderness		Other
CTCAE Topic	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson
Skin atrophy	1,273	0.02	1,273	0.03	1,274	0.07	1,274	0.06	1,273	0.07	1,259	0.12
Burn	1,273	0.31	1,273	0.47	1,274	0.28	1,274	0.28	1,273	0.35	1,259	0.10
Dermatology/skin	1,273	0.19	1,273	0.19	1,274	0.21	1,274	0.12	1,273	0.19	1,259	0.08
Hypopigmentation	1,273	0.09	1,273	0.01	1,274	0.05	1,274	0.02	1,273	0.02	1,259	0.06
Pruritus	1,273	0.39	1,273	0.39	1,274	0.74	1,274	0.31	1,273	0.37	1,259	0.12
Dermatitis radiotherapy	1,274	0.64	1,273	0.39	1,274	0.35	1,274	0.34	1,273	0.37	1,259	0.06
Striae	1,273	−0.05	1,273	0.07	1,274	0.12	1,274	0.13	1,273	0.11	1,259	0.01
Skin irritation	920	0.11	921	0.06	921	0.04	921	0.25	920	0.03	910	−0.02

Abbreviations: CTCAE = Common Terminology Criteria for Adverse Events (v. 3.0); STAT = Skin Toxicity Assessment Tool.

^*Statistical correlations <0.5 were considered mild; 0.5–0.7 were moderate; and those >0.7 were strong.

^†Highlighted values are correlation coefficients >0.5.

Table 2 details the correlation coefficients over all time points between the provider-assessed CTCAE and the PRO Skindex-16 scores (total, subscales, and individual symptoms). The only correlations >0.5 were between the Skindex-16 symptom subscale and the CTCAE pruritus item and between the Skindex-16 question of itching and the CTCAE item of pruritus.

Table 2.

Pearson correlation coefficients for provider-assessed CTCAE toxicities and patient-reported Skindex-16 scores^*,^†

	Skindex-16
	Total Score		Subscale		Subscale		Subscale		Itching		Burning		Hurting		Irritation		Persistence
CTCAE Topic	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson
Skin atrophy	1,247	0.04	1,238	0.07	1,247	0.01	1,240	0.01	1,238	0.11	1,238	0.05	1,238	0.05	1,237	0.06	1,237	0.07
Burn	1,247	0.37	1,238	0.41	1,247	0.25	1,240	0.32	1,238	0.27	1,238	0.41	1,238	0.39	1,237	0.40	1,237	0.35
Dermatology/skin	1,247	0.20	1,238	0.21	1,247	0.13	1,240	0.19	1,238	0.19	1,238	0.19	1,238	0.18	1,237	0.20	1,237	0.18
Hypopigmentation	1,247	0.03	1,238	0.03	1,247	0.01	1,240	0.02	1,238	0.08	1,238	–0.01	1,238	0.01	1,237	0.03	1,237	0.03
Pruritus	1,247	0.43	1,238	0.53	1,247	0.29	1,240	0.29	1,238	0.58	1,238	0.44	1,238	0.42	1,237	0.49	1,237	0.43
Dermatitis radiotherapy	1,247	0.34	1,238	0.38	1,247	0.25	1,240	0.24	1,238	0.23	1,238	0.35	1,238	0.36	1,237	0.43	1,237	0.34
Striae	1,247	0.08	1,238	0.15	1,247	0.01	1,240	–0.01	1,238	0.14	1,238	0.15	1,238	0.13	1,237	0.11	1,237	0.13
Skin irritation	897	0.03	892	0.03	897	0.05	891	–0.03	892	0.01	892	0.02	892	0.02	892	0.06	892	0.02

Abbreviation: CTCAE = Common Terminology Criteria for Adverse Events (v. 3.0).

^*Statistical correlations <0.5 were considered mild; 0.5–0.7 were moderate; and those >0.7 were strong.

^†Highlighted values are correlation coefficients greater than 0.5.

Table 3 details the correlation between the PRO STAT and Skindex-16 scores over all time points. Correlations were noted between the Skindex-16 scores for the total and symptom subscale and the STAT items of burning, itching, and tenderness. The Skindex-16 function subscale correlated only with the STAT item of burning, and the Skindex-16 emotion subscale did not have any correlations >0.5. Table 3 also shows the breakdown of Skindex-16 symptom subscales and their correlation to the PRO STAT individual items. The Skindex-16 itching question correlated with the STAT itching question. The Skindex-16 burning, hurting, irritation, and persistence items all showed the strongest correlation with the STAT burning question and moderate correlations with the STAT itching and tenderness questions.

Table 3.

Pearson correlation coefficients for patient-reported STAT and patient-reported Skindex-16 Scores^*,^†

	Skindex-16
	Total Score		Subscale		Subscale		Subscale		Itching		Burning		Hurting		Irritated		SKINDEX Persistence
STAT Topic	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson	n	Pearson
Erythema grading	1,245	0.38	1,236	0.43	1,245	0.27	1,238	0.29	1,236	0.30	1,236	0.39	1,236	0.41	1,235	0.46	1,235	0.39
Burning	1,245	0.64	1,236	0.69	1,245	0.47	1,238	0.51	1,236	0.41	1,236	0.71	1,236	0.67	1,235	0.65	1,235	0.62
Itching	1,246	0.57	1,237	0.65	1,246	0.41	1,239	0.39	1,237	0.69	1,237	0.55	1,237	0.52	1,236	0.58	1,236	0.54
Pulling	1,246	0.46	1,237	0.47	1,246	0.37	1,239	0.37	1,237	0.28	1,237	0.45	1,237	0.49	1,236	0.48	1,236	0.43
Tenderness	1,245	0.57	1,236	0.60	1,245	0.44	1,238	0.44	1,236	0.35	1,236	0.56	1,236	0.64	1,235	0.58	1,235	0.55
Other	1,231	0.12	1,222	0.14	1,231	0.08	1,225	0.07	1,222	0.09	1,222	0.12	1,222	0.17	1,221	0.13	1,221	0.14

Abbreviation: STAT = Skin Toxicity Assessment Tool.

^*Statistical correlations below 0.5 were considered mild; 0.5–0.7 were moderate; and those greater than 0.7 were strong.

^†Highlighted values are correlation coefficients greater than 0.5.

DISCUSSION

The results of the PRO Skindex-16 tool correlated well with the PRO portions of the STAT tool, but neither tool correlated well with the PA CTCAE version 3.0. This work supports previous reports of patient–provider discordance when measuring outcomes, particularly subjective ones. In this study, PROs delineated a wider spectrum of toxicity than did PA measures. PROs provided information on rash, redness, pruritus, and annoyance; this was more detail than that obtained with the CTCAE tool, which provided information regarding only rash and pruritus. PROs may provide a more complete measure of patient symptomology and experience than single-symptom assessment by providers in clinical trials assessing radiation skin toxicity.

Standardized, provider-reported toxicity assessments have good interobserver concordance for objective measures. However, for subjective symptoms such as nausea, diarrhea, pain, fatigue, and other symptoms, they can have high interobserver variability, even among expert clinicians. This has been shown in small studies examining the National Cancer Institute of Canada clinical trials toxicity scale, the World Health Organization toxicity scale, and the CTCAE version 2.0 (4, 5).

Considerable provider-patient discordance can occur when assessing toxicity of more subjective outcomes. Previous trials have shown that outside observers can underestimate both the severity and importance of symptoms. In 2005, Basch et al. (6) reported a trial of 400 patients receiving cancer treatment. Patients and clinicians filled out questionnaires regarding 11 common symptoms. The physician form was the CTCAE version 3.0, and the patient form was the same but with language adapted for nonmedical responders. They showed high agreement between patient- and provider-reported outcomes for observable symptoms such as vomiting and diarrhea. However, patients reported greater severity than providers for subjective symptoms such as fatigue and dyspnea. Parliament et al. (7) also noted this discrepancy as far back as 1985, when they compared physician notes with the patients’ perception of toxicity. They observed the greatest disparities in nausea, vomiting, alopecia, and performance status. Furthermore, they showed that in 46% of cases, the physicians did not even report the patients’ worst symptoms.

Radiation dermatitis appears to be a toxicity that can be objectively and reproducibly described. The CTCAE version 3.0 has been shown to score dermatitis reliably in head and neck patients (20). However, our findings suggest that the CTCAE grade of dermatitis does not necessarily correlate with patients’ symptom experience. Significant differences between the study arms were observed in the PROs for the more subjective symptoms such as itching, burning, persistence and recurrence of symptoms, and annoyance with symptoms, even though the PA CTCAE maximum grade of dermatitis showed similar scores in both arms (18).

In addition, although the study by Parliament et al. (7) showed generally good concordance between the provider and the patient when assessing skin toxicity, they assessed only rash and itching. Similarly, our study also showed some concordance between the CTCAE and PRO STAT when assessing rash and itching, as well as concordance between the CTCAE and Skindex-16 when assessing itching. It should be added, however, that the study by Parliament et al (7) reported a provider-identified skin reaction in only 1 of the 4 patients who reported skin toxicity as their most bothersome symptom. In addition, a meta-analysis by Huschka et al. (21) that examined six lung cancer trials reported a decline in quality of life that occurred earlier than CTCAE-reported changes. These studies suggest some discordance between what clinicians measure and what patients consider to be important. Our study confirms that even in the absence of objective differences in dermatitis, differences in related patient-reported, clinically significant symptoms, when taken together, can give a better picture of the patient’s experience.

We acknowledge the limitations of our study. Although there were more than 1,200 observations for most of the calculations, given the repeated nature of the questions, it is not the same as data from 1,200 different individuals.

On the basis of this report and the previous work in this field, PRO tools should be included in the design of clinical trials assessing radiotherapy skin toxicity. Bruner et al. (2) surveyed 12 cancer cooperative groups from the United States, Europe, and Canada. They reported that patient-reported primary and secondary endpoints were incorporated into approximately 5%–50% of cancer treatment trials and 50%–70% of cancer control trials. In their report, they described challenges of incorporating PROs into clinical trials, including standardizing training and mentoring for conducting PRO assessments and data collection, as well as development of minimal criteria for acceptability. Work regarding minimal criteria is being performed; Revicki et al. (22) have proposed criteria for minimally important differences in PROs. In addition, the Food and Drug Administration has published guidelines for PRO research that are based on its extensive research on quality-of-life issues (23).

PROs clearly are clinically significant, and the next step is to move toward a standardized assessment process. The challenge is to standardize PROs in a manner analogous to that of the CTCAE; in addition, the tool for each specific symptom and symptom group must be validated. These tasks currently are being undertaken by a multi-institutional group that is developing a standardized PRO version of the CTCAE (24), thus continuing the clinical work started by Basch et al (25).

Findings of our study suggest that clinicians do not always know the right questions to ask and that the quality of answers depends on the quality of the questions. The best guide for clinically important outcomes in cancer control studies is the patient, who clearly knows what is most bothersome and distressing. Even symptoms and toxicities that seem objective can have a large subjective component that should be addressed. Development of a reliable, consistent, and standardized patient-reporting tool will help achieve this goal.