Independent test of a model to predict severe acute esophagitis

Abstract

Purpose

Treatment planning factors are known to affect the risk of severe acute esophagitis during thoracic radiation therapy. We tested a previously published model to predict the risk of severe acute esophagitis on an independent data set.

Methods and materials

The data set consists of data from patients who had recoverable treatment plans and received definitive radiation therapy for non–small cell carcinoma of the lung at a single institution between November 2004 and January 2010. Complete esophagus dose-volume and available clinical information was extracted using our in-house software. The previously published model was a logistic function with a combination of mean esophageal dose and use of concurrent chemotherapy. In addition to testing the previous model, we used a novel, machine learning-based method to build a maximally predictive model.

Results

Ninety-four patients (81.7%) developed Common Terminology Criteria for Adverse Events, Version 4, Grade 2 or more severe esophagitis (Grade 2: n = 79 and Grade 3: n = 15). Univariate analysis revealed that the most statistically significant dose-volume parameters included percentage of esophagus volume receiving ≥40 to 60 Gy, minimum dose to the highest 20% of esophagus volume (D20) to D35, and mean dose. Other significant predictors included concurrent chemotherapy and patient age. The previously published model predicted risk effectively with a Spearman's rank correlation coefficient (r_s) of 0.43 (P < .001) with good calibration (Hosmer-Lemeshow goodness of fit: P = .537). A new model that was built from the current data set found the same variables, yielding an r_s of 0.43 (P < .001) with a logistic function of 0.0853 × mean esophageal dose [Gy] + 1.49 × concurrent chemotherapy [1/0] − 1.75 and Hosmer-Lemeshow P = .659. A novel preconditioned least absolute shrinkage and selection operator method yielded an average r_s of 0.38 on 100 bootstrapped data sets.

Conclusions

The previously published model was validated on an independent data set and determined to be nearly as predictive as the best possible two-parameter logistic model even though it overpredicted risk systematically. A novel, machine learning-based model using a bootstrapping approach showed reasonable predictive power.

Summary.

Treatment planning factors are known to affect the risk of severe acute esophagitis during thoracic radiation therapy. We tested a previously published model (logistic function with combination of mean esophageal dose and use of concurrent chemotherapy) to predict the risk of severe acute esophagitis on an independent dataset. Its predictive performance was nearly as good as the best logistic model that consisted of the same two variables found previously, derived from the new dataset.

Introduction

Severe acute esophagitis (AE) is a common, dose-limiting side effect of radiation therapy for patients with non–small cell lung cancer. When it occurs, AE often peaks in the first few weeks of a course of radiation therapy.1, 2, 3 Patient-related, tumor-related, and treatment-related risk factors3, 4, 5, 6, 7, 8 that have been reported to be statistically associated with the incidence or severity of AE include age, tumor nodal stage, concurrent chemotherapy, and body mass index. Two recent reviews9, 10 summarized the dosimetric predictors. Rose et al⁹ systematically reviewed 18 published studies of patients with non–small cell lung cancer who had radiation-induced esophagitis.3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23 Eleven studies specifically assessed AE, and the other studies assessed acute and chronic radiation-induced esophagitis together. Five dosimetric parameters were identified as predictive of AE with or without chemotherapy: mean esophageal dose (MED), maximal esophageal dose, percentage of esophagus volume receiving ≥20 Gy (V20), V35, and V60. In a QUANTEC review article, Werner-Wasik et al¹⁰ noted disparities in the dosimetric parameters that were identified as most predictive of AE.

Our recently published analysis²⁴ of a large cohort of single institution data suggested that a two-variable logistic model based on MED and use of concurrent chemotherapy robustly predicts risk of AE in a combined data set that includes data from patients at our institution between 1991 and 2000 and from the Radiation Therapy Oncology Group (RTOG) 93-11 trial. The main purpose of this study is to test the published two-variable model on a new, independent data set; update the model for clinical use; and propose a novel machine learning-based predictive model.

Methods and materials

Patient cohort

This study received approval from an institutional review board, and all patients provided informed consent. All patients who were treated at Washington University School of Medicine in St. Louis between November 2004 and January 2010, who received a minimal target dose of 46.2 Gy, and whose dose distributions were computed with modern dosimetry (ie, collapsed cone algorithm used in the Pinnacle treatment planning system as opposed to water-based methods) were included in this study. General patient characteristics are provided in Table 1.

Table 1.

Patient characteristics

Parameter	All patients (n = 115)	Grade <2 (n = 21)	Grade ≥2 (n = 94)	P value
Age (years), mean (range)	64.2 (37-86)	69.9	62.9	.005
Gender, n (%)				.950
Male	50/115 (43.5%)	9	41
Female	65/115 (56.5%)	12	53
Race, n (%)				.771
White	76/115 (66.1%)	14	62
Black	36/115 (31.3%)	6	30
Other	3/115 (2.6%)	1	2
Performance status (ECOG/ZUBROD SCORE), n (%)				.969
0	48/115 (41.7%)	8	40
1	50/115 (43.5%)	9	41
2	13/115 (11.3%)	2	11
Unknown	4/115 (3.5%)	2	2
Weight loss, n (%)				.235
<10%	87/115 (75.6%)	18	69
>10%	28/115 (24.4%)	3	25
Tumor stage, n (%)				.696
≥2	86/115 (74.8%)	15	71
<2	29/115 (25.2%)	6	23
Primary histology, n (%)				.272
Squamous cell	34/115 (29.6%)	7	27
Adenocarcinoma	73/115 (63.4%)	11	62
Non–small cell	8/115 (7.0%)	3	5
Treatment time (days), mean (range)	48.6 (31-68)	47.9	48.7	.836
Fraction size (Gy), median (range)	2.0 (1.8-2.5)	2	2	.296
Chemotherapy, n (%)		23	125	.062
Pre-radiation therapy	9/115 (7.8%)	0	9
Concurrent	85/115 (73.9%)	10	75
Post-radiation therapy	54/115 (47.0%)	13	41
Smoker, n (%)				.488
Current	42/115 (36.5%)	7	35
Former	68/115 (59.1%)	14	54
Never	5/115 (4.4%)	0	5
Mode of radiation therapy, n (%)
IMRT	30/115 (26.1%)	5	25	.793
3DCRT	85/115 (73.9%)	16	69

The χ² test was used for categorical variables, and Wilcoxon rank-sum test was used for continuous variables.

Treatment characteristics

Patients were treated either with two-course radiation therapy (73.9%) or a full course of image guided radiation therapy (26.1%). Two-course radiation therapy consisted of parallel opposed (anterior-posterior and posterior-anterior) beams for a few weeks, followed by off-cord oblique beams to spare part of the esophagus. Nine patients (7.8%) received neoadjuvant chemotherapy, 85 patients (73.9%) received concurrent chemotherapy, and 54 patients (47.0%) received adjuvant chemotherapy. Five patients (4.4%) received both neoadjuvant and concurrent chemotherapy, but 41 (35.7%) received both concurrent and adjuvant chemotherapy. One patient received neoadjuvant, concurrent, and adjuvant chemotherapy.

Acute esophagitis events

Complications were defined as Common Terminology Criteria for Adverse Events, Version 4 (CTCAE V4.0), Grade ≥2 events (ie, events that required medical attention or care). Ninety-four patients (81.7%) developed severe esophagitis, 79 of whom (68.7%) had Grade 2 complications and 15 of whom (13%) developed Grade 3 complications. No Grade 4 or 5 events were reported.

Dose-volume parameters

A wide range of esophagus dose-volume parameters was extracted for modeling. The entire length of the esophagus was contoured. D_x is defined as the minimum dose to the x% volume of the esophagus receiving the highest doses. V_x is the percentage volume of the esophagus receiving at least x Gy. MOH_x is defined as the mean dose of the x% volume of the esophagus receiving the highest doses, and MOC_x is similarly defined as the mean dose of the coldest x% volume of the organ. MOH_x and MOC_x were included in the model because they could potentially be more robust predictors of dosimetric effect than D_x and V_x due to their averaging characteristic compared with focusing on a single dose-volume histogram point. Thus, the dosimetric parameters of esophagus that were analyzed included D5 to D100, V5 to V100, MOH5 to MOH100, and MOC5 to MOC100 (in increments of 5), mean dose, International Commission on Radiation Units and Measurements maximum dose (ie, highest mean dose to any cuboid volume at least 1.5 cm on a side), and International Commission on Radiation Units and Measurements minimum dose (ie, lowest mean dose to any cuboid volume at least 1.5 cm on a side). Clinical variables included age, sex, race, Karnofsky performance status, weight loss, chemotherapy (neoadjuvant or concurrent), histology, and clinical stage.

Statistical analysis and modeling

Our first step was to test the previously published AE model²⁴ on the current cohort. To find a new model on the current cohort, univariate logistic analysis was performed for each available variable and r_s was computed to assess univariate correlation with risk of AE. Finally, multivariate logistic regression modeling was conducted as designed by Huang et al.²⁴ In this approach, during leave-one-out cross validation (LOOCV), the most frequently generated model was chosen as a final predictive model, and all samples were input into the model to calculate its predictive power.

In addition, as described in the supplementary material, we designed a novel two-step process to produce predictive models on the basis of machine learning methods. In the first step, we filtered irrelevant variables (ie, those with low univariate correlations) and reduced competing, redundant variables, which left 12 variables (D10, D25, D40, V20, V50, V65, MOH15, MOH45, age, grade, concurrent chemo, and mean esophagus dose) for input into the next step. A preconditioning step that was inspired by a study by Debashis et al²⁵ was applied next. The goal of preconditioning is to introduce a more useful response variable of outcomes than the observed binary outcomes. This is useful if the model to generate preconditioning has validity and is complementary to the next modeling stage. Finally, filtered variables with preconditioned outcomes were fed into a least absolute shrinkage and selection operator (LASSO) modeling process. We generated 100 bootstrapped data sets with replacement. At each iteration, a logistic regression model with forward variable selection was generated and its outputs (normal tissue complication probability [NTCP]) were used as a response variable in the LASSO regression modeling with 12 variables as independent variables. Predictive LASSO outputs were used to assess correlation with risk of AE. After completion of the entire process, the predictive performance was averaged.

Results

Test of the previously published AE model

The previously published two-variable model to predict risk of AE²⁴ that is shown below uses the variables of mean esophagus dose and concurrent chemotherapy (ConChemo = 1 if given, and = 0 if not given):

$$\text{NTC}{\text{P}}_{\text{AEH012}}=\frac{\text{1}}{\text{1}+\text{exp}\left(-{\text{x}}_{\text{A}\text{E}\text{H}\text{012}}\right)}\text{,}$$

where ${\text{x}}_{\text{AEH}\text{012}}=\text{0.0688}\times \text{M}\text{E}\text{D}+\text{1.5}\times \text{C}\text{o}\text{n}\text{C}\text{h}\text{e}\text{m}\text{o}–\text{3.13}$.

The model performed well on the current cohort with an r_s of 0.43 (P < .001) and good calibration (Hosmer-Lemeshow goodness of fit: P = .537, degrees of freedom = 6). Figure 1 shows the estimated logistic regression curves as a function of mean esophageal dose with and without concurrent chemotherapy for the AEH012 model. The model-predicted incidence of AE versus observed incidence is displayed in Figure 2, with patients divided into 6 equal bins and grouped in accordance with the model-predicted risk.

Fig. 1.

Estimated dose response curves according to the AEH012 model (red lines) versus the new model (blue lines) for Grade 2 or greater acute esophagitis with or without concurrent chemotherapy.

Fig. 2.

Mean predicted rates of acute esophagitis versus observed rates for patients. The patients were binned in accordance with the predicted risk of acute esophagitis by the AEH012 two-variable model. The mean predicted and observed event rates in each bin are (predicted risk; events/pts) (0.4590, 10/20), (0.6820, 14/19), (0.7843, 17/19), (0.8515, 18/19), (0.9023, 18/19), (0.9463, 17/19).

Univariate logistic regression

Mean, median, and range values for representative D_x, V_x, and MOH_x variables for the esophagus are listed in Table 2. Eighteen variables with r_s >0.3 and significance of P < .0005 are listed in Supplemental Table 1.

Table 2.

Summary of dosimetric statistics for the esophagus, mean (SD)

	Variable	Grade <2	Grade ≥2	P-value
Dx (Gy)	D10	49.1 (20.9)	64.4 (9.9)	.0004
	D20	41.1 (22.2)	59.3 (13.3)	.0003
	D30	33.8 (22.6)	52.6 (17.6)	.0005
	D40	25.2 (18.9)	41.5 (22.2)	.0023
	D50	16.3 (15.7)	29.2 (22.7)	.0291
	D60	11.0 (13.3)	17.9 (19.6)	.1840
	D70	7.3 (9.7)	9.1 (12.8)	.7860
	D80	2.9 (3.3)	4.0 (7.4)	.9049
Vx (%)	V10	48.3 (24.4)	57.6 (16.1)	.1205
	V20	40.1 (22.4)	52.6 (17.0)	.0293
	V30	30.6 (20.4)	48.3 (17.2)	.0004
	V40	24.8 (19.6)	43.7 (17.6)	.0001
	V50	18.7 (18.5)	37.7 (18.0)	.0001
	V60	12.3 (15.0)	28.5 (16.9)	.0001
	V70	2.8 (7.1)	8.0 (12.6)	.0871
MOHx (Gy)	MOH10	54.3 (19.0)	66.6 (8.6)	.0018
	MOH20	49.7 (19.8)	64.3 (9.5)	.0004
	MOH30	45.6 (20.3)	61.6 (10.9)	.0003
	MOH40	41.5 (20.1)	58.0 (12.5)	.0003
	MOH50	37.2 (18.9)	53.5 (13.8)	.0003
	MOH60	33.2 (17.3)	48.4 (14.2)	.0005
	MOH70	29.8 (16.0)	43.2 (13.8)	.0008
	MOH80	26.5 (14.5)	38.3 (12.9)	.0013
Other (Gy)	Mean Dose	21.8 (11.6)	31.6 (10.4)	.0011
	Minimum Dose	0.9 (1.6)	0.8 (1.4)	.6263
	Maximum Dose	62.9 (15.3)	69.7 (8.0)	.0767

D_x, minimum dose to the x% volume of the esophagus receiving the highest doses; MOH_x, mean dose of the x% volume of the esophagus receiving the highest doses; V_x, percentage volume of the esophagus receiving at least x Gy.

Multivariate risk modeling

A new two-variable model on the new cohort was suggested with use of the model building process that was employed by Huang et al²⁴ on the basis of LOOCV. Interestingly, the most frequently generated model during LOOCV consisted of the same combination of MED and ConChemo as the previous model. The resulting fitted risk of AE as a logistic function is given by:

$$\text{NTCP}=\frac{\text{1}}{\text{1}+\text{exp}\left(-\text{x}\right)}\text{,}$$

where $\text{x}=\text{0.0853}\times \text{M}\text{E}\text{D}+\text{1.49}\times \text{C}\text{o}\text{n}\text{C}\text{h}\text{e}\text{m}\text{o}–\text{1.75}$.

The r_s of this risk model on the current cohort was 0.43 (P < .001; note that it was not cross-validated). The Hosmer-Lemeshow test showed a good fit in this model (P = .659, degrees of freedom = 6). Figure 1 shows the estimated logistic regression curves as a function of MED with and without chemotherapy. Figure 3 compares the model-predicted incidence of AE and the observed incidence with patients divided into 6 equal bins and grouped in accordance with the model-predicted risk. For ConChemo and MED, the odds ratio of the observed AE rate between the one-third of patients at high risk and the one-third of patients at low risk was 3.59 (P = .037) and 1.08 (P = .003), respectively, which is another indicator that the model usefully distinguishes high-risk patients from low-risk patients.

Fig. 3.

Mean predicted rates of acute esophagitis versus observed rates for patients. The patients were binned in accordance with the predicted risk of acute esophagitis by the best fit two-variable model. The mean predicted and observed event rates in each bin are (predicted risk; events/pts): (0.5059, 10/20), (0.7504, 14/19), (0.8465, 17/19), (0.9034, 18/19), (0.9419, 18/19), (0.9715, 17/19).

Model-based preconditioning LASSO regression model

After filtering out irrelevant and redundant variables, the resulting 12 variables were entered in a logistic regression model with forward variable selection on a bootstrapped data set with replacement. Its output (ie, preconditioned outcomes) was entered in a LASSO regression model. After selecting a tuning parameter in LASSO using 10-fold cross validation, the final model was tested with samples that were not included in the bootstrapped data set. We iterated this process 100 times. An example of the LASSO model fit is shown in Supplemental Figure 1. Our approach, coupled with logistic regression and LASSO, performed better than a logistic regression model with forward variable selection alone, which was used to find the best model as shown in the above section with an average r_s of 0.38 and 0.34, respectively.

Discussion

Similarity of current patient cohort to previous cohort

When comparing the current cohort with the previous AE study cohort, there was no difference in terms of age (mean of 65.3 years in the current cohort vs 64.2 years in the previous cohort) and fraction size (1.8-2.5 Gy). In terms of sex, the current cohort included more female (56.5%) than male patients (43.5%), but the previous cohort included an almost even number of male and female patients (51.9% vs 48.1%, respectively). With regard to race, the current cohort was categorized as white 66.1%, black 31.3%, and other 2.6% compared with the previous cohort, which had white 74.7%, black 24%, and other 1.3%. As for chemotherapy, 73.9% of patients in the current cohort received concurrent chemotherapy, but only 36.7% of the previous cohort had concurrent chemotherapy. The increase in use of concurrent chemotherapy is the largest differentiator between the cohorts. Despite this, the models explicitly account for concurrent chemotherapy on a patient-by-patient basis; therefore, the difference does not invalidate the test.

Acute esophagitis events

A difference between this study and the previous study is that in the new cohort, AE events were scored on the basis of CTCAE V4.0 whereas the previously studied cohort was scored using RTOG AE criteria. Despite this difference, RTOG Grades 1 and 2 combined (ie, complications that do not require medical intervention) were judged as comparable with CTCAE V4.0 Grade 2. This correspondence has been used here.

Treatment characteristics

Patients in the previously studied cohort were treated completely with two-course radiation therapy, which included parallel opposed, anterior-posterior/posterior-anterior beams, for the first few weeks, followed by off-cord oblique beams to spare part of the esophagus and spinal cord. In the previous cohort, 54% (128 of 237 patients) received sequential or concurrent chemotherapy. Patients in the current cohort were treated with 3-dimensional conformal radiation therapy or image guided radiation therapy techniques, and all patients received chemotherapy before, during, or after radiation therapy.

Statistical analysis

In terms of univariate analysis, the percentage volume of the esophagus receiving a range of doses including 60, 50, 40, and 30 Gy; mean dose; and concurrent chemotherapy was highly correlated with AE events for both cohorts. The best two-variable model included mean dose of the esophagus and concurrent chemotherapy.

Model comparison

To compare the performance of the best model derived from the previously studied cohort and the best model derived from the current cohort, each model was applied to the current cohort. The predictive power for actual AE events is shown in Table 3. The ability to predict AE events with both models was almost identical in terms of r_s and the area under the receiver operating characteristic curve. In addition, we designed a novel preconditioning LASSO regression model to predict the risk of developing AE. For an unbiased assessment of the model, a bootstrapping approach was used, which resulted in a good level of predictive power.

Table 3.

Performance comparison of the model AEH012 with the new model for acute esophagitis prediction, both on the new data set (note that both models were not cross-validated)

Model	Spearman Correlation	AUC
New model	0.43 (p < 0.001)	0.78
AEH012	0.43 (p < 0.001)	0.78

AUC, area under the receiver operating characteristic curve.

Conclusions

The previously published model to determine the risk probability of severe AE (as indicated by medical management) was validated on a new, independent data set and showed to be nearly as predictive as the best possible two-variable logistic model derived from the new data set. Although we judged the model to be validated and statistically significant, it is important to further test the model with other institutions' data.