Normal tissue complication probability (NTCP) modelling of severe acute mucositis using a novel oral mucosal surface organ-at-risk

Abstract

Purpose

A normal tissue complication probability (NTCP) model of severe acute mucositis would be highly useful to guide clinical decision-making and inform radiotherapy planning. We aimed to improve upon our previous model by using a novel oral mucosal surface organ-at-risk (OAR) in place of an oral cavity OAR.

Methods and Materials

Predictive models of severe acute mucositis were generated using radiotherapy dose to the oral cavity OAR or mucosal surface OAR and clinical data. Penalised logistic regression and random forest classification models were generated for both OARs and compared. Internal validation was performed with 100-iteration stratified shuffle split cross-validation, using multiple metrics to assess different aspects of model performance. Associations between treatment covariates and severe mucositis were explored using RFC feature importance.

Results

Penalised logistic regression and random forest classification models using the oral cavity OAR performed at least as well as the models using mucosal surface OAR. Associations between dose metrics and severe mucositis were similar between the mucosal surface and oral cavity models. The volumes of oral cavity or mucosal surface receiving intermediate and high doses were most strongly associated with severe mucositis.

Conclusions

The simpler oral cavity OAR should be preferred over the mucosal surface OAR for NTCP modelling of severe mucositis. We recommend minimising the volume of mucosa receiving intermediate and high doses, where possible.

Introduction

Mucositis is a common and important acute toxicity of head and neck radiotherapy (RT), which may result in pain, dysphagia [1] and weight loss, and, hence, reduced quality of life [2,3]. Mucositis may lead to missed treatment fractions [4] and is frequently dose-limiting in dose-escalation and accelerated fractionation regimens designed to improve tumour control [5–7]. Furthermore, severe acute reactions have been implicated in the subsequent development of ‘late’ radiation toxicity [8–10]. A normal tissue complication probability (NTCP) model for severe mucositis, with sufficient predictive performance, could be used for clinical decision-support [11]. Associations between RT dose metrics and mucositis could inform changes to the RT planning dose objectives to reduce the incidence of severe mucositis. It has previously been demonstrated that intensity-modulated RT can be used to spare the oral mucosa in oropharyngeal RT patients [12].

Our group has previously generated and internally validated an acute mucositis NTCP model, with modest-to-good discriminative ability (mean area under the receiver operating characteristic curve (AUC) = 0.71, s.d. = 0.09 on internal validation) [13]. The below-perfect predictive performance was attributed to limitations of the organ-at-risk (OAR) segmentation, limitations of the toxicity-scoring instrument and not having data on relevant clinical and biological parameters. Only the first of these can be addressed retrospectively. We hypothesised that the discriminative ability could be improved through a more accurate description of the dose distribution to the oral mucosal surfaces, which were previously described by the oral cavity volume contours (OCC) method [14]. The OCC OAR predominantly describes the dose to the musculature of the tongue and floor of mouth, which are not expected to be relevant for mucositis, and does not incorporate the buccal mucosa or mucosa of the lips. We, therefore, developed a novel oral mucosal surface contours (MSC) OAR structure that includes the mucosal surfaces of the oral cavity and excludes the musculature of the tongue and floor of mouth [15]. It was expected that this novel OAR would be more relevant to NTCP modelling of mucositis. To aid implementation of this OAR to a large cohort of patients, we demonstrated that the segmentation of this structure could be fully-automated using atlas-based segmentation [16].

The aims of this study were to (i) improve the discriminative ability of our, previously reported, acute mucositis NTCP model [13] and (ii) gain greater insight into the RT dose-response relationship of the oral mucosa, through the application of our novel, automatically segmented MSC structure [15,16].

Methods and Materials

Patient data

A cohort of 351 head and neck RT patients, enrolled in six different clinical trials [17–23] (with institutional review board approval and signed patient consent; summarised in appendix A), diverse in terms of primary disease site, RT treatment technique and use of concurrent chemotherapy, was employed. The prescribed dose to the primary planning target volume was either 65 Gy in 30 fractions, 60 Gy in 30 fractions, 67.2 Gy in 28 fractions or 63 Gy in 28 fractions (described in appendix A). Toxicity was consistently scored for all studies using the mucositis score from the Common Terminology Criteria for Adverse Events (CTCAE) versions 2 (mucositis due to radiation) [24] or 3 (mucositis/stomatitis (clinical exam)) [25] instruments, which are near equivalent. Toxicities were recorded prospectively prior to the start of RT, weekly during RT, and at 1 - 4 and 8 weeks after RT. The toxicity endpoint of interest chosen for analysis was the maximum reported mucositis grade. Patients were dichotomised into severe (maximum toxicity score of grade 3 or worse) and non-severe (maximum toxicity score of less than grade 3) mucositis. Patients with baseline toxicity or any missing toxicity scores and a maximum score below 3 were excluded from the analysis. Our missing data handling strategy is discussed in appendix B. Complete DICOM RT data were available for 351 patients. After removing patients who had both missing toxicity data and maximum toxicity of grade 2 or lower, 182 patients were available for analysis. MSC atlas-based segmentation failed for 3 patients (for no obvious reasons), so these were excluded from the analysis, leaving 179. Severe mucositis incidence was 74%. Note that excluding the patients removed due to missing toxicity data skews the incidence values to higher than the actual incidences.

Induction chemotherapy (n = 89), concurrent chemotherapy regimen (cisplatin (n = 64), carboplatin (n = 10), one cycle of cisplatin followed by one cycle of carboplatin (n = 6) or none (n = 99); administered in two cycles, on days 1 and 29 of RT), definitive (n = 149) versus post-operative RT, age (median = 57 years, range = 17 – 88 years), sex (n_male = 114) and primary disease site (nasopharynx (n = 18), oropharynx (n = 100), hypopharynx/larynx (n = 18), unknown primary (n = 8) or parotid gland (n = 35)) were also included as covariates in the models. Unilateral versus bilateral irradiation was not explicitly included as a covariate in the models since it correlates perfectly with parotid gland primary disease site.

Radiotherapy dose data

The oral mucosa was contoured on computed tomography using two different techniques: the current guidelines (OCC) method [14] and our novel MSC technique [15,16]. Mucositis of the portion of the pharyngeal mucosa visible on clinical examination was included in the scoring of mucositis. Therefore, the pharyngeal mucosa was manually delineated from the roof of the nasopharynx to the level of the inferior border of the oral mucosa structure (OCC or MSC as appropriate) and combined with the OCC or MSC structure (denoted OCC-PM and MSC-PM). The inferior extents of the OCC and MSC structures were very similar. Figure 1 shows an example of these structures. Clinical oncologists performed the OCC and PM contouring, following the same guidelines, using the RayStation research version 4.6.100.12 treatment planning system (RaySearch Laboratories AB, Stockholm, Sweden). MSC contouring was performed manually for 41 patients (those included in the atlas and test cohort in [16]). For the remaining patients the MSC was delineated fully automatically. The automatically generated OARs were visually assessed for gross errors. Three patients were excluded due to failure of the automatic segmentation, as previously described in the patient data section. The techniques for manual and automatic MSC segmentation are described in [16].

Figure 1.

Example of oral cavity (OCC; blue), mucosal surface (MSC; green) and pharyngeal mucosa (PM; red) contours.

The physical dose distribution was converted to the fractional dose distribution (physical dose delivered in each fraction), as recommended for modelling of acute toxicity by Tucker et al. [26]. The fractional dose distribution was described by the dose-volume histogram (DVH) in 20 cGy intervals from 20 to 260 cGy per fraction. There may be regional variations in radiosensitivity across the oral mucosa, that cannot be detected by describing the dose distributions using only DVHs. For example, keratinised areas of the oral mucosa might be expected to be associated with lower mucositis scores [27] than non-keratinised regions of the oral mucosa [28]. Therefore, 3D moment invariants, η_abc [13,29] describing the spatial distribution of the dose were calculated using the expression

$${\eta }_{\text{abc}}=\frac{{\mu }_{abc}}{{\mu }_{000}^{\frac{a+b+c}{3}+1}}$$ (1)

where

$${\mu }_{abc}={\displaystyle \sum _x{\displaystyle \sum _y{\displaystyle \sum _z{\left|\left(x-\overline{x}\right)\right|}^a{\left(y-\overline{y}\right)}^b{\left(z-\overline{z}\right)}^{c}D\left(x,y,z\right)I\left(x,y,z\right)}}}$$ (2)

where x, y and z are the voxel coordinates, D(x, y, z) is the dose delivered to the voxel with coordinates (x, y, z), I(x, y, z) is an identity function, which takes a value of 1 if the voxel belongs to the OAR and 0 if it does not, and $\left(\overline{x},\overline{y},\overline{z}\right)$ is the centre of gravity of the OAR. The moments are translational and scale invariant. The left-right symmetry is accounted for by taking the modulus of the $\left(x-\overline{x}\right)$ term. Therefore, the moments in the left-right direction describe how lateralised or centralised the dose is. The different 3D moment invariants describe the centre of mass (η₀₀₁, η₀₁₀, η₁₀₀, η₀₁₁, η₁₀₁, η₁₁₀, η₁₁₁), spread (η₀₀₂, η₀₂₀, η₂₀₀) and skewness (η₀₀₃, η₀₃₀, η₀₀₃) of the dose distribution in the three orthogonal directions (left-right, anterior-posterior, superior-inferior) within each structure. These allow for regional variations in radiosensitivity to be probed. These would manifest as differences in one or more of the moment invariants between patients who experienced severe mucositis and those who did not. An example of the 3D moment invariants is shown in appendix C. The dose metrics were used as covariates in the statistical modelling.

Statistical analysis

All RT dose and clinical covariates were transformed to standardised scores (mean = 0, standard deviation = 1) to avoid scale-related feature dominance. To determine which of the structures resulted in the highest discriminative ability, penalised logistic regression (PLR) [30,31] models were generated and internally validated for both of the structures (OCC-PM and MSC-PM). PLR models reduce the regression coefficients to reduce the risk of overfitting the data. Models were generated with (spa) and without (sta) the addition of the spatial dose metrics. During model generation, the samples of each outcome class were over- or undersampled according to weights inversely proportional to the class frequencies in the training data to account for unbalanced numbers of patients experiencing severe and non-severe mucositis. Model hyper-parameter tuning was carried out using a cross-validated grid-search with 100 iterations of shuffled stratified cross-validation with a train/test split of 80/20. The possible hyper-parameters over which the cross-validated grid-searchers were performed are given in appendix D. To establish whether the discriminative ability could be improved through the use of more a complex model, random forest classification (RFC) [32] models were also generated (our choice of RFC models is discussed in [13]).

The generalisability of the models (aim i) was measured through internal validation. Randomly splitting datasets into a development and independent validation sample, although often performed, provides little additional information over internal validation and increases the risk of bias in small datasets [33,34]. Therefore, all available data were used for generating and internally validating the models. External validation of any promising models should be performed in the future. Internal validation used a nested 100 iteration shuffled stratified cross-validation, with a train/test split of 80/20, incorporating covariate transformation to standardised scores and hyper-parameter tuning with a 5-fold cross-validated grid-search within each iteration to give unbiased error estimates. AUC was used as the scoring metric for the grid-search. The predictive performance of the models was assessed using AUC to measure discrimination, Brier score [35] to measure overall model performance and log loss [36] to assess the model probability estimates. The slope and intercept of a logistic regression model of the actual mucositis outcomes against the predicted probabilities of severe mucositis made by the models was used to measure calibration [37,38]. Consideration of discrimination and calibration metrics and their 95% confidence intervals was used for model comparisons. Selecting models based on formal statistical testing of differences in AUC is inappropriate in this context as this metric gives equal importance weighting to sensitivity and specificity, which does not align with clinical objectives.

To determine associations between treatment factors and mucositis severity (aim ii), the RFC_OCC-PM,spa and RFC_MSC-PM,spa model Gini feature importance values were bootstrapped with 2000 replicates. We have previously demonstrated that this approach is more suitable than the, often-used, logistic regression coefficients, in the context of highly correlated RT dose-volume metrics [13]. Statistical analysis was performed using the Python version 2.7.9 programming language [39] and Pandas version 0.18 [40] and Scikit-learn version 0.17 [41] modules.

Results

The DVH data and correlation matrix for the data are summarised in figure 2 and appendix E, respectively, and demonstrate a relationship between the DVH and mucositis severity for both OARs. The correlation matrix indicates that the DVH-based covariates were highly correlated, both within the same OAR and between the two different OARs. Addressing the first aim of generating models to make accurate predictions of mucositis severity, the metrics describing the predictive performance of the models are shown in table 1. The discriminative abilities of all of the models were modest-to-good. For all models the use of the MSC-PM OAR did not lead to an improvement in predictive performance compared with the corresponding model using the OCC-PM OAR. For both structures (OCC-PM and MSC-PM) and both types of model (PLR and RFC), the addition of the spatial dose metrics did not result in improved model performance, as assessed by any of the metrics. The RFC models were better-calibrated (calibration slope closer to 1 and calibration intercept closer to 0) and provided better probability estimates (lower log loss) and overall performance (lower Brier score) than the PLR models. However, they had slightly worse discrimination (lower AUC). Preliminary analysis indicated that excluding the pharyngeal mucosa from the OARs did not substantially alter any of the results.

Figure 2.

Summary of dose-volume data, for (a) the OCC-PM structure and (b) the MSC-PM structure, grouped by peak mucositis severity. The lines represent the group medians and the error bars represent the bootstrapped 95 percentile confidence intervals on the medians. It should be noted that the volumes are measured at the same fractional dose values for both outcome groups. However, there is an artificial offset to aid visualisation of overlapping error bars.

Table 1.

Measures of predictive performance of models on internal validation.

Model	Hyper-parameters	Mean (standard deviation)
		AUC	Log loss	Brier score	Calibration slope	Calibration intercept
PLROCC-PM,sta	penalty = l1; C = 0.1	0.71 (0.10)	0.66 (0.04)	0.23 (0.02)	11.3 (11.2)	-4.4 (5.5)
PLRMSC-PM,sta	penalty = l2; C = 0.001	0.70 (0.09)	0.66 (0.04)	0.23 (0.01)	14.5 (12.1)	-6.0 (5.9)
PLROCC-PM,spa	penalty = l1; C = 0.1	0.71 (0.11)	0.67 (0.05)	0.23 (0.02)	10.7 (10.9)	-4.1 (5.4)
PLRMSC-PM,spa	penalty = l1; C = 0.1	0.69 (0.09)	0.67 (0.05)	0.23 (0.02)	11.1 (10.3)	-4.3 (5.0)
RFCOCC-PM,sta	max depth = 5, max features = square root	0.69 (0.09)	0.56 (0.08)	0.19 (0.03)	3.3 (2.1)	-1.0 (1.3)
RFCMSC-PM,sta	max depth = 5, max features = square root	0.68 (0.08)	0.56 (0.06)	0.18 (0.02)	3.4 (1.8)	-1.1 (1.1)
RFCOCC-PM,spa	max depth = 5, max features = square root	0.67 (0.10)	0.55 (0.07)	0.18 (0.03)	3.7 (2.6)	-1.4 (1.8)
RFCMSC-PM,spa	max depth = 5, max features = square root	0.67 (0.08)	0.56 (0.07)	0.19 (0.03)	3.4 (2.1)	-1.2 (1.4)

PLR - penalised logistic regression; RFC - random forest classification; l1 - LASSO regularisation; l2 - ridge regularisation; OCC-PM - combined oral cavity and pharyngeal mucosa (extending inferiorly to level of inferior border of oral cavity) contours; MSC-PM - combined oral mucosal surfaces and pharyngeal mucosa (extending inferiorly to level of inferior border of oral mucosal surfaces) contours; sta - standard model (dose-volume metrics only); spa - spatial model (dose-volume plus spatial dose metrics).

The regression coefficients, means and standard deviations for the covariates in the PLR_OCC-PM,sta and PLR_MSC-PM,sta models are given in table 2. These values are required to use the models. It should be noted that the regression coeeficients of PLR models, like other linear models, are unstable in the presence of multicollinearity. This does not prevent them from being to make accurate predictions of patient toxicity outcomes, but does mean that the regression coefficients should not be used to infer associations between the correlated dose metrics and severe mucositis. RFC models were employed for this purpose as they are more robust to multicollinearity.

Table 2.

Regression coefficients and covariate transformation values for PLR_OCC-PM,sta and PLR_MSC-PM,sta models.

Covariate	Regression coefficient		Mean	Standard deviation
	PLROCC-PM,sta	PLRMSC-PM,sta
intercept	0.000	0.002	-	-
definitiveRT	0.000	-0.009	0.832	0.374
male	0.000	0.007	0.637	0.481
age	0.000	-0.006	56.7	11.9
indChemo	0.000	0.010	0.497	0.500
noConChemo	0.000	-0.012	0.553	0.497
cisplatin	0.000	0.013	0.358	0.479
carboplatin	0.000	0.002	0.0559	0.230
cisCarbo	0.000	-0.004	0.0335	0.180
hypopharynx/larynx	0.000	0.005	0.101	0.301
oropharynx	0.000	0.021	0.559	0.497
nasopharynx	0.000	0.005	0.101	0.301
unknown primary	-0.012	-0.011	0.0447	0.207
parotid	-0.243	-0.028	0.196	0.397
V020	0.000	-0.006	96.2/91.8	9.72/16.1
V040	0.000	-0.004	94.6/88.3	12.2/18.3
V060	0.000	0.004	92.6/83.1	14.1/20.0
V080	0.000	0.011	88.2/76.6	17.1/21.7
V100	0.000	0.017	82.5/70.1	21.3/22.4
V120	0.000	0.020	76.8/63.4	24.4/22.3
V140	0.000	0.021	70.9/56.8	25.9/21.6
V160	0.000	0.023	64.5/50.0	26.5/20.4
V180	0.201	0.022	57.2/43.2	26.4/19.4
V200	0.000	0.023	47.4/35.2	27.1/19.6
V220	0.250	0.025	12.4/11.1	11.4/9.41
V240	0.000	0.003	0.170/0.303	1.24/2.16
V260	0.000	0.000	0.000/0.000	1.00/1.00

The regression coefficients of many of the covariates in the PLR_OCC-PM,sta model were set to 0 by the LASSO penalisation. Vx – volume of organ receiving x cGy of radiation per fraction. Mean and standard deviation for dose-volume metrics, Vx are given as OCC-PM/MSC-PM.

Regarding the second aim of establishing associations between treatment covariates, such as RT dose metrics, and mucositis severity, the feature importance values of the RFC models are displayed in figure 3. These indicate that, when the OCC-PM was considered as the OAR, the feature importance of the dose-volume metrics increased with increasing dose, peaking at V180 and V220. The spatial dose metric with the highest feature importance was η₀₁₀, which represents the centre of mass of the dose in the anterior-posterior direction. When the MSC-PM was considered as the OAR, V160 – V220 had the highest feature importance values. A sharp increase in feature importance was observed between V120 and V160. Feature importance declined from V220 to V260. The decline in feature importance does not indicate a reduction in biological effect at these dose levels, rather it is due to lack of variance in these covariates in our dataset (as the values of these dose metrics approach zero for all patients). The spatial dose metric with the highest feature importance was η₁₁₀, which represents the centre of mass of the dose in the diagonal left-right-anterior-posterior direction. Age was the clinical covariate with the highest feature importance in both models. However, it should be noted that this may be related to the fact that RFC feature importance can be artificially inflated for covariates having a large number of different values [42]. Age was negatively correlated with severe mucositis on univariable (appendix E) and multivariable (table 2; small negative regression coefficient) analysis.

Figure 3.

Random forest classification feature importance values for the (a) RFC_OCC-PM,spa and (b) RFC_MSC-PM,spa models, bootstrapped with 2000 replicates. The whiskers show the 95 percentile confidence intervals.

Discussion

We generated models of severe acute mucositis using a novel MSC OAR and spatial dose metrics with the aims of (i) improving the ability of NTCP models to predict which patients would experience severe mucositis and (ii) improving our understanding of the link between the RT dose distribution and severe mucositis. We determined that using our novel MSC OAR to describe the dose distribution did not improve predictive performance over our previously used OCC OAR. This is likely due to the fact that, although the dose metrics between the two contouring approaches were different, they were highly correlated (appendix E). We, therefore, recommend the use of the simpler OCC OAR for developing mucositis severity (not including spatial extent) clinical decision-support tools. The RFC models should be favoured over the PLR models for their ability to make superior probability predictions and only slightly inferior discrimination. Moreover, we established that adding spatial dose metrics, describing the centre of mass, spread and skewness of the dose distribution in 3-dimensions, did not improve the predictive performance of the models, compared with models using only dose-volume-based descriptors of the RT dose distribution. This is likely due to the fact that the CTCAE mucositis-scoring instrument used in the clinical trials does not capture information on the location or extent of mucositis. We suggest that investigators consider the use of a mucositis-scoring instrument sensitive to the spatial extent of mucositis, such as the Oral Mucositis Assessment Scale [43], for the study of radiation-induced mucositis in the future. We expect that the MSC OAR and spatial dose metrics may improve the discriminative ability of models of mucositis extent. However, this requires evaluation.

In regards to the second aim, we determined that the volumes of oral mucosa receiving high and intermediate doses (approximately 160 cGy per fraction and higher) were the treatment covariates with the strongest associations with mucositis severity. Whilst this is intuitive, mean oral cavity dose is currently used as an objective in RT planning protocols (for example in the RTOG 0912, RTOG 0920 and RTOG 1216 trials). The mean dose gives equal importance weighting to low doses as high doses. We recommend that protocols attempt to reduce the volumes of oral mucosa receiving high and intermediate dose levels instead.

Our group has previously published models of severe acute mucositis [13,44], which we attempted to improve upon in this study. We recommend our previous model [13] be favoured over the models in this study, as it was not outperformed by any of those described here. To the best of our knowledge, no other (internally or externally) validated models of severe acute mucositis, allowing individualised risk estimates to be made (aim i), have been published in the intensity-modulated RT era. Conversely, several studies have focussed on establishing the covariates that were most strongly associated with toxicity (aim ii). In support of our findings of associations between RT dose and mucositis, a small prospective study demonstrated that cumulative point doses of 32 Gy or lower (with varying fractionations) were associated with minimal mucositis at those points [27] and a larger study found associations between weekly doses of 10.1 Gy and severe mucositis [45]. Also, a small randomised trial has shown the benefits of sparing the oral mucosa outside of the planning target volume (in patients with oral tongue cancers), when the spatial location of mucositis was included in the toxicity scoring system [46]. Whilst there has been some evidence of increased mucositis with the use of concurrent chemotherapy [45,47], our results, in agreement with several other studies [48,49], did not support a strong association between concurrent chemotherapy and severe mucositis.

Our study possesses several limitations. We did not have access to data pertaining to all of the covariates (for example, genetic polymorphisms [50], tobacco and alcohol) in the published studies and, so, were unable to attempt to replicate all of their findings using our data. Moreover, competing dynamic biological processes govern the onset, progression and resolution of mucositis [28]. Substantial improvements in predictive performance may require relevant data relating to those processes. We recommend that, in the future, investigators should endeavour to generate NTCP models featuring both relevant biological data (for example [51]) and a sufficiently detailed description of the RT dose distribution (such as that employed in this study). Furthermore, the dose distribution was described using the dose calculated on planning computed tomography scans, which is an approximation of the delivered dose, likely featuring inaccuracies due to interfraction motion and patient positioning errors. Image-guide RT images were unavailable for many of the patients in this study so dose accumulation [52] could not be performed.

Conclusions

We determined that using a novel MSC OAR did not improve the predictive performance of severe acute mucositis NTCP models. Exploring dose-response associations using both OCC and MSC OARs led us to recommend that RT planning protocols should prioritise the reduction of the volumes of oral mucosa receiving high and intermediate doses, rather than reducing mean dose.