Individualized Risk Prediction of Outcomes for Oral Cavity Cancer Patients

Abstract

Background

Optimal management of oral cancer relies upon accurate and individualized risk prediction of relevant clinical outcomes. Individualized prognostic calculators have been developed to guide patient–physician communication and treatment-related decision-making. However it is critical to scrutinize their accuracy prior to integrating into clinical care.

Aim

To compare and evaluate oral cavity cancer prognostic calculators using an independent dataset. Methods: Five prognostic calculators incorporating patient and tumor characteristics were identified that evaluated five-year overall survival. A total of 505 patients with previously untreated oral cancer diagnosed between 2003 and 2014 were analyzed. Calculators were applied to each patient to generate individual predicted survival probabilities. Predictions were compared among prognostic tools and with observed outcomes using Kaplan-Meier plots, ROC curves and calibration plots.

Results

Correlation between the five calculators varied from 0.59 to 0.86. There were considerable differences between individual predictions from pairs of calculators, with as many as 64% of patients having predictions that differed by more than 10%. Four of five calculators were well calibrated. For all calculators the predictions were associated with survival outcomes. The area under the ROC curve ranged from 0.65 to 0.71, with C-indices ranging from 0.63 to 0.67. An average of the 5 predictions had slightly better performance than any individual calculator.

Conclusion

Five prognostic calculators designed to predict individual outcomes of oral cancer differed significantly in their assessments of risk. Most were well calibrated and had modest discriminatory ability. Given the increasing importance of individualized risk prediction, more robust models are needed.

Introduction

The management of head and neck cancer (HNC) is complex and relies heavily upon risk prediction of clinical outcomes (1). Oral cavity mucosal squamous cell carcinoma (OCSCC) continues to be difficult to treat, with patients engendering significant disease and treatment-related morbidity (2, 3, 4, 5), with only slight improvement in cure rates observed over the past 30 years. Treatment decisions have traditionally been based on tumor staging characteristics, but with increasing understanding of the impact of specific biologic, genetic and patient health characteristics, better decision making tools are needed to guide treatment selection. It is important to further develop and improve methods to help guide patient and physician individualized risk prediction.

Predicting survival for an individual patient’s malignancy is challenging. Survival is influenced by numerous factors, including multiple and diverse tumor specific (size, grade, genomics, biological features, and stage) and patient related (age, race, gender, immune status, smoking status and medical comorbidities) factors. The TNM (tumor–node–metastasis) staging system defined by the American Joint Committee on Cancer (AJCC) is an effective tool for prognostic prediction in oral cavity cancer patients (6). TNM staging provides a good estimate for survival of large groups of patients but it lacks the detailed information that facilitate accurate predictions of prognosis for individual patients. Furthermore, the TNM system does not incorporate new prognostic variables and does not adapt to advances in our understanding of cancer biology (6). Studies suggest that using prognostic algorithms which integrate multiple patient and tumor related factors can improve prognostic accuracy (7, 8).

Prognostic calculators for HNSCC have recently received increasing interest and several have been developed to guide patient–physician communication and decision-making (9-13). However, it is important that this information be both accurate and precise. The AJCC has recently published sixteen criteria for endorsement of any probability or risk model that allow for differences in the variables that the prognostic calculators use to make the predictions (14).

The accuracy, reliability and utility of the risk calculators currently available for head and neck cancer are not well studied. In theory, variance in predictive discrimination could lead to different recommendations for patients based solely on which calculator is used. Consequently, it is important to evaluate the accuracy of the existing online head and neck cancer prognostic tools. To better understand these issues, we sought to characterize the reliability and accuracy of five oral cancer prognostic calculators using a prospective epidemiology dataset involving previously untreated patients with OCSCC.

Material and methods

Prognostic Calculators

Head and neck cancer prognostic tools were identified using two approaches. First, a Pubmed search for peer-reviewed publications was performed using a combination of search terms (cancer, head and neck, oral cavity), prognosis (i.e. survival, and prediction), and methodology (calculator, nomogram). Results were reviewed to identify all published prognostic tools. Second, input from a multidisciplinary group of head and neck cancer specialists was requested regarding any existing or emerging prognostic tools that they might be aware of or use in their own practice, that were not identified in the online search.

Prognostic calculators identified through these searches were then assessed for eligibility. Inclusion criteria stipulated applicability to OCSCC, and provision of 5 year overall survival prediction estimates. Five prognostic calculators were identified (15-19). Each calculator was reviewed for content and format, and formulas for the models were either requested from the respective authors or derived.

The 5 calculators were developed from different data sources and have undergone different amounts of external validation. Table 1 summarizes the data sources of each calculator, including the period of the cohort, the sample size and other characteristics of the developmental cohorts.

Table 1.

Summary of datasets and models for each calculator

Calculator	Cancers in training dataset	Training dataset	Validation dataset	Model Type	Model Details
Knight (Wang et al 2011)	HN	SEER 1995-2003		Cox regression	Main effects only. Spline function fit for age.
LifeMath (Emerick et al 2013)	HN	SEER up to 2009	Massachusetts General Hospital 1362 patients	Statistical- mechanistic model of cancer metastasis.	Most complicated. -separate tumor and node contributions. -interactions and lots of parameters.
Leiden (Datema et al 2010)	HN	1371 patients at Leiden University Medical Centre 1981-1999	598 pts Barnes- Jewish Hospital between 1995- 2000	Cox regression	Main effects only.
MyCancer Journey (MyCJ)	All cancers	SEER 1973-1996 and 11,791 Barnes-Jewish Hospital patients 1995-2001		Cox regression	Main effects and many interactions.
MSK (Montero et al 2014)	Oral Cavity	1617 Memorial Sloan Kettering Cancer Center patients from 1985-2009		Cox regression	Main effects only. Spline function for age.

Each calculator was based on an equation that the investigators derived from their available data. The exact input variables for each calculator differed, both in terms of which variables were included and the form in which they were included. Inputs of each calculator are summarized in Table 2. The exact equation for each calculator was determined either from the publication or from the website or by reconstructing it through a trial and error process. The equations are given in the supplementary material section.

Table 2.

Summary of input variables for each calculator

	Demographic				Site		Staging									Unique
Calculator	Age	Sex	Race	Comorbidity	Oral Cavity Sub site	other HN sites	SEER TNM	T Size	T	T4	N	# Nodes	M	Muscle Invasion	Grade	ECS	Other
Knight	Y	Y	Y		Y	Y	Y			Y					Y
LifeMath	Y		Y			Y		Y		Y	Y	Y				Y
Leiden	Y	Y		Y		Y			Y		Y		Y				Prior tumors (yes/no)
MyCancer Journey (MyCJ)	Y	Y	Y	Y		Y	Y								Y		Histology (SCC/other)
MSK	Y		Y	Y				Y			Y			Y			smoking

The arithmetic average of the predictions from the 5 calculators was tested as a distinct (sixth) calculator.

Patients

The analysis data set represents a single-institution prospectively maintained head and neck cancer epidemiologic database (20, 21, 22). Patients with biopsy-proven, previously untreated, oral cavity squamous cell carcinoma diagnosed and treated with curative intent at the University of Michigan Health System between 2003 and 2014 were included. Median follow-up was 53 months. Tumor and patient specific variables were exported from the database, and confirmed via chart abstraction.

The calculators were designed to be used prior to any intervention, thus clinical information about tumor size and nodal stage and extent were used, with pathological information substituted only if the clinical information was not available.

Table 3 provides summary demographics for the 505 patients from the University of Michigan. Because of a small amount of missing data, a final analysis dataset of 492 patients where all required data were available was used for each calculator.

Table 3.

Patient Characteristics

	Characteristic		N (%)
Patient	Age at Diagnosis	<40	29 (6%)
		40-49	68 (13%)
		50-59	138 (27%)
		60-69	118 (23%)
		70-79	93 (18%)
		80+	59 (12%)
	Race	White	471 (93%)
		Black	13 (3%)
		Other/unknown	21 (4%)
	Sex	Male	288 (57%)
		Female	217 (43%)
	Smoking Status, n=492	Current Smoker (past 12 months)	215 (44%)
		Former Smoker (quit > 12 months)	152 (31%)
		Never Smoker	125 (25%)
	ACE-27 Comorbidity Score, n=502	none	112 (22%)
		mild	227 (45%)
		moderate	111(22%)
		severe	52 (10%)
	Prior Cancer, n=502	Yes	72 (14%)
		No	430 (86%)
Cancer Site and Stage	Oral Cavity site	Tongue	227 (45%)
		Floor of Mouth	95 (19%)
		Gum/other	180 (36%)
		Lip	3 (<1%)
	Grade, n=481	I: Well	141 (29%)
		II: Moderate	293 (61%)
		III: Poor	47 (10%)
	T class	T1	129 (26%)
		T2	136 (27%)
		T3	50 (10%)
		T4a	187 (37%)
		T4b	3 (<1%)
	N class	N0	315 (62%)
		N1	69 (14%)
		N2a	11 (2%)
		N2b	72 (14%)
		N2c	35 (7%)
		N3	3 (<1%)
	Tumor Size (greatest dimension), n=495	<1.5 cm	14 (3%)
		1.5 cm-2.9 cm	171 (35%)
		3 cm -5.9 cm	241 (49%)
		6 cm+	69 (14%)
	Number of Nodes, n=460	0	315 (68%)
		1	77 (17%)
		2	21 (5%)
		3	18 (4%)
		4	10 (2%)
		5+	19 (4%)
High Risk Features ECS	ESC	Present	62 (12%)
	Muscle Invasion	Present	50 (10%)

Metrics of evaluation

The predicted 5 year survival from each calculator was computed for every patient in the University of Michigan database. The agreement between these predictions was compared using scatterplots, Spearman’s correlation coefficients and measuring the proportion of patients in which the predictions differed by less than 0.10. The calibration of each calculator was assessed using Kaplan-Meier plots where subjects were divided into quintiles, that is 5 equal sized groups based on their predicted risk. The average predicted risk for each group was compared with the estimated 5 year survival for that group in a calibration plot. The discriminatory ability of each calculator was assessed using the area under the ROC curve at 5 years (23) and the C-index.

Results

The patient cohort represents the typical distribution and epidemiology of oral cancer. Most patients were Caucasian, the majority of patients were male, with none or mild comorbidities and more than half were current/former smokers. Slightly less than half had oral tongue primary disease. Additional patient characteristics are displayed in Table 3.

Figure 1 shows the distribution of predictions from the 5 calculators (on the diagonal), the scatterplots showing the agreement between pairs of calculators (below the diagonal) and the correlation between predictors (above the diagonal). The distributions show similar ranges of predictions for the calculators, except for Leiden, which tends to give lower survival predictions. The scatterplots show reasonable association between calculators, except for MyCancerJourney. The correlation coefficients confirm good association between some pairs of calculators. Knight and Leiden (rho=0.86) and Knight and LifeMath (rho=0.82), Weaker associations were found between other pairs with MyCancerJourney and MSK having a correlation of 0.59. Table 7 in the Supplementary Materials shows the percentage of patients for which each pair of calculators that give predictions within 0.10 of each other. There was only a modest level of similarity in the predictions, with many pairs of calculators agreeing to within 0.1 in less than 50% of the patients. Generally, MyCancerJourney showed the least agreement with the other calculators.

Fig. 1.

Predicted five year survival probabilities from the 5 calculators. Histograms of predicted values shown on the diagonal, scatterplots between pairs of calculators shown below the diagonal, correlation coefficients between pairs of calculators shown above the diagonal.

Figure 2 shows Kaplan-Meier plots of the outcome for our patient cohort when they were divided into the five equal size quintile groups based upon predicted risk. All calculators show a spread of survival curves, with the order essentially matching the order of the predicted risk. The calculators with the best separation were MSK and the one based on the mean of the other 5. The calibration plot in Figure 3, of predicted risk versus actual 5 year survival for the quintiles indicates that 4 of the calculators were close to the 45 degree line. The Leiden calculator tended to predict worse 5 year survival than actually observed, particularly in the two highest risk groups. The best calibration was obtained from the mean of all the calculators.

Fig. 2.

Kaplan-Meier plots of overall survival when the subjects were divided into the five equal size quintile groups based upon predicted risk.

Fig. 3.

Calibration plots shows predicted 5 year survival for the five quintile groups versus observed 5 year survival in the University of Michigan data.

Figure 4 shows ROC curves for 5 year survival and the accompanying AUC. The difference between the discriminatory ability of the 5 calculators was small, with all AUCs in the range 0.652 to 0.706. The C-index also showed general similarity between the 5 calculators, with a range of 0.634 to 0.677. The best calculators for discrimination were MSK with an AUC of 0.71 and a C-index of 0.66 and the mean of the other 5, with an AUC of 0.70 and a C-index of 0.68.

Fig. 4.

ROC curves for 5 year survival.

Discussion

We identified five oral cavity squamous cell carcinoma prognostic calculators. Utilizing a set of well characterized patients from a single institution where the outcomes were known, we found significant variability in five year overall survival prediction between the prognostic calculators. This variability between the calculators is concerning and could limit their routine use in counseling patients. There are numerous factors which have been reported to be significant prognostic indictors for patients with OCSCC. Each calculator was derived from a different dataset, with a different set of factors available and different choices made about which factors to include. This variation and the general uncertainty regarding which patient and tumor factors are most important to prognosis in OCSCC may account for the differences in input variables and outcomes across cancer site–specific calculators (19).

Calibration plots assess whether a calculator was accurate at predicting 5 year survival, and the AUC and C-indices assess whether a calculator can discriminate patients into different risk groups. Both are important. Although there are modest differences between the performances of the calculators, no single calculator was either substantially superior or substantially worse than the others. For discrimination, MSK was the best single calculator, with an AUC=0.71 and a C-index of 0.66. The mean of the five calculators had an AUC=0.7 and C-index= 0.68. For calibration, the mean of the five calculators was best, and all the others were similar except for a tendency of the Leiden calculator to predict worse survival outcomes than were actually observed. Overall, C-indices and AUC’s in the range of 0.63 to 0.71 would be considered respectable, but not high enough to provide great clinical utility.

The general similarity in the properties of the calculators probably reflects the overlap of included variables. Differences between calculators are determined by what factors are included in the equations, how they are included, as well as the quality and relevance of the datasets from which they were derived. Comorbidities, which are an important factor in determining overall survival (24), are included in three of the calculators, and MSK was the only calculator to use the Washington University Head and Neck Comorbidity Index (WUHNCI) (16, 24).

For the Knight calculator their internal validation had a C-index of 0.7, whereas our study found a C-index of 0.63. Thus we are providing an external validation of their calculator with somewhat worse properties than the authors had found.

The MSK calculator (16) excluded patients if they had previous head and neck cancer, radiotherapy, chemotherapy, M1 disease, or primary tumors of the lip. This method of construction suggests that this prognostic calculator should be suited to our pre-treatment data set. They did not externally validate their prognostic calculator (16) but we report data similar to their original C-index of 0.67.

The Leiden calculator was externally validated on 598 patients (17,18). They found a C-index of 0.73 with their data and 0.69 when they performed external validation. Our data shows a C-index of 0.66, thus we provide a second external assessment of their calculator, and show slightly worse properties. The Leiden calculator did show a slight tendency to predict worse survival than was actually observed.

For the LifeMath calculator the authors used all head and neck cancer sites in the SEER population (18). As there is significant variation in the behavior and response of squamous cell cancer from different subsites in the head and neck, these differences have to be accounted for, which the authors attempted to do by including a term for cancer site in their equations. Emerick et al, did externally validate their results with a Massachusetts General Hospital data set. Their metric for evaluation was based on a correlation that is not directly comparable to the AUC or the C-index (18). We provide an external validation from our data and find a C-index of 0.665 and an AUC of 0.681.

We were unable to find a peer reviewed publication describing the MyCancerJourney calculator. Our external validation found an AUC = 0.68 and a C-index of 0.65, which was not substantially worse than the other calculators. However, the individual predictions from MyCancerJourney were more variable and could differ substantially from the other calculators.

The calculators developed by Knight, LifeMath, and MyCancerJourney all used data from the SEER database (15, 18). It is important to remember that the accuracy of these prediction tools is dependent upon the limitations of SEER data. For example, if the model predicts a worse outcome for certain ethnic groups, this may be a reflection of historical patterns of socioeconomic disparities (15). Another limitation of the SEER data for predicting overall survival is that it does not contain information on comorbidities. The MyCancerJourney calculator recognized this limitation and included data from their own institution to overcome it. Since all of these prognostic calculators used similar or overlapping data in creation of their models it might be expected that they all give similar outcomes when applied to the same patients. Our data reveals that Knight and LifeMath have rho=0.85 when comparing the distributions of predictions. However, MyCancerJourney and Knight have rho=0.62 while MyCancerJourney and LifeMath have rho=0.7 which show much weaker associations. This indicates that which specific variables to include and how they are included can lead to substantial differences in the prediction of 5 year survival, despite the calculators being derived from the same dataset.

Leiden, Knight and MyCancerJourney all used older patient cohorts. Leiden used 1371 patients treated at LUMC between 1981 and 1999 (17). Knight used SEER data between 1995 and 2003 (15). MyCancerJourney used a combination of SEER patients from 1973 to 1996 and Barnes-Jewish hospital data from 1995 to 2001. Our data set extends from 2003 to 2014. If a majority of the patients used to develop the calculator are from an era prior to the introduction of postoperative chemoradiation, the influence of this modality is unaccounted for in their analysis. This could account for the lack of accuracy in predicted survival when compared to the actual survival in a more current patient data set.

The mean of calculator predictions had the best overall performance, although only marginally better than some individually. This better performance is not surprising, as it is a common occurrence that ensemble methods perform better than any individual predictor. The mean calculator can be thought of as incorporating the “collective wisdom” of the calculators. For practical purposes a calculator that is based on a small number of variables is easier to use, however for obtaining the most accurate predictions limiting to a small number of variables is not likely to be beneficial. The mean calculator does use a larger set of input variables, and this may be one reason for its better performance. From an intuitive perspective the mean calculator works well because it down-weights any outlying high or low predictions that may arise from any one calculator.

The size, quality and relevance of the dataset and the statistical methods used to derive the calculator also impact the performance of predictions. Only the MSK calculator was derived from oral cavity patients alone, with a reasonable sample size of 1617 patients. Other calculators were derived from datasets that included other head and neck sites, enabling a much bigger overall sample size, in the tens of thousands in the case of the Knight and LifeMath calculators. This will help characterize the effect of variables such as age, comorbidities and sex which are likely to have similar effects across different sites. But variables such as tumor size and stage probably have differences between sites, and thus assuming a common effect of these variables for all sites could have a negative impact on the predictions for a particular site. Site differences in the effect of tumor size and stage can be handled by including interactions on the models, as was done by LifeMath and MyCancerJourney, however large datasets need to be available and considerable care taken to avoid over-fitting in such cases.

Since the calculators use overlapping variables in their calculation, patients and physicians would expect that they would give an accurate and consistent prediction for prognosis. The currently existing OCSCC prognostic calculators do not fully meet this expectation and therefore are not sufficiently reliable at predicting prognosis for individual patients. Most prognostic calculators utilize basic patient and tumor characteristics, but make limited use of patient comorbidities and health behaviors. Moreover, they do not incorporate our ever-expanding knowledge of biomarkers and genomic data as predictors of outcome (10). Incorporating more significant biological markers into the prognostic calculation promises to improve the accuracy of these tools. Thus, refining and improving patient-centric individualized prognostic calculators will be critical in risk prediction, treatment planning, and patient counseling (25).

Highlights.

1. Evaluated five calculators of 5 year overall survival in oral cavity cancer patients.

2. There could be large differences in the 5 year predicted risk for an individual patient between the different calculators.

3. Four of the five calculators were well calibrated.

4. The calculators had modest discriminatory ability with C-indices ranging from 0.63 to 0.67