Validation of Colorectal Cancer Models on Long-Term Outcomes from a Randomized Controlled Trial

Abstract

Microsimulation models are often used to predict long-term outcomes and guide policy decisions regarding cancer screening. The United Kingdom Flexible Sigmoidoscopy Screening (UKFSS) Trial examines a one-time intervention of flexible sigmoidoscopy that was implemented before a colorectal cancer (CRC) screening program was established. Long-term study outcomes, now a full 17 years following randomization, have been published. We use the outcomes from this trial to validate three microsimulation models for CRC to long-term study outcomes. We find that two of three models accurately predict the relative effect of screening (the hazard ratios) on CRC-specific incidence 17 years after screening. We find that all three models yield predictions of the relative effect of screening on CRC incidence and mortality (i.e., the hazard ratios) that are reasonably close to the UKFSS results. Two of the three models accurately predict the relative reduction in CRC incidence 17-years after screening. One model accurately predicted absolute incidence and mortality rates in the screened group. The models differ in their estimates related to adenoma detection at screening. While high-quality screening results help to inform models, trials are expensive, last many years, and can be complicated by ethical issues and technological changes across the duration of the trial. Thus, well-calibrated and validated models are necessary to predict outcomes for which data are not available. The results from this validation demonstrate the utility of models in predicting long-term outcomes, and in collaborative modeling to account for uncertainty.

Introduction

The impact of colorectal cancer (CRC) screening on population trends in cancer incidence and mortality can be understood through microsimulation models that describe the natural history of the disease. Such models can be used to predict the impact of interventions and can aid in decisions regarding recommendations for CRC screening. The Cancer Intervention and Surveillance Modeling Network (CISNET) consortium includes three CRC models, developed separately by independent modeling groups, to simulate the natural history of colorectal disease in individuals. These models have been used to inform policy decisions regarding CRC screening (Lansdorp-Vogelaar, et al., 2010) (Knudsen, et al., 2010), (Zauber, et al., 2008) (Knudsen, et al., 2016).

While microsimulation models reflect available knowledge and evidence regarding the disease process, some parts of this process are unobservable, and therefore it is important to ensure that model predictions match observed or known outcomes. Model validation is a critical component of developing a sound model and involves predicting outcomes (e.g., incidence of CRC) and comparing these predictions with observed outcomes from published studies. External model validation assesses outcomes not used in model development, and internal model validation compares predicted outcomes to those used in model development. The three CISNET models were previously validated to the United Kingdom Flexible Sigmoidoscopy Screening (UKFSS) Trial of screening for CRC with a single flexible sigmoidoscopy in a population not yet routinely screened for the disease (Atkin, et al., 2010) (Atkin, et al., 2002).

The previous validation test of the CISNET models involved using the models to predict published results from the UKFSS Trial at 10 years following randomization (Atkin, et al., 2010) to intervention (flexible sigmoidoscopy screening) and control (not contacted for screening) (Rutter, et al., 2016). The primary outcomes of the UKFSS Trial were incidence and mortality of CRC, expressed via hazard ratios of intervention versus control participants. The results from this validation showed that all three CISNET models accurately predicted the relative reduction in CRC mortality 10 years after screening, and two of three models predicted hazard ratios for CRC incidence that were within the study-estimated 95% confidence intervals. This third model was recalibrated, so that all models yielded predictions that were within the study-estimated confidence intervals. In 2017, new results were published that contained outcomes from the UKFSS Trial after approximately 17 years of follow-up (Atkin, et al., 2017). Given the strength and uniqueness of information provided by this large trial to inform the long-term (Knudsen, et al., 2012) effectiveness of screening, it is important to verify that our models continue to validate well to the 17-year results. Comparison to 17-year results can also identify differences between models and study results, and so may help to improve the models. Since the previous validation, one of the models has been updated and recalibrated using baseline UKFSS results as targets, and the findings described here are the first published validation results from this revised model. Because two of the three models used results from the UKFSS Trial as calibration targets, this is only an external validation for the one, unchanged, model.

Methods

The three CISNET models evaluated are: Colorectal Cancer Simulated Population model for Incidence and Natural history (CRC-SPIN) (Rutter, Miglioretti, & Savarino, 2009), Simulation Model of Colorectal Cancer (SimCRC) (Knudsen, et al., 2012), and Microsimulation Screening Analysis (MISCAN) (van Hees, et al., 2014). All models make similar assumptions about the natural history of CRC in which individuals begin in a disease-free state and may transition to an adenoma state, a preclinical CRC state, and a clinically detected CRC state. Individuals may then die of CRC, or they may die from other causes at any time. All models assumed that after 2003 there are no improvements in stage-specific CRC survival. Detailed model descriptions are available online (CISNET Model Registry, Colorectal cancer models overview, https://resources.cisnet.cancer.gov/registry/packages/filter/Colorectal/).

The CRC-SPIN model has been updated and recalibrated since the prior validation (CRC-SPIN 2.0) (Rutter, Ozik, DeYoreo, & Collier, 2019); one of the calibration targets for the revised model included sex-specific rates for detection of cancer at screening from the UKFSS study. The SimCRC model remains unchanged since the previous validation paper was published (Rutter, et al., 2016), and the MISCAN model reported here is the revised MISCAN model included in the appendix of the previous validation paper, which was calibrated to 10-year UKFSS outcomes. Assumptions about the sensitivity of flexible sigmoidoscopy and colonoscopy, referral to adenoma surveillance, and survival after treatment were unchanged from the previous validation paper (Rutter, et al., 2016).

Model predictions are generated by simulating the UKFSS sample 2,000 times, matching the simulated sample to the study sample in terms of sample size, age, and gender. The models were calibrated to United States CRC incidence from 1975–1979 (National Cancer Institute, 2004), and no modifications were made to reflect the United Kingdom population incidence rates. We assume that the only screening that occurred for the screening group was a one-time flexible sigmoidoscopy, with those that screened positive being referred to colonoscopy with probability based on published results. We assume that the control group did not receive any screening.

We compare model predictions to observations from the UKFSS Trial. Our primary prediction targets are 17-year relative hazard ratios for CRC incidence and mortality in the screened group versus the control group. Secondary prediction targets are CRC incidence and mortality rates per 100,000 person-years. We define accurate model predictions (point estimates) as predictions that are within the 95% confidence intervals of UKFSS Trial estimates. This confidence level criterion is consistent with the 10-year validation. When validating to outcomes reported for the screened group, we are referring to the subset of the intervention group that was actually screened, e.g., those that were adherent.

Results

Effect of screening on CRC incidence and mortality

MISCAN and SimCRC accurately predicted the primary target of 17-year relative hazard ratios for CRC incidence in the screened group versus the control group based on per-protocol analyses (Figure 1 and Table 1). CRC-SPIN 2.0 predicted hazard ratios that were too low, but just outside of the 95% confidence intervals of the UKFSS Trial estimates. Looking more closely at CRC incidence by distal versus proximal location (distal was defined as the rectum and sigmoid colon), the hazard ratios for distal incidence were low for CRC-SPIN 2.0, and high for the other two models. The hazard ratios for proximal incidence were accurate for SimCRC, but too low for MISCAN and CRCSPIN 2.0. SimCRC accurately predicted hazard ratios for CRC-specific mortality, and the other two models predicted hazard ratios that were just below the 95% lower confidence limit from the UKFSS study (Figure 1 and Table 1).

Figure 1: Hazard ratios.

UKFSS Trial estimated and model-predicted hazard ratios for intervention effects. Point estimates as well as 95% confidence or credible intervals for model predictions and 95% confidence intervals for estimates are displayed. Both estimated and predicted hazard ratios compare screened intervention group participants to control participants who underwent no screening.

Table 1:

UKFSS Trial estimated and model-predicted hazard ratios, and 17-year CRC incidence and mortality per 100,000 persons screened based on per protocol analysis. All estimates are expressed with a point estimate and 95% credible or confidence intervals.

Outcome	Source	Hazard Ratio	17-year rate control group	17-year rate screened group
Incidence (All-site)
	CRC-SPIN2.0	.56 (.51, .61)	200 (193, 207)	114 (105, 123)
	MISCAN	.61 (.57, .66)	231 (223, 238)	144 (134, 154)
	SimCRC	.66 (.61, .72)	212 (204, 219)	143 (122, 153)
	UKFSS	.65 (.59, .71)	184 (178, 191)	120 (112, 128)
Incidence (Distal)
	CRC-SPIN2.0	.33 (.28, .38)	99 (94, 104)	32 (28, 37)
	MISCAN	.59 (.53, .65)	117 (112, 123)	69 (62, 75)
	SimCRC	.53 (.48, .58)	144 (138, 150)	76 (69, 82)
	UKFSS	.44 (.38, .50)	112 (107, 117)	50 (45, 56)
Incidence (Proximal)
	CRC-SPIN2.0	.81 (.73, .90)	101 (96, 105)	82 (74, 89)
	MISCAN	.67 (.60,.74)	113 (108, 118)	75 (68, 82)
	SimCRC	.99 (.88,1.12)	68 (64, 72)	67 (61, 74)
	UKFSS	.95 (.83, 1.09)	71 (67, 75)	66 (60, 73)
Mortality
	CRC-SPIN2.0	.47 (.40, .54)	70 (66, 74)	33 (29, 38)
	MISCAN	.49 (.41, .57)	54 (50, 57)	26 (22, 30)
	SimCRC	.60 (.52, .69)	73 (69, 77)	44 (39, 49)
	UKFSS	.59 (.49, .70)	56 (53, 59)	33 (29, 38)

Absolute Incidence and Mortality

All models over-predicted secondary targets of CRC incidence rates per 100,000 person-years in the control group. SimCRC and MISCAN also over-predicted CRC incidence in the screened group while CRC-SPIN 2.0 accurately predicted CRC incidence in the screened group (Table 1 and Figure 2).

Figure 2: Cumulative incidence.

UKFSS Trial estimated and model predicted colorectal cancer incidence over the 17 years after randomization (in 1997) in the screened and control group. The UKFSS study-estimated curves were plotted by extracting data from figure 1 in (Atkin, et al., 2017) using WebPlotDigitizer software (Rohatgi, 2019).

CRC-specific mortality rates in the control group were accurately predicted by the MISCAN model, but were too high for both SimCRC and CRC-SPIN 2.0 models. The CRC-specific mortality rate prediction in the screened group was too high for SimCRC, too low for MISCAN and accurate for CRC-SPIN 2.0 (Table 1 and Figure 3).

Figure 3: Cumulative Mortality.

UKFSS Trial estimated and model predicted colorectal cancer mortality over the 17 years after randomization (in 1997) in the screened and control groups. The UKFSS study-estimated curves were plotted by extracting data from figure 1 in (Atkin, et al., 2017) using WebPlotDigitizer software (Rohatgi, 2019).

In the control group, about 61% of UKFSS Trial CRCs were distal (Table 1). The MISCAN and CRC-SPIN 2.0 models underestimated the proportion of cancer cases that are distal (both predicted around 50%), and the SimCRC model over-estimated this proportion (68%).

Disease detection at screening

As described in previous validation results (Rutter, et al., 2016), the recalibrated MISCAN model over-predicted prevalence of detected adenomas at flexible sigmoidoscopy and referral rates to colonoscopy, while both SimCRC and CRC-SPIN 1.0 under-predicted prevalence of detected adenomas at flexible sigmoidoscopy and referral rates to colonoscopy. While these are not long-term outcomes, we report them here in order to compare the CRC-SPIN 2.0 model to the others. CRC-SPIN 2.0 over-predicted detected adenoma prevalence at screening and referral rates to colonoscopy. All models over-predicted prevalence of adenomas detected at colonoscopy. MISCAN and SimCRC predicted screen-detected cancer rates that were too high, CRC-SPIN 1.0 predicted too low, and CRCSPIN 2.0 predicted accurately (as expected, because the model was calibrated to these screen-detection rates). Relative to CRC-SPIN 1.0, CRC-SPIN 2.0 improved the prediction of prevalence of adenomas detected at flexible sigmoidoscopy and referral to colonoscopy, but predictions for prevalence of adenomas detected at follow-up colonoscopy worsened (Table 2).

Table 2:

Predicted percentage of patients with adenomas detected, referred to colonoscopy, and with cancer detected at screening via flexible sigmoidoscopy or colonoscopy. All estimates are expressed with a point estimate and 95% credible or confidence intervals.

Outcome	Source	Percent
Adenomas Detected at Flexible Sigmoidoscopy (‘distal’ location) (Percentage of Patients)
	CRC-SPIN2.0	14.4 (14.0, 14.7)
	CRC-SPIN1.0	9.4 (9.2, 9.7)
	MISCAN	27.7 (27.2, 28.2)
	SimCRC	8.8 (8.5, 9.0)
	UKFSS	12.1 (11.8, 12.4)
Referred to Colonoscopy (Percentage of Patients)
	CRC-SPIN2.0	6.5 (6.3, 6.8)
	CRC-SPIN1.0	3.6 (3.4, 3.8)
	MISCAN	13.6 (13.3, 14.0)
	SimCRC	4.0 (3.8, 4.2)
	UKFSS	5.2 (4.3, 6.2)
Among Patients with Colonoscopy, the Percentage with Adenomas Detected*
	CRC-SPIN2.0	59.2 (57.3, 61.2)
	CRC-SPIN1.0	39.1 (36.7, 41.7)
	MISCAN	73.0 (71.8, 74.1)
	SimCRC	35.6 (33.3, 37.9)
	UKFSS	18.8 (17.1, 20.5)
Percentage of Patients with Screen Detected CRC
	CRC-SPIN2.0	0.34 (0.28, 0.40)
	CRC-SPIN1.0	0.11 (0.08, 0.15)
	MISCAN	0.45 (0.39, 0.52)
	SimCRC	0.44 (0.38, 0.51)
	UKFSS	0.34 (0.29, 0.40)

^*These adenomas have a proximal location

Discussion

Microsimulation models are often used to predict long-term or even lifetime outcomes to guide policy decisions and recommendations regarding cancer screening. The UKFSS Trial is unique because it examines a one-time CRC screen that was implemented before a screening program was established, and contains long-term study outcomes, now a full 17 years following randomization. The outcomes from this trial thus provide a unique opportunity to validate the three CISNET CRC models to long-term study outcomes. We found that all three models predicted primary targets of relative incidence and CRC mortality hazard ratios in the screened group versus control group that were reasonably close to the UKFSS results. This provides us with additional information about and confidence in model-predicted benefits of screening to reduce CRC incidence and mortality.

One caveat is that this is only truly an external validation for the SimCRC model, which did not use any UKFSS information in model development. The CRC-SPIN model incorporated information from baseline study results, and the MISCAN model incorporated information from 10-year study outcomes. SimCRC predicted 17-year incidence and mortality hazard ratios that were closer to the UKFSS estimates than the 10-year predictions. Interestingly, the SimCRC model predictions of the relative effect of screening were nearer to observed hazard ratios than the other two models. In addition, model predictions of secondary targets, including absolute rates, often differed from one another and in some cases were consistently high or low as compared to the UKFSS targets.

There are various reasons why we might anticipate differences between the UKFSS Trial results and model predictions. Model assumptions, which are inputs that are not calibrated, directly affect model predictions. For example, differences in predicted incidence and mortality rates may be due to differences between the population used for calibration (simulated to be like the US population) and the UKFSS study population (the UK population). Assumptions about screening uptake in the control group would also affect model-predicted screening benefits (Etzioni, et al., 2013). All three models assumed no screening among control participants. Uptake of screening in the control group would reduce the control group’s CRC incidence and mortality rates. All three models predicted incidence in the control group that was higher than observed. The SimCRC and CRC-SPIN models also predicted higher than observed mortality in the control group, while the MISCAN model accurately predicted mortality in the control group. The divergence between predicted and observed outcomes in the control group drove the CRC-SPIN model’s over-prediction of screening effects.

Assumptions about test accuracy are another key model component. The three models made common assumptions about the accuracy of flexible sigmoidoscopy and colonoscopy, based on the accuracy of colonoscopy estimated from back-to-back examinations (Lin, et al., 2016) (Lin, Perdue, Henrikson, Bean, & Blasi, 2020). These studies will overestimate the accuracy of colonoscopy if hard to detect lesions (e.g., small or sessile lesions) are missed by both exams, with overestimation that is worse when sensitivity is low. Sensitivity of flexible sigmoidoscopy is assumed to be the same as colonoscopy within reach of the endoscope. If the accuracy of flexible sigmoidoscopy and colonoscopy within the UKFSS Trial was lower than assumed, then models would predict detection of too many adenomas among the screened group, resulting in over-estimation of screening benefit relative to the trial. We found that all models over-predicted the percentage of patients with adenomas detected at colonoscopy. Methods for obtaining unbiased estimates of test sensitivity from these studies would allow modeling assumptions regarding accuracy to be modified and improved.

Assumptions about screening in the control group and endoscopic accuracy are expected to result in over-prediction of screening benefit. In contrast, assumptions about survival after detection are expected to result in under-prediction of the mortality benefit. Models assumed that stage-specific survival did not improve for people with CRC detected after 2003. While changes in staging algorithms over time make updating these survival inputs challenging, and there is insufficient data to update these survival inputs, there is evidence for continued stage-specific improvements in survival (Van Abbema, 2019), even among patients with stage IV disease (Hu, 2015).

In spite of the challenges of validating models, these findings help to highlight differences between these three CISNET models and suggest changes that may be warranted to the natural history models, the calibration data used by the models, and underlying model assumptions. There was wide variation in model-predicted location-specific hazard ratios. While these were not primary validation targets, this divergence suggests that models may be improved by developing location-specific natural history processes and targets. Such modifications could be especially important if models were used to evaluate a screening test with accuracy that varies systematically for lesions in the proximal versus distal locations.

This work highlights the need for high-quality screening results to inform model parameters, so that the models can more accurately predict outcomes when trial data are unavailable, and demonstrates how collaborative modeling that considers results from three independent models can be used to provide a range of possible screening outcomes and predict long-term outcomes when information used to inform the models is incomplete and trial data are not yet available. Comparative modeling and model validation is a general approach that is applicable to other cancer screening programs and more generally to modeling of other disease process. This approach can – and when possible should - be used whenever there exist multiple models describing the same phenomenon as it is a method for deepening our understanding of model differences and determining when predictions made with models are expected to be valid.