Simulation Modeling of Cancer Clinical Trials: Application to Omitting Radiotherapy in Low-risk Breast Cancer

Abstract

Background

We used two models to simulate a proposed noninferiority trial of radiotherapy (RT) omission in low-risk invasive breast cancer to illustrate how modeling could be used to predict the trial’s outcomes, inform trial design, and contribute to practice debates.

Methods

The proposed trial was a prospective randomized trial of no-RT vs RT in women age 40 to 74 years undergoing lumpectomy and endocrine therapy for hormone receptor–positive, human epidermal growth factor receptor 2–negative, stage I breast cancer with an Oncotype DX score of 18 or lower. The primary endpoint was recurrence-free interval (RFI), including locoregional recurrence, distant recurrence, and breast cancer death. Noninferiority required the two-sided 90% confidence interval of the RFI hazard ratio (HR) for no-RT vs RT to be entirely below 1.7. Model inputs included published data. The trial was simulated 1000 times, and results were summarized as percent concluding noninferiority and mean (standard deviation) of hazard ratios for Model GE and Model M, respectively.

Results

Noninferiority was demonstrated in 18.0% and 3.7% for the two models. The respective means (SD) of the RFI hazard ratios were 1.8 (0.7) and 2.4 (0.9); most were locoregional recurrences. The mean five-year RFI rates for no-RT vs RT (SD) were 92.7% (2.9%) vs 95.5% (2.2%) and 88.4% (2.0%) vs 94.5% (1.6%). Both models showed little or no difference in breast cancer–specific or overall survival. Alternative definitions of low risk based on combinations of age and grade produced similar results.

Conclusions

The proposed trial was unlikely to show noninferiority of omitting radiotherapy even using alternative definitions of low-risk, as the endpoint included local recurrence. Future trials regarding radiotherapy should address absolute reduction in recurrence and impact of type of recurrence on the patient.

Randomized clinical trials (RCTs) are the gold standard for understanding treatment effects and developing clinical guidelines (1,2). Simulation modeling can help evaluate designs of RCTs by synthesizing available evidence and predicting the trial’s outcome, including quantifying uncertainties in the outcome. The concept of simulating clinical trials was developed in part by clinical pharmacologists (3). Simulation is essential when designing complicated innovative trials (4). Simulations are seldom used for designing traditional cancer clinical trials, whether sponsored by the National Cancer Institute or industry. But these trials can be simulated while the trial is running (5) or being developed.

Traditional RCTs are usually designed with fixed type I error rate and statistical power, each of which requires assuming a particular value of the relevant parameter. Although its true value is unknown, there is usually some information available about this parameter, possibly from patient populations related to but different from that in the trial. We demonstrate how this information can be used to statistically model and simulate a clinical trial that has a particular design. One simulation of the model provides an outcome of the trial. Multiple simulations provide multiple different outcomes and enable predicting the trial’s result as a probability distribution. The variability in this distribution depends on sampling variability in the trial but also the uncertainty in the information about the parameter in question.

We illustrate our methods in the context of a particular trial. We developed the model and carried out simulations using two Cancer Intervention and Surveillance Modeling Network (CISNET) models (GE and M) (6–11). The specifics of the proposed trial and our predictions of its outcome demonstrate our methods and their utility in cancer research. We chose a trial that has been proposed in breast cancer for its clinical relevance. Previous studies have shown that patients with tumors having low Oncotype DX scores who receive radiotherapy have low rates of locoregional and distant recurrence (12,13). The proposed trial was designed to address whether patients who have low-risk invasive breast cancer with Oncotype scores of 18 or lower can avoid radiotherapy. Describing the results of our simulations as regards the substantial clinical question posed in this trial is the second major goal of our study. Our results have implications for designs of future clinical trials in low-risk breast cancer.

Methods

Proposed Clinical Trial

NRG Oncology proposed a clinical trial to evaluate omitting adjuvant radiotherapy in low-risk invasive breast cancer. Given the size of the proposed trial and concerns regarding feasibility and cost, CISNET modelers undertook simulation to predict its outcome based on the available evidence and to guide trial design revisions.

Eligible women were age 40 to 74 years with newly diagnosed, node-negative invasive breast cancer that was 2 cm or smaller, hormone receptor–positive (either estrogen receptor [ER]– or progesterone receptor [PR]–positive), human epidermal growth factor receptor 2 (HER2)–negative, with Oncotype scores of 18 or lower. Planned therapy was breast-conserving surgery, hormonal therapy, and no chemotherapy. The trial’s primary endpoint was recurrence-free interval (RFI), the time from random assignment to any invasive recurrence or breast cancer death (14,15). As the data sources used to develop the models did not report ductal carconima in situ (DCIS) recurrences, we excluded DCIS from the primary endpoint. Noninferiority would be concluded if the two-sided 90% confidence interval of the RFI hazard ratio (no-RT vs RT) was less than 1.7. Having a 5% type I error rate and 80% power required 88 events. The design assumed proportional hazards with 0.61% annual RFI hazard for RT and equal random assignment to RT:no-RT. The sample size of 2194 patients would be accrued at 40/month with 4.4 years of additional follow-up, for a total trial duration of nine years. Secondary endpoints included locoregional RFI, distant RFI, and breast cancer–specific and overall survival.

Model Overview

CISNET Models GE and M (10,11) were adapted to simulate the proposed trial (Supplementary Methods, available online). The models used somewhat different approaches and data inputs (Table 1). In both models, virtual patients were generated randomly based on eligibility criteria and were randomly assigned to RT or no-RT. Recurrence and time to recurrence were generated based on model inputs and assumptions of the scenario under consideration.

Table 1.

Structure and approach for trial simulation of radiotherapy omission vs radiotherapy in low-risk breast cancer patients by model

Model specifications	Model GE	Model M
Primary endpoint	RFI	RFI
Model structure*	Empirical Bayesian	Fully Bayesian
Analytic approach†	Monte Carlo sampling based on probabilistic sensitivity analysis	Monte Carlo sampling based on Bayesian posterior distributions
Embedded within respective CISNET model	No‡	Yes§
Distinguishes locoregional from distant recurrences within RFI	Yes	No
Simulates mortality separately from RFI‖	Yes	Yes
Hormone receptors considered	ER and PR	ER only
Conducts subset analyses based on RFI	Yes	No
Conducts sensitivity analyses of different eligibility criteria¶	No	Yes

^*In an empirical Bayesian model, the prior distribution is derived from a dataset. CISNET = Cancer Intervention and Surveillance Modeling Network; ER = estrogen receptor; PR = progesterone receptor; RFI = recurrence-free interval.

^†Model GE distributions were “shrunk” by using a hyperprior distribution that gives most of its support to hazard ratios between 0.25 and 4.00.

^‡Model GE created code separate from its CISNET model for ease of trial simulation.

^§Model M adapted its CISNET model (11) by randomly assigning women who were diagnosed with cancers and were eligible for the trial. Model M’s components related to method of detection are not considered in the trial simulation.

^‖Both models perform separate simulations for recurrence-free interval and breast cancer and overall mortality as there were insufficient data in any source(s) to quantify the joint relationship of time of locoregional recurrence, distant recurrence, and breast cancer death.

^¶Model M simulated RFI assuming alternative trial eligibility, such as age 50 to 74 years instead of 40 to 74 years and low-grade cancers instead of Oncotype DX score of 18 or lower.

Estimation of Model Input Distributions/Parameters

The principal sources of data and radiotherapy efficacy were meta-analyses of multiple RCTs and US population cancer registries. Model M derived RT benefits primarily from statistical summaries of seven RCTs in the Oxford Overview categorized as “evaluating the need for RT after lumpectomy in low-risk patients”; these RCTs are listed in Supplementary Table 1 (available online) using the trial labels from the Overview (16). Although the Overview did not consider local recurrence separately, the original investigators of all these RCTs regarded local recurrence to be an important endpoint, and they published the comparisons of local recurrence rates for RT and no-RT. Model GE used individual patient-level data from a subset of the RCTs in the Overview (17–20).

Patients’ Oncotype scores were generated based on joint distributions of tumor characteristics and age for patients matching trial eligibility from Surveillance, Epidemiology, and End Results data linked to Oncotype test results provided by Genomic Health, Inc. (SEER-GHI) (21,22). We used a left-truncated survival model estimated from SEER for patients approximately matching trial eligibility, excluding Oncotype scores higher than 18. This allowed for incorporating competing non–breast cancer mortality in the simulations (23). Specific data used varied based on model structure (Table 2). The Georgetown University Oncology Institutional Review Board approved this research.

Table 2.

Data used to develop distributions of values for modeling a trial simulation of radiotherapy omission vs radiotherapy in low-risk breast cancer patients by model

Parameter	Model GE		Model M
	Data used	Source	Data used	Source
Incident cases	Pooled trials*	(17–20,24–26)	SEER and SEER-GHI	(21)
Competing mortality	Non–breast cancer death among trial-eligible groups in the SEER population	(23)	Non–breast cancer death among trial-eligible groups in the SEER population	(23)
Tumor size†	Pooled trials, continuous	(17–20,24–26)	N/A	N/A
Tumor grade†	SEER-GHI	(21,22)	SEER-GHI	(21)
ER and PR†	SEER-GHI	(21,22)	N/A	N/A
Oncotype DX	Assigned from SEER-GHI based on age, tumor size, ER/PR, and grade	(21,22)	Oncotype DX score assigned based on observed distribution in SEER-GHI for age 40–74 y, stage I patients, tumor size ≤2 cm, ER+, HER2-, undergoing breast-conserving surgery	(21)
Event times/Oncotype DX	NSABP B14 and B20 by Oncotype DX‡	Personal comm., S. Shak	Given expected distribution of grade within Oncotype DX groups based on SEER-GHI, Oxford data used to define the effects of grade (and age) on events	(16) (Webtables 5b, 3a, 3b, 3c, 6 and Figure 5) and (21)
Event times by age, grade, size, hormone receptors	Pooled trials	(17–20,24–26)	SEER, Oxford Overview for low-risk and pooled trial data§	(16) (Webtables 5b, 3a, 3b, 3c, 6 and Figure 5)
RT efficacy	Pooled trials	(10–13)	Oxford Overview for low-risk and pooled trial data	(16) (Webtables 5b, 3a, 3b, 3c, 6 and Figure 5)

^*“Pooled trials” refers to a de-identified individual-level dataset from seven clinical trials. Four of the trials were randomized trials of radiotherapy (17–20), and the other three were trials of systemic adjuvant therapy and included radiotherapy data from patients not receiving chemotherapy (24–26). ER = estrogen receptor; HER2 = human epidermal growth factor 2; NSABP = National Surgical Adjuvant Breast and Bowel Project; N/A = Not Applicable; PR = progesterone receptor; RT = radiotherapy; SEER = Surveillance, Epidemiology, and End Results (SEER) data; SEER-GHI = Surveillance, Epidemiology, and End Results data linked to Oncotype DX test results provided by Genomic Health, Inc.

^†In Model GE, the distributions of age, grade, ER, PR, and tumor size were each derived in a regression model conditional on the other variables.

^‡Data on recurrence and death events by the joint distribution of Oncotype DX scores and other patient eligibility characteristics were not available from published sources (12,28). Therefore, Genomic Health, Inc., provided an honest broker, de-identified dataset with individual-level, de-identified data under a data use agreement for this project (S. Shak, personal communication, 2016).

^§Based on differences in events over follow-up time in the Overview (16) data, a piecewise exponential distribution with two constant hazard rates was used to derive prior distributions for first events by radiotherapy (assuming a beta distribution)—one for the first five years and another for the time thereafter.

Model GE

Model GE relied primarily on the pooled clinical trial dataset for inputs (“pooled dataset”) (17–20,24–26). This dataset was used to develop the joint distribution of characteristics of trial-eligible patients (eg, age, grade).

We derived distributions of times to recurrence and death given patient characteristics other than Oncotype from competing risk models in the pooled dataset (27). Radiation effects from randomized and nonrandomized trials in the pooled data set were separately estimated, and only the randomized effects were used in the simulations. De-identified data linking Oncotype scores to data from two NSABP trials (24,25) were used to estimate the effects of the Oncotype scores on time to events. However, this dataset contained no information about surgery or radiotherapy (28). These models provided a distribution of plausible effect estimates given patient and clinical characteristics. Each of the 1000 trial replicates was then randomly assigned its own set of effect estimates from that distribution. Within each trial simulation, each patient was randomly assigned whether to experience an event or not and the time of any event based on that distribution.

Model M

Model M simulations were embedded in its CISNET model (10,11). In each simulation, we tracked the US population over time, as represented by SEER. Women who developed breast cancer were assessed for eligibility in the proposed trial. For simulating the trial itself, Model M followed a similar approach to Model GE. In particular, joint distributions of Oncotype score with clinical characteristics were derived from SEER-GHI (21,29). Taking a Bayesian approach, Model M used published results from the Oxford Overview (16) to estimate prior distributions required to simulate the trial. The Overview does not distinguish between local, regional, and distance recurrences. Therefore, neither did Model M.

The distribution of effects of radiotherapy on RFI was derived from the Overview by age and grade, conditional on tumor size of 2 cm or smaller, ER-positive, tamoxifen-treated, and undergoing breast-conserving surgery (16). Data from both the Overview (16) and the pooled dataset were used to derive prior distributions for breast cancer and overall survival.

Analyses

The hazard ratio of no-RT vs RT and its two-sided 90% confidence interval were determined for each of the 1000 simulations using Cox proportional hazards regression. The predictive probability that the proposed trial would show noninferiority was estimated by the proportion of the 1000 simulations for which the upper bound of the confidence interval was less than 1.7. RFI by treatment group for each of the 1000 simulations was found from Kaplan-Meier curves. We provide the means of these curves for each treatment group at years 5 and 9.

While the trial was not designed to assess subgroup effects, we explored results categorized by clinically relevant patient subgroups: Oncotype scores 0–10 and 11–18 and age younger than 60 years and 60 years and older. For sensitivity analyses, the models evaluated how results would vary if the proposed noninferiority hazard ratio threshold was increased from 1.7 to as high as 2.0. Additionally, Model M simulated two alternative scenarios of low risk not specified in the proposed trial: low-grade tumors for ages 50 to 74 years and that same group but also including intermediate grade for ages 70 to 74 years.

Results are presented as means and standard deviations of the endpoint distributions for Model GE and Model M, respectively. The estimates are based on simulations, and their accuracy for any given model assumptions depends on the number of simulations. Simulation variability (±2 standard errors) for proportions based on 1000 iterations is never greater than ±3.2%, and for proportions above 95%, it is always less than ±1.4%.

Results

The characteristics of simulated patients were similar for the two models, except that Model M had fewer patients with low-grade tumors (Table 3). The mean trial duration was longer in Model GE than in Model M (Table 3; Supplementary Figure 1, available online). Model GE considered site of recurrence and found that most first recurrences were locoregional (Supplementary Figures 2 and 3, available online): 86.0% vs 62.0% in the no-RT vs RT arms.

Table 3.

Summary of mean patient and trial characteristics of simulations of a proposed trial of omission of breast radiotherapy vs radiotherapy in low-risk breast cancer^*

Characteristics	Model GE	Model M
Patient characteristics
Mean age, y	59.5	59.4
Mean tumor size, cm	1.0	N/A
Tumor grade, %
Low	45.4	39.2
Intermediate	47.3	53.7
High	7.3	7.1
Mean Oncotype DX score	12.6	12.2
Trial characteristics
No. enrolled (SD)	2174 (77)	2127 (97)
Range	1560–2194	1640–2194
Mean trial length (SD), y	6.3 (1.4)	4.6 (0.4)
Range, y	3.2–9.0	3.4–5.9
Mean follow-up (SD), y	3.7 (1.2)	2.4 (0.3)
Range, y	1.6–6.2	1.7–3.6
Mean No. (SD) of first events	87 (6)	88 (0.14)
Range	17–88	87–88
Type of first event (SD), %
Locoregional	76.9 (17.8)	N/A†
Distant	23.1 (17.8)	N/A
Breast cancer death	N/A	N/A
Vital status at 5 y (range), %
Alive	96.4 (87.1–99.3)	97.0 (94.9–98.5)
Died of breast cancer	1.7 (0–10.3)	2.0 (0.9–3.4)
Died of other causes	1.9 (0.4–3.5)	1.0 (0.3–1.9)

^*Results for 1000 trial replicates. Low-risk is defined in the proposed trial as estrogen receptor–positive and/or progesterone receptor–positive, human epidermal growth factor receptor 2–negative, lymph node–negative breast cancer with pathologic tumor size of 2 cm or smaller, Oncotype DX scores of 18 or lower, for whom hormonal therapy following breast-conserving surgery was planned, but not adjuvant chemotherapy.

^†Model M did not separate locoregional from distant recurrences as the Oxford Overview (16) did not present this information.

The mean RFI hazard ratios (SD) for no-RT vs RT were 1.8 (0.7) and 2.4 (0.9) (Table 4 and Figure 1A) for Models GE and M, respectively. Only 18.0% of the simulations in Model GE and only 3.7% of those in Model M concluded that omitting radiotherapy was noninferior (Table 4 and Figure 1B). The mean five-year RFI rates across the 1000 simulations for no-RT vs RT (SD) were 92.7 (2.9)% vs 95.5 (2.2)% in Model GE and 88.4 (2.0)% vs 94.5 (1.6)% in Model M (absolute differences = 2.8 [2.3]%, and 6.1 [2.6]%, respectively) (Table 4). For Model GE, the mean hazard ratio for any first locoregional recurrence was 2.8 (1.6), and the corresponding five-year RFIs (SD) were 93.7 (3.0)% vs 97.1 (2.0)% for no-RT vs RT (absolute difference (SD) = 3.4 [2.0]%) (Table 4). There was no increased hazard (mean HR = 0.8, SD = 0.7) for distant RFI as first event.

Table 4.

Simulated five-year follow-up results for omission of breast radiotherapy vs radiotherapy in low-risk invasive breast cancer patients^*

Primary and secondary endpoints	Mean hazard ratio (SD)	Mean 5-y rate		Absolute difference (SD), %
		No radiotherapy (SD), %	Radiotherapy (SD), %
Recurrence-free interval†
Model GE	1.8 (0.7)	92.7 (2.9)	95.5 (2.2)	2.8 (2.3)
Model M	2.4 (0.9)	88.4 (2.0)	94.5 (1.6)	6.1 (2.6)
Recurrence-free interval—loco-regional recurrences as the first event
Model GE	2.8 (1.6)	93.7 (3.0)	97.1 (2.0)	3.4 (2.0)
Recurrence-free interval—distant recurrences as the first event
Model GE	0.8 (0.7)‡	99.0 (0.9)	98.4 (1.5)	–0.6 (1.1)
Breast cancer–specific survival
Model GE	1.0 (0.5)§	97.8 (1.6)	97.4 (2.0)	–0.4 (0.9)
Model M	1.2 (0.4)	95.7 (0.9)	96.2 (0.9)	0.5 (1.3)
Overall survival
Model GE	1.1 (0.4)	97.2 (0.7)	97.2 (0.7)	0 (0.8)
Model M	1.1 (0.2)	93.6 (1.1)	94.1 (1.1)	0.5 (1.6)

^*Low risk was defined in the proposed trial as patients with estrogen receptor–positive and/or progesterone receptor–positive, human epidermal growth factor receptor 2–negative, lymph node–negative invasive breast cancers with pathologic tumor size of 2 cm or smaller, Oncotype DX scores of 18 or lower, who were given hormonal therapy following breast-conserving surgery but no adjuvant chemotherapy. The use of an Oncotype DX score cut-point of 18 or lower was based on the proposed trial specifications.

^†The corresponding median hazard ratios for the recurrence-free interval for no-radiotherapy (RT) vs RT across 1000 replications were 1.7 and 2.3 for Model GE and Model M, respectively. The median hazard ratio for loco-regional recurrence at first event was 2.5 (Model GE only); breast cancer–specific survival rates were 0.8 (GE) and 1.1 (M); and overall survival rates were 1.0 (GE) and 1.1 (M).

^‡This result is based on 959 trial replicates. Replicates in which no distant recurrences occurred in the no-radiotherapy group were excluded.

^§This result is based on 995 trial replicates. Five replicates in which no breast cancer deaths occurred in the no-radiotherapy group were excluded.

Figure 1.

Distributions of hazard ratios and upper limits of the two-sided 90% confidence intervals for recurrence-free interval by model. A) Distribution of hazard ratios by model for recurrence-free interval for omission of radiotherapy vs radiotherapy in 1000 trial replications. Results are for patients age 40 to 74 years. Low-risk was defined in the proposed trial as estrogen receptor–positive (ER+) and/or progesterone receptor–positive (PR+), human epidermal growth factor 2 (HER2)–negative, lymph node–negative invasive breast cancers with pathologic tumor size less than or equal to 2 cm, Oncotype DX scores of 18 or lower, who were given hormonal therapy following breast conserving surgery but no adjuvant chemotherapy. The use of an Oncotype DX score cut-point of 18 or lower was based on the proposed trial specifications. B) Distribution of the upper limits of the two-sided 90% confidence intervals of the hazard ratios for recurrence-free interval for omission of radiotherapy vs radiotherapy in 1000 trial replications by model. The histograms illustrate the distribution of the upper limits of a two-sided 90% confidence interval of the hazard ratios for recurrence-free interval by model across 1000 simulated clinical trials evaluating noninferiority of omission of breast radiotherapy among low-risk invasive breast cancer patients. The proposed trial specified that the null hypothesis of inferiority would be rejected if the upper limit of the 90% confidence interval were less than 1.7. The gray line indicates an upper limit of the 90% confidence interval of 1.7. HR = hazard ratio; RT = radiotherapy.

There was little or no difference in either model between arms for breast cancer–specific survival or overall survival (Table 4). Estimates of change in breast cancer survival by omitting radiotherapy are within simulation variability of the null hypothesis for both models (Table 4).

The mean RFI hazard ratios for treatment arms were similar across Oncotype and age subgroups. However, RFI rates at five years for no-RT and RT were lower among patients age 60 years or older vs younger than age 60 years, as were differences in rates (Model GE: mean [SD] = 1.5 [1.9]%, vs 4.2 [3.4]%; Model M: mean [SD] = 5.1 [2.3]% vs 7.4 [4.0]%) (Table 5). For higher noninferiority thresholds of HR, the proportion of trials concluding noninferiority increases (Table 6). However, even at a noninferiority threshold of 2.0, only 30.4% (Model GE) and 8.7% (Model M) of trials showed noninferiority (Table 6). When eligibility was based on low risk defined by tumor grade instead of Oncotype groups or age groups, the mean RFI hazard ratios were somewhat lower than that of the original analysis (Table 7, Model M).

Table 5.

Scenario analyses: RFI rates for 1000 simulations of a proposed trial of omission of breast radiotherapy vs radiotherapy in low-risk^* breast cancer for age and Oncotype DX subgroups, and nine years of follow-up

Primary endpoint	Mean recurrence-free interval rate†		Absolute difference (SD), %
	No radiotherapy (SD), %	Radiotherapy (SD), %
Age
5-y RFI- age <60 y
Model GE	89.6 (4.4)	93.8 (3.4)	4.2 (3.4)
Model M	86.2 (3.2)	93.6 (2.5)	7.4 (4.0)
5-y RFI- age 60+ y
Model GE	95.5 (1.9)	97.1 (1.6)	1.5 (1.9)
Model M	90.1 (1.9)	95.2 (1.5)	5.1 (2.3)
Oncotype DX recurrence score group
5-y RFI- RS <11
Model GE	93.2 (3.1)	95.8 (2.4)	2.6 (2.9)
Model M	88.5 (1.9)	94.5 (1.5)	6.0 (2.4)
5-y RFI- RS 11–18
Model GE	92.5 (3.1)	95.4 (2.3)	2.9 (2.5)
Model M	88.3 (2.0)	94.4 (1.7)	6.2 (2.6)
Follow-up time
9-y RFI†
Model GE	87.4 (4.9)	92.1 (3.8)	4.7 (3.8)
Model M	80.4 (3.2)	90.4 (2.7)	10.2 (4.3)

^*Low risk was defined in the proposed trial as patients with estrogen receptor–positive and/or progesterone receptor–positive, human epidermal growth factor receptor 2–negative, lymph node–negative invasive breast cancers with pathologic tumor size of 2 cm or smaller, Oncotype DX scores of 18 or lower, who were given hormonal therapy following breast-conserving surgery but no adjuvant chemotherapy. The use of an Oncotype score cut-point of 18 or lower was based on the proposed trial specifications. RFI = recurrence-free interval; RT = radiotherapy.

^†RFI rate at nine years was derived as (1-r)⁹, where r is the annual recurrence rate.

Table 6.

Percentage of trials showing noninferiority out of 1000 simulations of a proposed trial of omission of breast radiotherapy vs radiotherapy in low-risk breast cancer at alternative noninferiority margins

Alternative noninferiority margins	Percentage of trials
1.8 hazard ratio for RFI—no-RT vs RT*
Model GE	22.2
Model M	5.3
1.9 hazard ratio for RFI—no-RT vs RT
Model GE	26.1
Model M	6.5
2.0 hazard ratio for RFI—no-RT vs RT
Model GE	30.4
Model M	8.7

^*RFI = recurrence-free interval; RT = radiotherapy.

Table 7.

Simulated five-year follow-up results for omission of radiotherapy vs radiotherapy in alternative risk groups defined by age and grade, model M only

Risk category	Mean hazard ratio (SD)	Mean 5-y RFI*		Absolute difference (SD), %
		No RT (SD), %	RT (SD), %
Age 50–74 y/low grade	2.0 (1.0)	94.0 (1.5)	96.5 (1.3)	2.5 (1.9)
Age 50–74 y/low grade or age 70–74 y/intermediate grade	2.1 (0.9)	92.9 (1.6)	96.1 (1.2)	3.2 (2.0)

^*RFI = recurrence-free interval; RT = radiotherapy.

Discussion

Clinical trial modeling and simulation cannot be a substitute for randomized controlled trials. RCTs are crude but reliable instruments. But RCTs are enormous consumers of time and resources. Modeling and simulation can make an RCT more efficient, and perhaps even demonstrate that running it is unnecessary.

There are two sources of uncertainty in predicting a future clinical trial. One is the widely understood sampling variability for any given parameter value. The other type of uncertainty is more important but less well understood. The parameters are themselves unknown. Information about them is generally available from earlier trials in other populations or with other therapies. Moreover, even if the future trial is identical to a previous trial, the parameters governing it may be different. This additional uncertainty should be factored into the model input process.

Our study applied modeling and simulation to predict the results of a proposed clinical trial. The clinical question was whether radiotherapy can be omitted in patients with low-risk invasive breast cancer. We reported results from two independent models. Both concluded that the trial would be unlikely to demonstrate noninferiority and that omitting radiotherapy would increase the rate of recurrence even in low-risk patients for every definition of risk we considered. The two models concluded different estimates of the probability that omitting RT would be noninferior (18.0% and 3.7% for Models GE and M, respectively). The difference in conclusions reflects the differences in model inputs. Model M used RT benefits from RCTs published in the Oxford Overview (16), while Model GE used individual patient-level data from a subset of those RCTs (17–20). The Overview considered 17 trials in which RT was randomized. They categorized seven of these as “evaluating the need for RT after lumpectomy in low-risk patients” (Supplementary Table 1, available online). Model M used the “low-risk” results summarized in the Overview by patient age, tumor size and grade, ER status, and whether assigned tamoxifen (16).

The first four trials (17–20) in Supplementary Table 1 (available online) were included in our pooled analysis and were considered by both models, with Model M incorporating the results via the statistical summaries in the Overview. The last three trials (30–32) in the table were considered by Model M (via the Overview) but not by Model GE. These latter trials included patients with lower risk on average than the first four trials. However, the lower-risk trials evinced an even greater relative reduction in rate of local recurrence for RT. Therefore, defining a population at lower risk may actually increase the relative benefit of RT.

The dominant role played by local recurrence in RFI and the greater relative benefit of RT in the lowest risk trials in the Overview (30–32) help explain the greater relative benefit of RT concluded by Model M. More importantly, it suggests that when using a relative measure of RT efficacy such as hazard ratio, there is no identifiable level of risk below which omitting RT is noninferior.

Clinical trials assessing the omission of RT in low-risk patients should use an absolute measure of efficacy. With such an approach, when risk is sufficiently low, random assignment may not be necessary. A case in point is the low-risk cohort of TAILORx, which showed that omitting chemotherapy for patients with an Oncotype score of 10 or lower achieved a five-year RFI of 98.7% (95% CI = 97.9% to 99.2%) (26). Having a chemotherapy arm, presumably with five-year RFI between 98.0% and 100.0% would have been irrelevant. At least four ongoing single-arm trials are assessing outcomes for omitting radiotherapy in low-risk breast cancer (33–36).

There are several limitations that should be considered in evaluating our results. First, we used evidence provided by meta-analyses. Their value in modeling is limited by any gaps in evidence from the trials included. One gap is the effect of RT by Oncotype. Another limitation in breast cancer trials is that the disease is dynamic because of improvements in therapy and use of screening mammography. Prognosis depends greatly on method of detection in addition to factors such as tumor size, nodal status, tumor grade, and hormone receptor status (37–40). No RCT has evaluated RT depending on method of detection.

Another evidence gap relates to our methods for deriving estimates of breast cancer–specific mortality. We intended to model the course of disease from type or types of first recurrence to second recurrence (if any) to death. This turned out to be impossible. The Overview does not provide the joint relationship between first recurrence and death, and our pooled analysis contained little information about deaths following first recurrence.

An additional limitation is that HER2 testing was not usually done in the trials considered. Perhaps most importantly, we did not consider excess mortality attributable to radiotherapy, which is a special concern for high-risk individuals such as smokers (41).

Our study highlights several issues. One is the clinical and design implication of use of relative vs absolute risk. A related issue is the appropriate primary endpoint in clinical trials of RT in patients with low-risk breast cancer. RFI counts local, regional, and distant recurrences equally. But these events are not comparable in terms of patient management or impact on survival and quality of life. The benefit of RT on RFI is driven by local recurrences. For low-risk patients, this benefit may not translate into longer survival. Given the burden of RT and its potential adverse effects, patient preferences should be considered in trials evaluating the omission of RT.

What new information does our modeling and simulation provide about radiotherapy in breast cancer? Although there is substantial uncertainty in the trial’s outcome, it would be unlikely to show noninferiority of omitting radiotherapy. The primary reason is using relative instead of absolute benefit of radiotherapy. In addition, the evidence is clear that radiotherapy is effective in lowering the rate of local recurrence regardless of risk. For lower-risk patients, distant recurrence becomes less common, and local recurrence becomes even the more important when RFI is the endpoint. An implication is that for low-risk populations the primary statistical measure of benefit of RT should not be the hazard ratio.

Our study illustrates the utility of a collaborative approach. Modelers cannot appropriately model a disease and its clinical trials without working closely with experts in managing and researching the disease. One way that collaboration improves the designs of clinical trials is that preparing for carrying out models forces designers to systematically think through design issues, data sources, and implications of the trial.

Modeling and simulation can help designers by 1) predicting a proposed trial’s outcome based on available information and quantifying the uncertainty associated with that prediction; 2) investigating key inputs, such as patient eligibility criteria and assumed treatment effects, and revealing assumptions for which results are highly sensitive to conclusions and pointing to preparatory investigations that could help reduce the uncertainty in key factors; 3) supplementing results of actual trials by synthesizing with information from historical trials. Overall, this type of simulation modeling adds value by helping to modify or redesign a proposed trial, including possibly changing its primary endpoint and ultimately informing whether a particular trial has the likelihood to change practice. Such analyses could be broadly employed by various stakeholders to inform prioritization and resource allocation of future trial proposals.

Funding

This work was supported by the National Institutes of Health under National Cancer Institute (NCI) Grant U01CA12958, U10CA180868, UG1CA189867, and U10CA180822. The research was also supported in part by supplemental funding provided by the NCI’s Coordinating Center For Clinical Trials, a Lombardi Comprehensive Cancer Center American Cancer Society Young Investigator Award (ACS IRG 92-152-20), and the Cancer Prevention Research Fellowship sponsored by the American Society of Preventive Oncology and Breast Cancer Research Foundation (ASPO-17-001) to Dr. Jayasekera.

Notes

Affiliations of authors: NRG Oncology, and the Department of Biostatistics, University of Pittsburgh Graduate School of Public Health, Pittsburgh, PA (SJA); Canadian Cancer Trials Group, Queen’s University, Kingston, ON, Canada (JAWC, retired); Division of Cancer Control and Population Sciences, National Cancer Institute, Bethesda, MD (EJF); Cancer Clinical Research Unit, University of Toronto Princess Margaret Cancer Centre, Toronto, ON, Canada (AWF); Department of Biostatistics at Harvard University and Department of Biostatistics and Computational Biology at the Dana-Farber Cancer Institute, Boston, MA (RG); Department of Surgery, Duke Cancer Institute, Duke University Medical School, Chapel Hill, NC (ESH); Novant Health Oncology Specialists, Winston-Salem, NC (JOH); Department of Radiation Oncology, University of Michigan, Ann Arbor, MI (RJ); Department of Oncology, Georgetown University Medical Center and Cancer Prevention and Control Program, Georgetown-Lombardi Comprehensive Cancer Center, Washington, DC (JJ, JM); NRG Oncology, and the Division of Breast Surgical Oncology, Allegheny General Hospital, Allegheny Health Network, Pittsburgh, PA (TBJ); McMaster University and Hamilton Heath Sciences, Hamilton, ON, Canada (TW); Department of Biostatistics, University of Texas M.D. Anderson Cancer Center, Houston, TX (YL, JS, XH, DB); Department of Biostatistics, Bioinformatics and Biomathematics, Georgetown University Medical Center, Washington, DC (GL); Institute for Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg, Freiburg, Germany (WS); Departments of Family and Social Medicine and Epidemiology and Population Health (CBS) and Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY (JAS); Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA (NKS); Department of Radiation Oncology, The James, The Ohio State University Comprehensive Cancer Center, Columbus, OH (JW); Department of Radiation Oncology, Indiana University, Indianapolis, IN (RZ).

The study funders had no role in the design of the study; the collection, analysis, or interpretation of the data; the writing of the manuscript; or the decision to submit the manuscript for publication.

This project was a collaboration of CISNET and the Breast Oncology Local Disease (BOLD) Task Force of the Breast Cancer Steering Committee from the National Clinical Trials Network.

Members of the CISNET-BOLD Collaborative included: Donald A. Berry, PhD*, Judith-Anne W. Chapman, PhD, Joseph P. Costantino, PhD, Anthony Fyles, MD, FRCPC, Robert Gray, PhD, Judith O. Hopkins, MD, Xuelin Huang, PhD, E. Shelley Hwang, MD, MPH, Reshma Jagsi, MD, DPhil*, Jinani Jayasekera, MS, PhD*, Thomas B. Julian, MD, Yisheng Li, PhD*, George Luta, PhD, Jeanne Mandelblatt, MD, MPH*, Willi Sauerbrei, PhD, Clyde B. Schechter, MD, MA*, Joseph A. Sparano, PhD, Juhee Song, PhD, Stewart J. Anderson, PhD, Natasha Stout, PhD, Timothy Whelan, MD, Julia White, MD*, Richard C. Zellars, MD, Eric J. Feuer, PhD, Ex Officio. Those indicated with an asterisk were the writing committee.

Jinani Jayasekera, Yisheng Li, Clyde Schechter, Reshma Jagsi, Juhee Song, Julia White, George Luta, Judith-Anne Chapman, Eric Feuer, Richard Zellars, Natasha Stout, Thomas B. Julian, Tim Whelan, Xuelin Huang, E. Shelley Hwang, Judith Hopkins, Stewart Anderson, Anthony Fyles, Willi Sauerbrei, and Jeanne Mandelblatt have nothing to disclose. Julia White received honoraria from Qfix. Joseph A. Sparano owns stock in Metastat, has served in an advisory role for Genentech/Roche, Novartis, AstraZeneca, Celgene, Lilly, Celldex, Pfizer, Prescient Therapeutics, Juno Therapeutics, and Merrimack, and has received research funding from Presceint Therapeutics, Deciphera, Genentech/Roche, Merck, Novartis, and Merrimack. Robert Gray has received research funding from Abbott Molecular, Agios, Amgen, AstraZeneca, Bristol-Myers Squibb, Boehringer Ingelheim, Celgene, Genentech/Roche, Genomic Health, Genzyme, GlaxoSmithKline, ImClone Systems, Janssen-Ortho, Kanisa, Millennium, Nodality, Onyx, OSI Pharmaceuticals, Pfizer, Sanofi, Sequenta, Syndax, and Novartis. Donald Berry is co-owner of Berry Consultants, LLC, a company that designs adaptive Bayesian clinical trials for pharmaceutical and medical device companies, National Institutes of Health cooperative groups, international consortia, and patient advocacy groups.

Jinani Jayasekera was responsible for conception, design, data analyses, interpretation of results, manuscript preparation, and approval of the final manuscript. Yisheng Li was responsible for data analyses, interpretation of results, manuscript preparation, and approval of the final manuscript. Clyde Schechter was responsible for conception, design, data analyses, interpretation of results, manuscript preparation, and approval of the final manuscript. Reshma Jagsi was responsible for conception, interpretation of results, and approval of the final manuscript. Joseph Sparano was responsible for conception, interpretation of results, and approval of the final manuscript. Juhee Song was responsible for data analyses and approval of the final manuscript. Julia White was responsible for conception, interpretation of results, and approval of the final manuscript. George Luta was responsible for data analyses, interpretation of results, and approval of the final manuscript. Judith-Anne Chapman was responsible for interpretation of results and approval of the final manuscript. Eric J. Feuer was responsible for conception, provision of data, interpretation of results, and approval of the final manuscript. Richard C. Zellars was responsible for interpretation of results and approval of the final manuscript. Natasha Stout was responsible for design, data analyses, and interpretation of results, manuscript preparation, and approval of the final manuscript. Thomas B. Julian was responsible for conception, interpretation of results, and approval of the final manuscript. Timothy Whelan was responsible for interpretation of results and approval of the final manuscript. Xuelin Huang was responsible for data analyses, interpretation of results, and approval of the final manuscript. E. Shelley Hwang was responsible for interpretation of results and approval of the final manuscript. Judith O. Hopkins was responsible for interpretation of results and approval of the final manuscript. Stewart J. Anderson was responsible for provision of data, interpretation of results, and approval of the final manuscript. Anthony W. Fyles was responsible for provision of data, interpretation of results, and approval of the final manuscript. Robert Gray was responsible for provision of data, interpretation of results, and approval of the final manuscript. Willi Sauerbrei was responsible for provision of data, interpretation of results, and approval of the final manuscript. Jeanne Mandelblatt was responsible for conception, design, data analyses, interpretation of results, manuscript preparation, and approval of the final manuscript. Donald Berry was responsible for conception, design, data analyses, interpretation of results, manuscript preparation, and approval of the final manuscript.

We are grateful to the researchers who shared de-identified individual data for pooling and use in development of model parameters: Toronto/Vancouver Trial: Anthony Fyles; National Surgical Adjuvant Breast and Bowel Project (now NSABP, Radiation Therapy Oncology Group, and Gynecologic Oncology Group [NRG Oncology]) Trials: Stewart J. Anderson, Joseph P. Costantino; Cancer And Leukemia Group B (now Alliance) 9343: Donald Berry; Eastern Cooperative Oncology Group-American College of Radiology Imaging Network Trial Assigning IndiviuaLized Options for Treatment (Rx; Oncotype scores <11): Joseph Sparano, Robert Gray; German Breast Cancer Study Group-V: Willi Sauerbrei.

We also thank Genomic Health, Inc. (S. Shak), for provision of proprietary, de-identified, locked NSABP-Genomic Health data. We thank Jo Anne Zujewski and the National Cancer Institute staff for assistance facilitating data requests. We also thank Jennifer Hayes for facilitating the CISNET-BOLD collaboration.