Image-Based Deep Neural Network for Individualizing Radiotherapy Dose Is Transportable Across Health Systems

PURPOSE

We developed a deep neural network that queries the lung computed tomography–derived feature space to identify radiation sensitivity parameters that can predict treatment failures and hence guide the individualization of radiotherapy dose. In this article, we examine the transportability of this model across health systems.

METHODS

This multicenter cohort-based registry included 1,120 patients with cancer in the lung treated with stereotactic body radiotherapy. Pretherapy lung computed tomography images from the internal study cohort (n = 849) were input into a multitask deep neural network to generate an image fingerprint score that predicts time to local failure. Deep learning (DL) scores were input into a regression model to derive iGray, an individualized radiation dose estimate that projects a treatment failure probability of < 5% at 24 months. We validated our findings in an external, holdout cohort (n = 271).

RESULTS

There were substantive differences in the baseline patient characteristics of the two study populations, permitting an assessment of model transportability. In the external cohort, radiation treatments in patients with high DL scores failed at a significantly higher rate with 3-year cumulative incidences of local failure of 28.5% (95% CI, 19.8 to 37.8) versus 10.2% (95% CI, 5.9 to 16.2; hazard ratio, 3.3 [95% CI, 1.74 to 6.49]; P < .001). A model that included DL score alone predicted treatment failures with a concordance index of 0.68 (95% CI, 0.59 to 0.77), which had a similar performance to a nested model derived from within the internal cohort (0.70 [0.64 to 0.75]). External cohort patients with iGray values that exceeded the delivered doses had proportionately higher rates of local failure (P < .001).

CONCLUSION

Our results support the development and implementation of new DL-guided treatment guidance tools in the image-replete and highly standardized discipline of radiation oncology.

INTRODUCTION

Stereotactic body radiotherapy (SBRT) is a highly effective treatment for patients with early-stage lung cancer or oligometastatic disease to the lung.^1-5 As an effective and noninvasive treatment, it has been widely integrated into clinical practices. However, despite its overall efficacy, failure rates after lung SBRT remain unacceptably high in some patient subgroups, which is in part attributed to limitations in prescription dose guidance.^6-8

CONTEXT

• Key Objective

• Artificial intelligence (AI) has the potential to extract, store, and transport manifest and latent expert knowledge across hospital systems, an advance that can dramatically alter the current clinical paradigm. However, clinical adoption of these models beyond the originating health system requires model transportability or the ability to produce accurate predictions on new sets of patients in other clinical settings.

• Knowledge Generated

• We demonstrate that a computed tomography–based deep neural network trained to predict local failures after high-dose lung radiation treatments in one hospital system is transportable to another. We also show than an individualized radiation dose estimate (iGray) on the basis of the AI-generated risk score is similarly transportable.

• Relevance

• Our AI framework is poised to provide readily implementable radiation dose guidance across varied medical facilities and patient populations.

Dose guidance in radiotherapy is challenged by patient and tumor heterogeneity. There is significant biologic heterogeneity across patients and their tumors.^9-11 Heterogeneity indicates that uniform radiotherapy doses optimized for populations of patients leads to the under- and overtreatment of a significant proportion of individuals.¹² The current dose guidance paradigm for lung SBRT uses a generic anatomically stratified approach. For example, peripherally located lung tumors typically receive higher doses of radiation and have a lower local failure rate of approximately 7% at 5 years.^1,2 On the other hand, central tumors receive lower doses and have local failure rates of approximately 15%-20% at 3 years.³ Therefore, some tumors are not controlled even at the highest doses and a significant proportion of lung tumors (approximately 80%-85%) are controlled using relatively lower doses of radiation (approximately 100 Gy biologically effective dose [BED]). Efforts to calibrate doses on the basis of individual tumor features represent an unmet clinical need.

We previously developed an image-based deep learning (DL) framework for individualizing lung SBRT dose from a single hospital system.⁷ We devised a multitask learning design that intentionally added constraints and regularity to the network so that it can both reconstruct classical radiomics features and accurately estimate the time to local failure for each patient. These constraints were designed to decrease the number of free parameters in the learning process, thereby reducing the risk of overfitting. As such, this model was optimally designed for transportability across other clinical sites and health systems. In this article, we examine the transportability of this model across distinct clinical settings.

METHODS

Patients

An Institutional Review Board–approved study at Northwestern University (NU; NU00212113) was used to identify 302 patients treated with lung SBRT from 2010 to 2020. Patients with primary (stage I-IV) or recurrent lung cancer and patients with other cancer types with solitary or oligometastases to the lung were included. The study was conducted retrospectively and granted a waiver of consent, meeting NU's Institutional Review Board requirements. Patients were treated on the basis of either a pathologic or radiographic diagnosis. A diagnosis of lung cancer was confirmed by histologic examination of biopsy specimens for 85.6% of the patients studied. A radiographic diagnosis was established in cases where a biopsy was contraindicated or was nondiagnostic. Patients without digitally accessible computed tomography (CT) images and radiotherapy structure data were excluded from the study; a total of 271 (from 302) patients met our eligibility criteria. The use of systemic therapy after SBRT in patients considered at risk for distant failure was not routinely recommended given patient comorbidities. A small fraction of patients received systemic treatments (13.6%) at the discretion of the multidisciplinary treatment team. The patient population used as input for the development (training and validation) of the DL model comprised 849 patients from another hospital system (Cleveland Clinic) who met the same eligibility criteria.⁷

CT Image Data Set

Pretherapy CT simulation images with corresponding physician-designated gross tumor volumes were used for our analysis. Free breathing CT images were obtained with a gantry rotation speed (approximately 1.5 seconds) to produce a low-pitch (0.313) CT scan. The detector collimation of 16 × 1.5 mm was used, which provided a minimum slice thickness of 3 mm. Acquisitions were obtained at 120 kVp, with a tube current of 100 mA. Several scanner types were used for the internal study population (from Cleveland Clinic), namely, three Philips Brilliance CT Big Bore (annotated CT-1, CT-2, and CT-3) and a Philips AcQSim (CT-4). The number of cases scanned on each of the scanners was 499, 244, 40, and 61, respectively; the identity of the scanner could not be definitively determined for five cases.⁷ A Philips Brilliance CT Big Bore (Koninklijke Philips N.V., Amsterdam, the Netherlands) was used for the external study population (NU), annotated CT-5.

Model Development and External Validation

A step-by-step procedure for generating DL scores and a description of the multitask learning framework are given in detail.⁷ In brief, the network consists of three main parts: an encoder for extracting imaging features and building a task-specific fingerprint, a decoder for estimating handcrafted radiomic features, and a task-specific network for generating image signature for therapy outcome prediction. The original model developed from the internal data set used a nested five-fold cross-validation design.⁷ In this article, we developed a new model that used 80% of the internal data set for training and 20% for validation. Parameters were optimized using the training set and selected on the basis of the performance in the validation set. None of the images in the external, holdout set were used in the training or validation processes. The performance of this model was evaluated using the holdout external data set by inputting pretherapy CT images from the external cohort into the model to generate an image score for treatment outcome. We evaluated the performance of both models (nested cross-validation v external validation) using the respective holdout cases. Descriptions of the derivations of the DL score and iGray are given in the Data Supplement.

Statistical Analysis

Tumors were grouped into the following categories for analyses: clinical (radiographic diagnosis only), lung adenocarcinoma, lung squamous cell carcinoma, nonlung primaries (metastatic to lung), and other lung or pleura. The clinical group included patients who did not undergo a tissue biopsy or those with a nondiagnostic biopsy. The nonlung primary group included patients with disease that originated outside of the lung but comprised solitary or oligometastatic intrathoracic disease treated with SBRT. Patients in the other lung or pleura category were excluded from univariate and multivariate analyses because of the low sample size and variability in histologic subtypes. Local failure was defined as radiographic progression within 1 cm of the planning target volume to maintain a consistent definition of local/marginal failure in clinical trials of SBRT.^1,13,14 Prescription radiation dose was adjusted for the number of fractions of radiation by calculating the BED with a standard α/β ratio of 10. Length of follow up was determined from the end date of SBRT, and patients who had not died were censored at the time of last chest imaging. Death without evidence of local failure was treated as a competing event, and Fine and Gray regression modeling was used to examine potential predictors of local failure. Cumulative incidence curves (CICs) for local failure were estimated using the competing risk method, and Gray's test was used to determine significance between CICs.¹⁵ Actuarial analysis was used to estimate rates of overall survival, and the Kaplan-Meier method was used to generate overall survival curves. The concordance index (C-index) was measured between network output and actual event (local failure) time. The 2.5th and 97.5th percentile of the bootstrapped C-index distribution was used as an estimation of the 95% confidence interval. Statistical analysis was performed using R 4.1.2 (The R Foundation, Vienna, Austria).¹⁶

RESULTS

Patient Characteristics

A total of 849 and 271 patients met our eligibility criteria in the internal and external study populations, respectively. There were several substantive differences in baseline patient characteristics between the external and internal study populations. These included longer median time to follow-up (P = .017), ethnic representation (P < .001), intent of the treatment (P < .001), use of motion control (P < .001), stage of tumors (P < .001), cancer types or histology (P < .001), tumor axial diameter (P < .001), use of adjuvant systemic therapy (P < .001), and radiation dose (P < .001; Table 1). These results indicated significant differences across several categories between the two groups.

TABLE 1.

Baseline Characteristics of the Study Populations

Image-Based DL Scores Predict Local Treatment Failures

We developed a new DL model that used 80% of the internal data set for training and 20% for validation (Fig 1A). We used this model to calculate a network-derived image risk or DL score for each patient in the external study group (Fig 1B). The median risk score trends across cancer types were similar in both cohorts. For example, patients with lung squamous cell carcinoma and those with a diagnosis on the basis of radiographic criteria only (“clinical) had the highest and lowest overall scores, respectively, in both data sets. Despite these general trends, however, there remained considerable variance within each cancer type category in both data sets, indicating significant intragroup differences in the risk for local failure.

FIG 1.

Study design and DL score estimates in the external study population. (A) The internal study model is trained and validated using all 849 cases from the internal study population. The external study population was used to test the model using a holdout cross-validation design. The DL score captures image-based latent variables that are associated with local treatment failures. iGray is derived from a regression model that combines the DL score and historical prescribed doses to estimate an individualized radiation dose that projects a treatment failure probability of < 5% at 24 months. (B) DL scores had significant variation across and within cancer types. Box plot (median, interquartile range, and minimum/maximum) of DL scores across the designated cancer types. Differences between two means were measured by a Welch two sample t-test. *P < .05. DL, deep learning; LUAD, lung adenocarcinoma; LUSC, lung squamous cell carcinoma.

Estimated CICs of local failure for each risk group stratified by the median(s) from the internal study group are shown in Figs 2A and 2B. Gray's tests for equality across risk groups were significant. In the external study population, patients in the low-risk group had a significantly lower rate of treatment local failure than do those in the high-risk group, with 3-year cumulative incidences of 10.2% (95% CI, 5.9 to 16.2) and 28.5% (95% CI, 19.8 to 37.8), respectively.

FIG 2.

DL scores predict local failures in the external study population. Estimated CICs for local failure in the (A) internal study population (n = 849) apportioned by the median from each partition of all five folds of the internal study training set or (B) external study population (n = 271) apportioned by the median of the DL scores from the internal study cohort. Estimated CICs for local failure in the external study group apportioned by the median DL score from the internal study cohort for (C) early-stage (IA-IIB) or (D) late-stage (IIIA-IV) tumors. Gray's test was used to test for equality across risk groups. CIC, cumulative incidence curve; DL, deep learning; HR, hazard ratio.

To determine the breadth of clinical setting(s) in which DL scores may be predictive of local failure in the external study group, we examined the impact of the tumor stage on DL score outputs and their prediction accuracy. There were no significant differences in mean scores across the stages of disease (P = .69, analysis of variance [ANOVA] F-test). However, there was significant variation within and across individual stages (Data Supplement). These results suggested that information beyond tumor stage was learned by the model. Consistent with this observation, the DL score predicted local failure in patients with early- or late-stage cancers in the external study population (Figs 2C and 2D).

To assess the influence of possible variation in the types of treatments delivered or CT image acquisition, we assessed the impact of study group, motion management, and CT scanners on the model. Scores were not significantly different on the basis of the study group (internal v external, P = .10; t-test with Welch correction; Data Supplement) or the use of motion management for treatment in the external study group (no compression v abdominal compression, P = .11; t-test with Welch correction; Data Supplement). Across five CT scanners used in both the external and internal study groups, the medians approximated 0 with a range of –0.17 to 0.15 (Data Supplement). The median scores across most scanners were not significantly different with a singular exception: the median score derived from images in the CT-4 group (Philips AcQSim) were significantly higher compared with CT-2 and CT-5 (P < .002; ANOVA F-test). It was unclear if this reflected the relatively lower number of cases scanned using the Philips AcQSim, systematically higher values obtained with this scanner type, or differences in tumor characteristics in this population compared with the others. Patients scanned on CT-4 had a significantly higher tumor axial diameter than CT-2 and CT-5 (P < .002, ANOVA F-test), suggesting, at least in part, a role for distinct tumor characteristics as a source for the higher DL score values.

DL Score Performance Within and Across Data Sets

We compared the performance of the DL score in the external study cohort with other clinical and pathologic variates. On univariate analysis, a higher DL score, larger tumors (on the basis of volume), and cancer type (clinical v all others) were associated with an increased risk of local failure (Data Supplement). On multivariate analyses, the DL score and cancer type remained significantly associated with local failure (Data Supplement). The higher association between local failure and the DL score compared with tumor volume indicated that features beyond tumor size, which has been previously shown to be associated with local failure after high-dose radiotherapy to the lung,¹⁷ can be identified using the DL model.

The DL score predicted local treatment failure with a C-index of 0.68 (95% CI, 0.59 to 0.77), which was comparable with the C-index from a nested model developed from the internal study cohort (Table 2). Cancer type improved the C-index in both cohorts by 0.02. The addition of BED to a model that included the DL score and cancer type led to a small improvement in the internal study cohort but did not improve concordance in the external study group. This is likely attributed to the low variance in the dose delivered in the external study cohort: 87.4% of the patients received either 50 Gy in five fractions or 48 Gy in four fractions. These results indicated that an image-based score can provide complementary information to the established clinical variables and performed well across distinct clinical settings.

TABLE 2.

Prognostic Performance Across Models and Cohorts

Individualized Dose Projections Predict Local Failures

We studied the dose-effect relationships across data sets to identify patients who may benefit from dose modulation. The iGray value was calculated for each patient using a permuted holdout set design in the internal study group and holdout set in the external study group. The kernel densities of dose delivered (ie, actual) compared with iGray had lower variances in both data sets, reflecting the discrete doses and fractionations prescribed by physicians (Fig 3A). In addition, the kernel densities between iGray and actual dose delivered had a greater overlap in the internal study group, consistent with the generally higher prescription doses in the internal study group (Fig 3A and Table 1).

FIG 3.

Individualized radiation dose guidance projections predict local failure outcomes. (A) The kernel density estimation of the actual dose of radiation delivered and iGray. The heatmap provides the tail probability in each kernel density distribution. iGray is in BED units. (B) Histogram and density distribution of the differenced values between iGray and actual dose delivered for each patient in the external study population. The histogram is stratified into three differenced categories (blue, red, or teal). (C) Estimated cumulative incidence curves for local failure in the external study population apportioned by the three different categories shown in (B). The rate of local failure increases as the magnitude of the difference values increases. Gray's test was used to test for equality across risk groups. BED, biologically effective dose.

We next determined if the extent of discordance between the individualized dose projection and the actual dose delivered was predictive of outcomes. Patients in the external study group were stratified into intervals on the basis of the difference between the iGray and the delivered dose (from lowest to highest risk): (1) ≤ 0 Gy, iGray exceeded or matched the delivered dose; (2) 1-100 Gy, iGray was between 1 and 100 Gy higher than the delivered dose; and (3) > 100 Gy, iGray was more than 100 Gy higher than the delivered dose. A histogram of these difference is shown in Figure 3B. Patients in the lowest-risk group had a significantly lower rate of treatment local failure than do those in the intermediate- and highest-risk group, with 3-year cumulative incidences of 0% (95% CI, not available), 13.0% (95% CI, 8.4 to 18.6), and 39.9% (95% CI, 25.5 to 53.5), respectively. Estimated CICs of local failure demonstrated an increase in the risk of local failure on the basis of the designated risk groups (1)-(3) (Fig 3C). Moreover, Gray's tests for both equality for iGray and dose delivered were statistically significant in a multivariable model (Data Supplement).

Local Failures Are Associated With Other Disease Progression and Death

The impact of local recurrences on other disease progression and overall survival in patients with early-stage lung cancer remains unclear.¹⁸ Since the goal of dose optimization is to improve local outcomes and prevent other disease progression and death, discerning these relationships is pivotal. We assessed the impact of local failures on other cancer outcomes and death in patients with early-stage lung cancer treated with curative intent. To increase statistical power, we federated our data sets resulting in 939 patients. Sankey diagrams of patients with local failures (Fig 4A) compared with those with locally controlled disease (Data Supplement) demonstrated higher crude rates of nodal and distant failure. Patients with a local failure experienced nodal metastasis (hazard ratio [HR], 3.69 [95% CI, 2.48 to 5.49]), nodal or distant metastases (HR, 3.64 [95% CI, 2.70 to 4.93]), and death (HR, 1.49 [95% CI, 1.15 to 1.82]) at significantly higher rates than those with a controlled primary tumor (Figs 4B-4D). We next assessed the temporal sequence of treatment failures in the same population. We showed that local recurrence occurred before or concurrent with nodal or distant failure in the majority (approximately 80%) of cases in which the respective events co-occurred during the follow-up period (Fig 4E).

FIG 4.

Local failures precede additional disease progression and are associated with higher rates of nodal/distant failures and death. (A) The relationship between local, nodal, and distant recurrences and death in patients with local failure is demonstrated in a Sankey diagram. Estimated cumulative incidence curves for (B) nodal failures, (C) nodal and distant failures, and (D) death in the external study population apportioned by local failure status. The numbers at risk represent patients with early-stage lung cancer treated with lung stereotactic body radiotherapy from both cohorts stratified by local failure status. (E) The proportion of cases with the indicated temporal sequence of local, nodal, and distant failure. HR, hazard ratio.

DISCUSSION

Image-based algorithms have not been integrated into routine clinical practices to date because of general concerns about model accuracy and robustness.¹⁹ These limitations emerge mainly from the inadequate testing of these models across varying clinical settings. Our results demonstrate that a deep neural network trained to predict local failures after lung SBRT in one hospital system is transportable to another, despite significant interpopulation differences. A model that included cancer type and image scores demonstrated similar accuracy in predicting outcomes in both health systems. These results indicated that the DL model is robust to variance across clinical sites. The clinically significant implication of our study is that the knowledge acquired from the DL of SBRT treatments in one hospital setting could be transported to other clinical sites.

We also used an integrated method that combines CT image features with historical dose data to project individualized dose prescriptions. Since approximately 96% of all local failure occurred within 24 months from treatment in our data sets, iGray estimates should approximate long-term local control. We showed that patients with iGray values less than or equal to the delivered dose had no local failures at 3 years in the external data set. On the other hand, greater discordance between the iGray and delivered dose (> 0) was associated with significantly higher rates of local failure. These results indicate that iGray is similarly transportable, supporting a role for individualized dose guidance in radiotherapy-based dose optimization trials.

Relatedly, estimates of iGray ranged from 73 to 267 Gy BED in the external study cohort. Although most of the recommended doses can be delivered safely, some of the iGray doses exceed the current dose recommendations. Of note, there is some uncertainty at the high end of the iGray range related, in part, to inaccuracies in extrapolation beyond the scope of the regression model. In addition, iGray is based on a linear regression model and is therefore unbound (no upper limit), unlike the experimentally validated and bound sigmoidal function that characterizes tumor control probability.²⁰ Rather than defining an arbitrary asymptote, empirical data are needed to accurately identify the bound at the maximal end of the iGray range. Therefore, prospective clinical trials using higher doses for at risk tumors are warranted.

Although there is a compelling rationale for local treatment failures conferring poor prognoses, their impact is likely dependent on multiple variates including intercurrent death. Using one of the largest data sets of lung SBRT to date, we showed that local failures significantly increase the risk of nodal and distant disease progression and death. The temporal sequencing of categories of disease recurrence suggests that local failures can serve as niduses for regional and distant disease dissemination. Overall, these findings add importance to the prevention of local failures by SBRT treatment optimization approaches as developed, in part, herein.

The strengths of our study include the large number of patients evaluated, the explicit and implicit heterogeneity across hospital systems, the use of a carefully annotated radiotherapy-specific outcome (local failure) rather than surrogate of treatment failure (eg, progression-free survival), and the use of a robust image-based score as a backbone for our analyses. The limitations of our study include the following: we cannot fully account for all potential causes of bias, most patients were scanned using a Philips Brilliance CT Big Bore scanner, good performance in a single external validation population does not mean that it will generalize to all other populations, and we do not assess additional orthogonal variates that could affect treatment failures and enhance prediction accuracy. These limitations can, in part, be addressed by the incorporation of new data sets and/or other variates including tumor motion²¹ and/or next-generation sequencing (ie, tumor genetic) data²² to improve prediction accuracy.

There is cumulative evidence to suggest a genetic basis for local failures after lung SBRT.^23-25 Although CT-based features seem to provide complementary prognostic data when combined with other variate classes,²⁶ it remains unclear whether image-based features capture the underlying genetic alterations of individual lung tumors.²⁷ The development of integrated multiomic lung SBRT data sets will be necessary to assess whether these distinct classes of predictive variates, namely radiomics and genomics, are tautologic, orthogonal, or somewhere in between. These ongoing efforts represent a pivotal step in the development of integrated models that can more accurately predict radiation sensitivity and project individualized doses of radiation.

In summary, our deep neural network and dose guidance frameworks are poised for pretreatment risk stratification and risk-adapted dose optimization in clinical trials and, ultimately, in routine clinical practices.

DATA SHARING STATEMENT

The data sets analyzed during the current study will be available from the corresponding author at the time of publication. Per institutional policy, the data sets are designated limited access. On receiving access, the investigator may only use them for the purposes outlined in the request to the data provider and redistribution of the data is prohibited.

AUTHOR CONTRIBUTIONS

Conception and design: Jainil Shah, Jyoti Patel, Ali Kamen, Mohamed E. Abazeed

Financial support: Jainil Shah, Mohamed E. Abazeed

Administrative support: Ali Kamen, Mohamed E. Abazeed

Provision of study materials or patients: Jyoti Patel, Mohamed E. Abazeed

Collection and assembly of data: James Randall, P. Troy Teo, Mohamed E. Abazeed

Data analysis and interpretation: James Randall, Bin Lou, Jyoti Patel, Ali Kamen, Mohamed E. Abazeed

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).