Performance of lung cancer prediction models for screening-detected, incidental, and biopsied pulmonary nodules

Abstract

Purpose:

To evaluate the performance of eight lung cancer prediction models on patient cohorts with screening-detected, incidentally-detected, and bronchoscopically-biopsied pulmonary nodules.

Materials and Methods:

This study retrospectively evaluated promising predictive models for lung cancer prediction in three clinical settings: lung cancer screening with low-dose CT, incidentally detected pulmonary nodules, and nodules deemed suspicious enough to warrant a biopsy. The area under the receiver operating characteristic curve (AUC) of eight validated models including logistic regressions on clinical variables and radiologist nodule characterizations, artificial intelligence (AI) on chest CTs, longitudinal imaging AI, and multi-modal approaches for prediction of lung cancer risk was assessed in 9 cohorts (n=898, 896, 882, 219, 364, 117, 131, 115, 373) from multiple institutions. Each model was implemented from their published literature, and each cohort was curated from primary data sources collected over periods within 2002 to 2021.

Results:

No single predictive model emerged as the highest-performing model across all cohorts, but certain models performed better in specific clinical contexts. Single timepoint chest CT AI performed well for screening-detected nodules but did not generalize well to other clinical settings. Longitudinal imaging and multimodal models demonstrated comparatively good performance on incidentally-detected nodules. When applied to biopsied nodules, all models showed low performance.

Conclusion:

Eight lung cancer prediction models failed to generalize well across clinical settings and sites outside of their training distributions.

Summary Statement

The performance of 8 different statistical models for lung cancer prediction depended heavily on the clinical setting they are applied in and generalized poorly on sites outside of their training distributions.

Graphical Abstract

1. Introduction

Every year, an estimated 1.57 million Americans have at least one pulmonary nodule detected either incidentally on routine chest CT or during lung cancer screening ¹. While biopsy of the nodule remains the reference standard diagnostic test for malignancy, it involves an invasive procedure associated with morbidity, mortality, additional healthcare costs and anxiety for patients ^2,3. With 95% of indeterminate pulmonary nodules (IPNs) being benign ⁴, clinical guidelines ^1,5–7 recommend to risk-stratify nodules before resorting to invasive diagnostics and surgical intervention. Statistical models for predicting lung cancer have the potential to improve this risk-stratification, aiding in earlier diagnosis of malignancy as well as reducing morbidity, costs, and anxiety associated with workup of benign disease.

Validated predictive models developed to stratify pulmonary nodules consist of (1) clinical prediction models, (2) cross-sectional or longitudinal artificial intelligence (AI) models, and (3) multimodal approaches. We consider a predictive model validated if it has demonstrated competitive discriminatory performance (area under the receiver operating characteristic curve [AUC] above 0.75) on a separate test cohort. The Brock ⁸ and Mayo ⁹ models are two of the most used models in clinical practice and recommended by clinical guidelines. They are well-validated logistic regressions and are based on readily available variables, such as demographics ¹⁰, smoking history, and radiologists’ assessment of nodule features. However, they require radiologists to first detect and characterize the nodule, a step that can be subject to inter-reader variability ^11–13.

Recent research has validated several AI models for cancer prediction. These operate directly on the voxels of the image, negating the need for radiologists to first describe nodule morphology or measure sizes. One of the early AI successes in lung cancer prediction was Liao et al ¹⁴. Their two-step approach involved first detecting suspicious lesions in the lung field from a single chest CT image and then computing malignancy risk from the proposed regions of interest (ROIs). Recently, Mikhael et al.¹⁵ publicly released Sybil, a predictive model that extracts global chest features along with regional attention features to predict lung cancer risk up to 6-years.

Previous work has also leveraged AI on longitudinal imaging. Gao et al.¹⁶ and Li et al.¹⁷ extended the work of Liao et al. to leverage consecutive chest CTs and the time interval between scans. In another longitudinal imaging approach, Ardila et al.¹⁸, demonstrated impressive AUC performance of a model including global chest features outside of local ROIs, but their model was not released publicly. Recently, efforts that leveraged data from multiple modalities have shown ^19,20, with limited validation, that the combination of clinical variables and imaging AI can improve performance over single modality approaches.

The plethora of research is promising, but several concerns arise when considering the clinical utility of predictive models in lung cancer diagnostics. First, the comparative advantage of AI models versus commonly used linear models has not been quantitatively characterized in settings where a predictive model would arguably have the most impact. Second, almost all of the AI models are, to some extent, trained on lung screening scans from the National Lung Screening Trial (NLST) ²¹, which raises the question of whether they generalize across institutions and to patients with incidentally-detected nodules and metastases to the lung from other sites. Third, these models predict different outcomes. Some assess the risk of developing lung cancer over a multi-year period, whereas others estimate the probability that an observed pulmonary nodule is malignant. A comparative analysis of these models using a standardized outcome (e.g. 2-year diagnosis of lung cancer) can inform “off-label” use of models but has not yet been performed. Finally, there is an urgent need to risk-stratify intermediate-risk nodules and reduce the number of biopsies on nodules that appear indeterminate but turn out to be benign. To our knowledge, a systematic analysis of how existing models perform in this setting has not been performed.

This study aimed to evaluate 8 validated lung cancer prediction models on cohorts with screening-detected nodules, incidentally-detected nodules collected in both retrospective and prospective fashion, and nodules that underwent a bronchoscopic biopsy. These different settings are where we envision a well-designed predictive model will have a tangible impact on patient care. We implemented each model from their published code repositories and curated each cohort from their primary available source.

2. Materials and Methods

2.1. Cohorts

The data included in this retrospective study were sourced from the National Lung Screening Trial (NLST), through the Cancer Data Access System and our medical center A. This study also included data from a multi-institute consortium, which includes the Veterans Affairs associated with academic medical center A, academic medical center B, Detection of Early lung Cancer Among Military Personnel (DECAMP), and academic medical center C. We derived 10 named cohorts from these sites using different inclusion criteria (Table 1). We obtained CT scans, demographics, questionnaire data from CT arm of the NLST upon request with a data use agreement (https://cdas.cancer.gov/learn/nlst/images/). Data from Lung Screening-A (LS-A) was acquired under our home institutional review board (IRB) supervision #181279. Longitudinal Incidental-A (LI-A) and BRONCH were acquired under IRB supervision #140274. Consortium-A was acquired under IRB supervision #030763 and #000616. Consortium-B, Consortium-C, and DECAMP were acquired via academic collaborations under different grants. Data acquisition of these cohorts were HIPAA compliant. Regarding patients that have been previously reported, the NLST is a widely studied public dataset. Portions of LS-A have been reported in ²² (n=1189) and ²³(n-147), LI-A was previously reported in ^24,25. Patients from Consortium-A, Consortium-B, Consortium-C, and DECAMP have been previously studied in ¹⁹ (n=1331) and ²⁶ (n=457).

Table 1.

Cohort Inclusion/exclusion criteria

	Inclusion/exclusion criteria
NLST-test	Patients correspond to the Ardila et al.18 test set. These patients were not seen by any of the predictive models in this study. Lung cancer events were the biopsy-confirmed lung cancers reported by the NLST. Patients without a confirmed outcome were excluded from this study.
NLST-test-nodule	Subset of the NLST-test cohort in which included patients had at least one positive nodule finding from their CTs as defined in the NLST (>= 4 mm).
NLST-dev	All patients enrolled in the CT arm of the NLST and not part of NLST-test. Those without available imaging or without a confirmed outcome were excluded.
LS-A	Patients meeting the American Cancer Society criteria for lung screening and were enrolled in the lung screening program at medical center A from 2015 to 2018. Patients receive longitudinal follow-up after a positive screen and lung cancer events were confirmed via biopsy reports. Nodule characteristics are missing because radiology reports were not available.
LI-A	Patients from medical center A who acquired three chest CTs within five years between 2012 and 2019. These patients were identified through International Classification of Diseases (ICD) codes to have a pulmonary nodule and no cancer before the nodule. We defined lung cancer outcomes through ICD codes representing any malignancy found in the bronchus or lung parenchyma, including metastases from other sites 24. Nodule characteristics are missing because radiology reports were not available.
Consortium-A	Prospectively enrolled patients from medical center A and its associated VA between 2003 and 2017. Cohorts prefixed with consortium- meet the following inclusion criteria. Patients must be aged 18–80 and were detected incidentally to have a pulmonary nodule with a diameter between 6–30mm. Patients consented at initial nodule detection and serum and CT scan were acquired at that time. Longitudinal imaging and biopsy-confirmed diagnosis for malignant nodules were collected during a 2-year period following initial nodule detection.
Consortium-B	Prospectively enrolled patients from medical center B according to consortium inclusion criteria. Longitudinal imaging after initial nodule detection was not available.
Consortium-DECAMP	Prospectively enrolled patients from 12 clinical centers associated with the DECAMP 37 study protocol. Of note, cases and controls are matched on nodule size.
Consortium-C	Prospectively enrolled patients from medical center C according to MCL inclusion criteria. Longitudinal imaging after initial nodule detection was not available.
BRONCH	Prospectively collected cohort of patients who underwent a bronchoscopic lung biopsy for a pulmonary nodule (defined as a lesion < 3cm) at the lung nodule clinic of medical center A between the years of 2017–19. The subsequent biopsy report from the bronchoscopy was used to determine benign vs. malignant status of the nodule.

Note.—NLST =National Lung Screening Trial , DECAMP = Detection of Early lung Cancer Among Military Personnel.

2.2. Image Preprocessing

We used a documented pipeline ²⁷ that includes algorithmic analysis and manual visual assessment to ensure every scan used in this study passed certain image quality standards (Figure 1). Specifically, we excluded scans with severe imaging artifacts, scans with slice thickness greater than or equal to 5 mm, and scans without the full lung field in the field of view. 713 studies in total were excluded due to insufficient quality (Supp. Fig 7.). Patient health information was removed using the MIRC Anonymizer²⁸.

Figure 1.

Image quality assurance pipeline for each cohort. Exclusion criteria included severe artifact, non-standard chest or body orientation, field of view that did not fully include the lung, and slice thickness > 5mm.

2.3. Predictive Models

We selected an array of models for lung cancer prediction (Table 3), including models designed to estimate lung cancer risk (i.e. Sybil) as well as models designed to predict the malignancy probability of a pulmonary nodule. We included the Brock ⁹ and Mayo ⁸ models, as they are among the most cited, validated, and used in clinical practice. We studied a several AI models incorporating a range of approaches that would allow us to examine the efficacy of three strategies: Liao¹⁴ and Sybil¹⁵ as a single timepoint chest CT approach, Distanced LSTM (DLSTM)¹⁶ and Time-distance Vision Transformer (TDViT)¹⁷ as a longitudinal chest CT approach, and DeepLungScreening (DLS)²⁰ and DeepLungIPN (DLI)¹⁹ as models with multimodal inputs.

Table 3.

Lung Cancer Predictive Model Characteristics

Model	Year Published	Input	Training Distribution	Cancer Prevalence	Outcome Criteria	Approach
Mayo 8	1997	Age, PH, SS, NSpic, NUL, NSize†	Mayo Clinic (n=419)	23%	2-year LC risk proven via tissue biopsy or no findings in follow up	Logistic regression
Brock 9	2013	Age, Sex, FH, Emp, Nsize, Nspic NUL, Ncount, Ntype‡	PanCan 38 (n=1871)	5.5%	2-year LC risk proven via tissue biopsy or no findings in follow up	Logistic regression
Liao 14	2017	Single chest CT	NLST-dev (n=5436)	17%	1-year LC risk proven via tissue biopsy or no findings in follow up	ResNet, nodule detection and ROI-based prediction
Sybil 15	2023	Single chest CT	NLST-dev (n=12672)	17%	Up to 6-year LC risk proven via tissue biopsy or no findings in follow up	ResNet, global chest features and guided attention
DLSTM 16	2020	Longitudinal chest CT	NLST-dev (n=5436)	17%	6-year LC risk proven via tissue biopsy or no findings in follow up	LSTM network, ROI-based prediction, encodes time interval between scans
TdViT 17	2023	Longitudinal chest CT	NLST-dev (n=5436)	17%	6-year LC risk proven via tissue biopsy or no findings in follow up	Transformer network, ROI- based prediction, encodes time interval between scans
DeepLungScreening 20	2021	Single chest CT, Age, Education, BMI, PH, FH, SS, Quit, PYR	NLST-dev (n=5436)	17%	2-year LC risk proven via tissue biopsy or no findings in follow up	ResNet, ROI-based prediction, late fusion of imaging and clinical features
DeepLungIPN 19	2021	Single chest CT, Age, BMI, PH, SS, PYR, Nsize, NSpic, NUL, Serum biomarker¶	Consortium cross validation§ (n=1232)	59%	2-year LC risk proven via tissue biopsy or no findings in follow up	DeepLungScreening, serum biomarker

^*PH: Personal history of any cancer, FH: Family history of lung cancer, SS: smoking status (former v.s. current smoker), SI: smoking intensity (average number of cigarettes smoked a day), SD: smoking duration, Quit: years since the person quit smoking, PYR: pack-years of smoking fNSpic: Nodule spiculation present or absent, NUL: Nodule in the upper lobes, Nodule Size: largest diameter in mm

^†Emp: Presence of emphysema, Ncount: Number of nodules, Ntype: nodule type, categorized as (1) nonsolid or with ground-glass opacity, (2) part-solid, and (3) solid.

^‡Serum concentration of hs-CYFRA 21–1 (natural log of ng/ml) ³⁹

^§Combination of Consortium-A, Consortium-DECAMP, Consortium-C

We split patients with confirmed follow-up in the NLST into development (NLST-dev) and test (NLST-test) sets. NLST-test contains the patients in the Ardila et al. test set that had confirmed follow-up and these scans remained unseen until evaluation. NLST-dev was used to retrain, from random weights, several of the models using a standardized 2-year lung cancer outcome. The purpose of retraining was to (1) ensure that the models were blinded to NLST-test and (2) standardize the predicted outcome across each model. Specifically, the years between initial observation of the patient and the outcome, or year-to-outcome, was not standardized across the evaluated models (Table 3). Models developed using a shorter year-to-outcome have an easier task than models developed using a longer year-to-outcome. In this way, differences in year-to-outcomes can confound model comparisons. For longitudinal imaging models, the outcome was whether the patient was diagnosed with lung cancer within two years of the patient’s latest scan. The logistic regression models were not retrained and were evaluated as published since they were already blinded to NLST-test and we did not have their original development dataset which is needed to control for year-to-outcome. Sybil was also evaluated as published because the model was already blinded to NLST-test and its prediction includes a 2-year outcome. Lastly, DeepLungIPN was originally trained using a cross-validation of Consortium-A, Consortium-DECAMP, and Consortium-C. This model was evaluated as published since the model includes a blood biomarker that was only collected in the MCL cohorts.

Implementation and training of models followed their original methodology unless otherwise specified. Details on the site of development and training distribution are reported in Supp. Table 4. Apart from removing scans that did not meet our image quality standards, we did not add` or remove any pre- or post-processing steps included in the models’ pipeline. The code supporting model training, evaluation, and statistical analysis is available at https://github.com/MASILab/lcancer_baselines.

2.4. Evaluation and Statistical Analysis

Evaluation included all the named cohorts except NLST-dev. A model-cohort evaluation was not feasible when a substantial portion of the input data was missing, specifically when an input variable was missing in more than 10% of cohort patients (Supp. Table 2). When an input variable was missing in less than 10% of patients, we conducted an evaluation using imputation based on a multivariable regression of the other available variables. The effect of imputation on model performance is evaluated in Supp. Table 3. We did not evaluate longitudinal imaging models (DLSTM and TDViT) on cohorts where longitudinal imaging was unavailable. When evaluating DeepLungIPN on the consortium cohorts, we report the out-of-fold cross validation results.

We used AUC to measure model performance for classifying lung cancer cases and benign controls. For each model-cohort evaluation, we used bootstrapping procedure to estimate the model’s performance on the cohort’s true population. The procedure drew 1000 samples of the same size with replacement from the original cohort. Each model’s AUC was calculated for each sample and we reported the mean AUC and 95% confidence interval (CI) over all bootstrapped samples. A two-sided Wilcoxon-signed rank test evaluated significance of differences, at p<0.05, in mean AUC between models within a single cohort. We did not test statistical differences across cohorts because patients were not paired across cohorts.

Model performance for non-small cell lung cancer (NSCLC) vs. small cell lung cancer (SCLC) cases were compared in NLST-test-nodules and Consortium-A. The mean AUC and 95% CI of each model was computed using the same bootstrap procedure drawn from the pool of benign patients and patients with either SCLC or NSCLC. An unpaired t-test evaluated whether a model’s discrimination of malignant versus benign was significantly different at p<0.05 with these two lung cancer subtypes.

An analysis of calibration before and after confidence correction was conducted. In each model-cohort evaluation, 10-fold cross validation was used to fit isotonic regressions on the training set of each fold. Calibration was then evaluated on the validation set of each fold (Supp. Fig. 3–6). All statistical analysis were performed in python with support from the SciPy package.

4. Results

Cohort Characteristics

The size of the evaluation cohorts ranged from 898 patients (NLST-test) to 117 patients (Consortium-B). Mean age ranged from 59 with standard deviation 13 (LI-A) to 69 with a standard deviation of 11 (Consortium-A). Male sex proportion ranged from as low as 23% (Consortium-DECAMP) to high as 61% (NLST-test). Cancer prevalence in the biopsied nodules cohort (BRONCH: 62%) was the highest, followed by incidentally-detected nodules (Consortium-A: 65%, Consortium-B: 41%, Consortium-DECAMP: 49%, Consortium-C: 50%), and lastly screening cohorts (NLST-test: 17%, LS-A: 3%, LI-A: 17%). Mean smoking pack years fell within 47 to 59 except for the biopsied nodules cohort at 29 due to the high proportion of never-smokers. Overall cohorts were distributed differently in terms of cancer prevalence, demographics, smoking background, and nodule characteristics (Table 2).

Table 2.

Cohort Characteristics

Cohort	NLST-dev	NLST-test	NLST-test-nodule	LS-A	LI-A	Consortium-A	Consortium-B	Consortium-DECAMP	Consortium-C	BRONCH
Program Type	Screening	Screening	Screening	Screening	Screening, Incidental	Incidental	Incidental	Incidental	Incidental	Bronchosc opy
Institution	Multi-institute	Multi-institute	Multi-institute	Medical Center A	Medical Center A	Medical Center A, VA A	Medical Center B	Multiinstitute	Medical Center C	Medical Center A
Program Period	2002–09	2002–09	2002–09	2015–18	2012–21	2003–17	2006–15	2013–17	2010–18	2017–19
No. of patients  with lung cancer	5436 901 (17%)	898 149 (17%)	896 147 (16%)	882 24 (3.0%)	219 37 (17%)	364 238 (65%)	117 48 (41%)	131 64 (49%)	115 57 (50%)	373 230 (62%)
No. of scans  with lung cancer	14748 1866 (13%)	2523 313 (12%)	2440 298 (12%)	1483 51 (3.4%)	515 50 (10%)	760 517 (68%)	117 48 (41%)	241 100 (41%)	115 57 (50%)	387 240 (62%)
Slice thickness (mm)	2.1 ± 0.65	2.1 ± 0.42	2.1 ± 0.42	0.81 ± 0.21	0.77 ± 0.61	1.8 ± 1.1	2.2 ± 0.69	1.7 ± 0.91	1.4 ± 0.79	0.9 ± 0.38
Age (y)	62 ± 5.2	62 ± 5.2	62 ± 5.2	65 ± 5.8	59 ± 13	69 ± 11	68 ± 8.5	68 ± 7.9	66 ± 8.3	64 ± 12
Sex (male)	3270 (60%)	546 (61%)	546 (61%)	483 (55%)	109 (50%)	165 (45%)	49 (42%)	30 (23%)	31 (27%)	168 (45%)
BMI (kg/m2)	28 ± 4.8	28 ± 4.9	28 ± 5.0	28.4 ± 6.0	27 ± 7.4	28 ± 6.5	28 ± 4.9	26 ± 5.4	29 ± 6.2	28 ± 6.8
Personal cancer history	256 (4.7%)	43 (4.8%)	43 (4.8%)	135 (15%)	N/A	129 (35%)	3 (2.6%)	65 (50%)	13 (11%)	194 (52%)
Family lung cancer history	1194 (22%)	179 (20%)	177 (20%)	149 (17%)	N/A	41 (11%)	0	0	10 (8.7%)	88 (24%)
Smoking status					N/A
Never	0	0	0	0		33 (9%)	0	11 (8.4%)	22 (19%)	90 (24%)
Former	2781 (51%)	469 (52%)	468 (52%)	357 (40%)		195 (54%)	76 (65%)	67 (51%)	54 (47%)	214 (57%)
Current	2655 (49%)	429 (48%)	428 (48%)	525 (60%)		121 (33%)	41 (35%)	53 (40%)	39 (34%)	69 (18%)
Smoking pack-years	56 ± 25	59 ± 28	59 ± 28	48 ± 21	N/A	47 ± 33	48 ± 23	50 ± 25	50 ± 33	29 ± 30
Nodule size (mm)	8.0 ± 6.2	7.9 ± 5.9	7.9 ± 5.9	N/A	N/A	19 ± 13	16 ± 8.9	15 ± 7.0	18 ± 15	2.2 ± 1.3
Nodule count	1.2 ± 1.3	1.3 ± 1.2	1.3 ± 1.2	N/A	N/A	1.0 ± 0.0	1.0 ± 0.0	1.0 ± 0.0	1.0 ± 0.0	1.0 ± 0.0
Nodule attenuation				N/A	N/A
Solid	7494 (51%)	1351 (54%)	1351 (55%)			725 (95%)	99 (85%)	241 (100%)	115 (100%)	325 (84%)
Part-solid	507 (3.4%)	69 (2.7%)	69 (2.8%)			21 (2.8%)	18 (15%)	0	0	51 (13%)
Non-solid or GGO	1439 (10%)	241 (9.6%)	241 (9.9%)			14 (1.8%)	0	0	0	11 (2.8%)
Nodule spiculation (present)	997 (6.7%)	200 (7.9%)	200 (8.2%)	N/A	N/A	229 (30%)	15 (13%)	126 (52%)	30 (26%)	173 (45%)
Nodule location				N/A	N/A
Upper lobe	5554 (38%)	996 (39%)	996 (41%)			447 (59%)	63 (54%)	143 (59%)	71 (62%)	203 (52%)
Lower lobe	4391 (30%)	777 (31%)	777 (32%)			313 (41%)	54 (46%)	98 (41%)	44 (38%)	184 (48%)

Note.—Data values presented as mean ± SD or number of patients (percentage).

Model Performance

Table 4 reports the mean AUC and 95% CI for each feasible model-cohort evaluation. Supp. Table 1 also reports corresponding sensitivity and specificity using an optimal cut-point for each model-cohort evaluation. Comparing the results row-wise reveals that almost all predictive models exhibited noticeable differences in performance across cohorts (Figure 2A)

Table 4:

Model Classification of n-year lung cancer risk across selected cohorts

		NLST-test (n=898)	NLST-testnodules (n=896)	LS-A (n=882)	LI-A (n=219)	Consortium-A (n=364)	Consortium-B (n=117)	Consortium-DECAMP (n=131)	Consortium-C (n=115)	BRONCH (n=373)	Average Rank (range) n=‰
Input	Model
Clinical variables	Mayo	‡	0.804 [0.798, 0.809]	‡	‡	0.706 [0.704, 0.708]	0.864 [0.862, 0.867]	0.568 [0.565, 0.571]	0.716 [0.712, 0.719]	0.621 [0.615, 0.628]	3.5 (7, 1) n=6
Clinical variables	Brock	‡	0.789 [0.782, 0.796]	‡	‡	0.716 [0.714, 0.718]	0.885 [0.883, 0.886]	0.662 [0.659, 0.666]	0.713 [0.710, 0.716]	0.497 [0.494, 0.499]	3.2 (5, 2) n=6
Single CT AI	Liao et al.	0.751 [0.747, 0.756]	0.755 [0.750, 0.759]	0.723 [0.712, 0.734]	0.644 [0.635, 0.653]	0.662 [0.660, 0.664]	0.779 [0.776, 0.782]	0.706 [0.703, 0.709]	0.660 [0.656, 0.663]	0.621 [0.614, 0.628]	3.9 (6, 1) n=9
Single CT AI	Sybil	0.881 [0.877, 0.885] *	0.879 [0.872, 0.885] *	0.779 [0.768, 0.789]	0.763 [0.756, 0.770]	0.700 [0.694, 0.706]	0.889 [0.884, 0.895]	0.606 [0.597, 0.616]	0.764 [0.756, 0.772]	0.623 [0.618, 0.629]	2.6 (6, 1) n=9
Longitudi nal CT AI	DLSTM	0.738 [0.734, 0.743]	0.727 [0.721, 0.731]	¶	0.711 [0.702, 0.720]	0.743 [0.741, 0.745]	§	0.778 [0.774, 0.781]	§	§	3.6 (6, 2) n=5
Longitudi nal CT AI	TDViT	0.797 [0.793, 0.802]	0.790 [0.785, 0.794]	¶	0.773 [0.764, 0.781] *	0.753 [0.750, 0.755]	§	0.823 [0.820, 0.825] *	§	§	1.8 (3, 1) n=5
Multimodal	DLS	0.783 [0.778, 0.788]	0.776 [0.771, 0.782]	0.810 [0.799, 0.820] *	†	†	†	†	†	†	2.7 (4, 1) n=3
Multimodal	DLI	†	†	†	†	0.856 [0.854, 0.858] *	0.936 [0.935, 0.938] *	0.742 [0.739, 0.745]	0.851 [0.849, 0.854] *	†	1.5 (3, 1) n=4

Note.—Data are reported as bootstrapped mean area under the receiver operating characteristic curve [95% CI]. The n-year lung cancer risk for each cohort was 2-year risk for each cohort except LI-A, which was 3-year risk, and BRONCH, which was 1-year risk.

^*result was significantly different compared to each other method in the column for p<0.01

^‡nodule characteristics unavailable (missing >10% of nodule size, attenuation, count, spiculation, or lobe location)

^¶prohibitive class imbalance (only 6/23 lung cancer cases have more than one scan)

^†Missing demographic, smoking history, COPD, or CYFRA covariates

^§No longitudinal imaging

‰ Number of cohort evaluations performed with this model

Figure 2.

(A) Mean area under the receiver operating characteristic curve (AUC) for all lung cancer prediction models applied on all study cohorts. Almost all methods demonstrate a high degree of variance in performance across cohorts within most methods, which demonstrates the importance of contextualizing a models performance by comparing it with the performance of baseline models. 95% CIs visualized in Supp. Fig 2. (B) Best- and worst-case performance for 8 predictive models reveals robust performance of longitudinal and multimodal AI methods (i.e. TDViT, DLSTM, DLS, DLI) compared to other models. A model’s worst case performance is defined as its lowest ranked performance across all cohorts except BRONCH.

A). The performance gaps were the largest between cohorts from different sites and different clinical settings (i.e. Brock on NLST-test-nodules: 0.79 [0.78, 0.80] vs. Brock on Consortium-DECAMP: 0.66 [0.66, 0.67]). The performance gap remained large between cohorts from different sites but the same clinical setting (i.e. Sybil on NLST-test: 0.88 [0.88, 0.89] vs. on LS-A: 0.78 [0.77, 0.79]). In contrast, the performance gap between different cohorts from the same site but different clinical setting was generally smaller (i.e. Liao et al. on LS-A: 0.64 [0.64, 0.65] vs. on Consortium-A: 0.66 [0.66, 0.66]).

Comparing the relative performances between multiple models across cohorts highlights the following findings.

Single chest CT AI performed well in lung cancer screening cohorts (i.e. Sybil on NLST-test: 0.88 [0.88, 0.89], Figure 3A). These models were generally competitive with linear models while longitudinal and multi-modal AI significantly outperformed linear models in every cohort. Results for Consortium-B represent this well, with Sybil (0.89 [0.88, 0.90]) performing close to Brock (0.89 [0.88, 0.89]), and DeepLungIPN (0.94 [0.94, 0.94]) outstripping the performance of both.

Figure 3.

Area under the receiver operating characteristic curves (AUCs) of 1000 bootstrapped samples from applying three selected methods across all study cohorts. Brock and Liao et al. are selected as baselines to compare with the method achieving the highest classification performance in the corresponding cohort. The best performing method differs across cohorts. Among baselines and the best performers, bootstrapped AUC distributions demonstrate high variance across cohorts. DLS seems to perform the best in cohorts with incidental nodules. Unsurprisingly, TDViT excels in the longitudinal imaging cohort (LI-A). The box and line within the box denotes the interquartile range (IQR) and median respectively. The whiskers denote 1.5*IQR and points outside the whiskers denote outliers beyond this range.

Longitudinal or multimodal AI were top performers across all cohorts with incidental nodules (Figure 3, Figure 4). They showed better worst case performances in comparison to the other approaches (Figure 2B). Ranking the results within each cohort, we define a model’s worst case as its lowest ranked performance across all cohorts except BRONCH. The worst-case performances of DLSTM (on NLST-test-nodules: 0.73 [0.72, 0.73]), TDViT (on NLST-test-nodules: 0.79 [0.79, 0.79]), DeepLungScreening (on NLST-test-nodules: 0.8 [0.77, 0.78]), and DeepLungIPN (on Consortium-DECAMP: 0.74 [0.74, 0.75]) were all moderate in terms of absolute AUC. In contrast, the worst-case performance of Mayo, Brock, Liao et al., and Sybil were low in terms of absolute AUC.

Figure 4.

The receiver operating characteristic (ROC) curves demonstrate a failure to generalize across four select cohorts. Top performers in lung screening cohorts (A) are different than the top performers in cohorts with incidentally-detected nodules (B), and vice-versa. (C) All evaluated models performed poorly on a retrospective cohort of patients selected to undergo diagnostic bronchoscopic biopsy for a pulmonary nodule. ROC curves for remaining evaluation cohorts are provided in Supp. Fig 1.

BRONCH cohort.

Models evaluated on this cohort, representing nodules that are suspicious enough to warrant a biopsy, performed poorly with mean AUCs ranging from 0.50 to 0.62 (Figure 4).

NSCLC vs SCLC (Table 5).

Table 5:

Model Classification of lung cancer risk by cancer subtype.

	NLST-test-nodules (n malignant=147, n benign=749)		Consortium-A (n malignant=238, n benign=126)
Model	SCLC (n=18)	NSCLC (n=119)	SCLC (n=39)	NSCLC (n=194)
Mayo	¶	0.810 [0.808, 0.812]	0.774 [0.772, 0.776]	0.683 [0.681, 0.685]
Brock	¶	0.792 [0.790, 0.794]	0.794 [0.792, 0.797]	0.688 [0.687, 0.690]
Liao et al.	0.683 [0.680, 0.687]	0.770 [0.768, 0.771]	0.617 [0.614, 0.620]	0.688 [0.686, 0.690]
Sybil	0.728 [0.723, 0.733] *	0.899 [0.897, 0.900] *	0.701 [0.698, 0.703]	0.701 [0.699, 0.702]
DLSTM	0.663 [0.658, 0.668]	0.808 [0.806, 0.809]	0.730 [0.726, 0.735]	0.754 [0.751, 0.757]
TDViT	0.707 [0.702, 0.711]	0.771 [0.769, 0.773]	0.667 [0.661, 0.673]	0.760 [0.757, 0.763]
DLS	0.659 [0.654, 0.664]	0.792 [0.791, 0.794]	†	†
DLI	†	†	0.904 [0.901, 0.907] *	0.853 [0.851, 0.855] *

Note.—Data are reported as bootstrapped mean area under the receiver operating characteristic curve [95% CI].

^*result was significantly different compared to each other method in the column for p<0.01

^¶prohibitive class imbalance (n=5)

^†Missing demographic, smoking history, COPD, or CYFRA covariates

AI models discriminated NSCLC cases from benign better than SCLC cases in the lung screening setting (Sybil on NLST-test for NSCLC: 0.90 [0.90, 0.90] vs. for SCLC: 0.73 [0.72, 0.73]). In both lung screening and incidental-detected nodules, longitudinal models demonstrated better performance with NSCLC cases compared with SCLC cases (TDViT on Consortium-A for NSCLC: 0.76 [0.76, 0.76] vs. for SCLC: 0.67 [0.66, 0.67]).

5. Discussion

The most prominent result was perhaps that there was no clear winner among the models evaluated. The performance of each model varied with site and clinical setting, which reflects a moderate degree of generalization failure that is often observed in both open sourced and commercial predictive models across many medical domains ²⁹. Those interested in using predictive models in lung cancer should be aware that these models, despite previous reports of successful external validation, most reliably achieve their expected performance when they are used in the same clinical context and site as they were developed in ³⁰. Those involved in model deployment should consider fine-tuning models with a cohort that matches the site, clinical setting, and year-to-outcome in which the model will be used. Steps should be taken during model development to mitigate a failure to generalize when site and setting are unmatched with techniques such as image harmonization ³¹, fine-tuning ³², and potentially directly modeling the site-specific effects ²⁹. These results motivate further investigation into the site- and context-specific factors that are driving a variance in performance and how they can be harmonized.

This study reveals the importance of interpreting a model’s performance relative to the performance of other models on the same cohort. Doing so revealed several findings that were sustained across cohorts. Single chest CT AI (Liao et al. and Sybil) performed on par with linear models that included nodule variables (Mayo and Brock). As demonstrated previously ^9,10,15, single chest CT AI is well suited for identifying individuals at risk for lung cancer who can benefit from starting or having more frequent lung imaging. Longitudinal and multimodal models demonstrated comparatively favorable performance on incidentally-detected nodules. In contrast to other models, longitudinal and multimodal AI also appeared to be more robust across cohorts, as seen from their worst-case performances.

Given that nodules in the BRONCH cohort were inherently difficult to diagnose, the poor performance on this cohort was unsurprising. Due to missing data, we were not able to evaluate longitudinal and multimodal AI on BRONCH. A predictive model that is highly specific for lung cancer in this setting has the potential to prevent invasive management of benign nodules. Therefore, evaluation of longitudinal and multimodal AI on a retrospective cohort of biopsied nodules is a high priority area for future investigation.

Longitudinal imaging models performed better on NSCLC than SCLC. One explanation for this is that NSCLC is, on average, observed more frequently as an indeterminant nodule compared to faster progressing SCLC which is often advanced stage at first observation ³³. These results warn that longitudinal imaging models may underperform on SCLC cases.

The regression calibrator improved calibration for most models evaluated on the NSLT, Consortium, and BRONCH cohorts. Calibration remained poor or became worse for models evaluated on highly imbalanced cohorts (LS-A and LI-A), which align with previous findings ³⁴.

Within the AI approaches, leveraging additional sources complementary data appears to be an effective strategy for improving classification performance. For instance, Sybil makes use of the entire chest CT whereas Liao et al. predicts on a few ROIs, a technique that crops out portions of the lung field and discards the overall chest anatomy. Additionally, using longitudinal imaging when, when available, leads to performance gains in across most of the cohorts. The integration of two or more consecutive chest CTs allows the model to consider how imaging features change over time. The use of data from multiple modalities also appears to be effective. From a clinical perspective, the advantage of multimodality is expected, as imaging findings are often interpreted in the context of the patient’s clinical risk factors. The improved performance of longitudinal AI and multimodal AI in this study suggest that combining the two approaches is a promising direction.

We note the following limitations of this study. Since the evaluation cohorts are a few years dated, we expect different numerical results on cohorts drawn from today’s practice but a similar failure to generalize across clinical context and site. Several model-cohort evaluations were not conduced due to incomplete data. Extreme class imbalance in the models’ training cohort is another confounding factor that can affect a model’s sensitivity and specificity. This is concerning for the Brock model which was trained on a cohort with a cancer prevalence much smaller than those of other models. Other confounding sources include the differences in cohort size, scanner manufacturers, and scanner protocols ^35,36. Finally, the evaluation of DeepLungIPN on its training cohort is limited because the results are from cross-validation. However, it still performed well when evaluated on a true external cohort (Consortium-B).

In summary, this study presents a comparative analysis of 8 lung cancer prediction models against 9 cohorts that represent clinical relevant use cases. Our results revealed a lack of generalized performance and that certain modeling strategies excelled in lung screening vs. incidentally-detected nodules, while all models fell short in a cohort with biopsied nodules. We highlight approaches in lung cancer predictive modeling that, if investigated further, have the potential to overcome these observed limitations.

Key Points.

• Models predicting lung cancer risk from a single time point had higher area under the receiver operating characteristic curve (AUC) for patients that underwent lung cancer screening but performed worse on patients with incidentally-detected nodules relative to other models.

• Longitudinal models and multimodal models had comparatively high AUCs for patients with incidentally-detected nodules, but showed lower performance than single time point models in lung cancer screening cohorts.

• Both clinical variable-based models and artificial intelligence-based models performed poorly on patients with pulmonary nodules that were triaged for invasive biopsies.