Novel tools for early diagnosis and precision treatment based on artificial intelligence

Abstract

Lung cancer has the highest mortality rate among all cancers in the world. Hence, early diagnosis and personalized treatment plans are crucial to improving its 5-year survival rate. Chest computed tomography (CT) serves as an essential tool for lung cancer screening, and pathology images are the gold standard for lung cancer diagnosis. However, medical image evaluation relies on manual labor and suffers from missed diagnosis or misdiagnosis, and physician heterogeneity. The rapid development of artificial intelligence (AI) has brought a whole novel opportunity for medical task processing, demonstrating the potential for clinical application in lung cancer diagnosis and treatment. AI technologies, including machine learning and deep learning, have been deployed extensively for lung nodule detection, benign and malignant classification, and subtype identification based on CT images. Furthermore, AI plays a role in the non-invasive prediction of genetic mutations and molecular status to provide the optimal treatment regimen, and applies to the assessment of therapeutic efficacy and prognosis of lung cancer patients, enabling precision medicine to become a reality. Meanwhile, histology-based AI models assist pathologists in typing, molecular characterization, and prognosis prediction to enhance the efficiency of diagnosis and treatment. However, the leap to extensive clinical application still faces various challenges, such as data sharing, standardized label acquisition, clinical application regulation, and multimodal integration. Nevertheless, AI holds promising potential in the field of lung cancer to improve cancer care.

Introduction

Lung cancer remains the leading cause of all cancer deaths (18.0%) around the world.¹ Although the treatment paradigm for lung cancer has changed dramatically, its overall 5-year survival rate is below 20%, which is considerably lower than that of breast cancer and cervical cancer.² The primary reason is that patients with lung cancer usually have non-specific symptoms at an early stage, resulting in 68% of lung cancer patients being diagnosed at an advanced stage.³ Late-stage tumors pose a significant public health burden.⁴ Early and accurate diagnosis is the key to improving the prevention and treatment effect of lung cancer.

Radiological images are powerful tools for pulmonary nodules screening, diagnosis, and monitoring of lung cancer. Low-dose computed tomography (LDCT) has been demonstrated to reduce lung cancer mortality by 20% and is recommended for lung cancer screening. Its sensitivity in detecting lung nodules is excellent; however, the specificity is limited, with 96% of the nodules being false-positive.^{5,6 18}F-fluoro-d-glucose (FDG)-positron emission tomography (PET)/computed tomography (CT) is a functional imaging tool developed based on CT scans, which can accurately reflect the glucose metabolism of tumors.^7,8 However, large heterogeneity exists in the interpretation of imaging by different physicians, making it impossible to determine the characteristics of the lesions. Currently, pathological confirmation remains the gold standard for tumor diagnosis. The interpretation of pathological slides is a time-consuming and labor-intensive task that relies on the experience of the pathologists. There is an urgent need for imaging intelligence tools with high automation, diagnostic efficiency, and accuracy to assist in the precise management of pulmonary nodules.

Over the past few years, the resurgence of artificial intelligence (AI) has revealed a promising potential to assist with complex medical tasks.^9,10 For instance, the deep learning system based on CT or chest X-ray images to diagnose coronavirus disease 2019 (COVID-19) pneumonia and deep convolutional neural networks (CNNs) to assess tumor origin utilizing whole-slide images (WSIs) have achieved superior performance.11, 12, 13, 14 AI is expected to reshape precision oncology care and improve the experience of clinicians.¹⁵ In terms of lung cancer, AI is also utilized to detect lung nodules, predict genetic status, and distinguish pathological subtypes to facilitate precision medicine.16, 17, 18 These novel models are able to extract high-throughput features in radiologic images and digital pathology to achieve comparable results to senior doctors and improve the diagnostic efficiency of physicians.^19,20 Nevertheless, the clinical usability of AI models still requires improvement.

This study is aimed to promote the application of the latest AI technology in the field of lung cancer screening, diagnosis, treatment, and prognosis. Of particular interest is the integration of basic, translational, and patient-oriented research.

Emerging AI methods

AI technologies greatly impact almost all aspects of oncology, from cancer detection and classification to prediction of patient outcomes by integrating and synthesizing data on clinical presentations, patient history, medical imaging, tumor pathology, and genomics to predict the pathway of patient care and ultimately improve management decisions.²¹ The first computer-aided diagnosis (CAD) system was experimented for lung cancer to convert X-rays into a quantitative form for computer analysis.²² Although this exploration did not lead to the detection or classification of lung nodules, it opened the path for computer-assisted lung cancer diagnosis by converting medical images into quantitative features. Since then, plenty of theoretical and empirical research work has been conducted in computer-aided lung cancer detection, diagnosis, and prediction (Fig. 1). To date, several AI methods have been employed to detect and classify lung nodules.

Fig. 1.

Milestone of AI in lung cancer. LDCT: Low-dose computed tomography; NLST: National Lung Screening Trial; NELSON: Nederlands...Leuvens Longkanker Screenings Onderzoek; FDA: Food and Drug Administration; LUNA16: Lung Nodule Analysis 16.

Machine learning extracts pre-defined features of a lesion. For example, the radiomics features of lung nodules based on CT or FDG-PET images include textural heterogeneity features, intensity-based measures, shape and volumetric features, etc.^23,24 Moreover, statistical models such as random forest classifiers are used for modeling and analysis. Radiomics and pathomics are relatively mature methods that require fine layer-by-layer annotation by physicians, and the heterogeneity of manual work might cause differences in features and affect the generality of the model.

Deep learning is a subfield of machine learning that includes common neural networks such as deep neural networks (DNN), recurrent neural network (RNN), and CNN.^25,26 DNN has a simple structure, each layer is connected to the neurons of the previous layer, and the signal propagates unidirectionally from the input layer to the output layer. CNN utilizes a convolutional kernel to transform imaging data into deep features through convolutional computation of the feedforward neural network layers, resulting in a vector after pooling layers. This vector is then extended through various techniques, including fully CNNs and fully connected networks. It can directly take image data as input, eliminating the need for complex operations such as manual image pre-processing and additional feature extraction. Additionally, its fine-grained feature extraction method enables the processing of images to approach a nearly human level of accuracy. More recently, advanced deep learning methods have emerged to continuously improve the efficiency of model operations such as the graph CNN and transformer pipeline.²⁷

The supervised learning model training process usually requires collecting a large amount of labeled image data, which is a labor-intensive process. However, semi-supervised learning and augmented learning models alleviate the reliance on manual labeling.²⁸ The deep learning training process is not transparent and thus hinders clinical applications to a certain extent. In addition, there is complementarity between different dimensions of information such as imaging, clinical indicators, and molecules.²⁹ The performance and mechanism of multimodal fusion models need to be further investigated.

AI application in CT imaging for diagnosis

CT imaging plays a significant role in the initial detection of pulmonary nodules and diagnosis of lung cancer (Table 1).^16,30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46 The critical tasks include detecting nodules, predicting the malignancy risk, and classifying the subtypes for precision therapy (Fig. 2).

Table 1.

Representative research of AI for lung cancer diagnosis based on CT images.

Application	Authors	Year	Dataset/sample size	Imaging modality	Algorithm	Task	Performance
Nodule detection	Gu et al. 30	2018	1186 nodules in the LUNA16 database	CT	3D deep CNN combined with a multi-scale prediction strategy	Automatic lung nodule detection	Sensitivity: 0.9293 on testing dataset
	Xu et al. 31	2019	1590 nodules	CT	Multiple neural network models	Automatic lung nodule detection	Sensitivity: 0.9695 and 0.9117 for ThinSet and ThickSet on testing dataset
Malignancy evaluation	Ardila et al. 16	2019	6716 patients from NLST and 1139 cases for validation	CT	End-to-end three-dimensional deep learning	Malignancy prediction and risk bucket score of lung nodules	AUC: 0.944 on held-out NLST testing set
	Xu et al.32	2019	548 nodules from LIDC-IDRI dataset	CT	MSCS neural networks – DeepLN	Malignancy evaluation of lung nodules	AUC: 0.94 on testing dataset
	Massion et al.33	2020	14,761 benign nodules and 932 malignant nodules	CT	LCP-CNN	Risk stratification of lung nodules	AUC: 0.921 on internal validation dataset
	Baldwin et al.34	2020	1397 nodules (5–15 mm)	CT	LCP-CNN	Malignancy evaluation of lung nodules	AUC: 0.896 on external validation dataset
	Venkadesh et al.35	2021	16,077 nodules from NLST	CT	Deep learning	Malignancy risk estimation of lung nodules	AUC: 0.93 on external validation dataset
	Shi et al.36	2021	3038 nodules	CT	Semi-supervised Deep Transfer Learning	Benign–malignant lung nodule diagnosis	AUC: 0.795 on independent testing dataset
	Park et al.37	2021	359 patients	CT and FDG PET/CT	Deep learning classification models based on ResNet-18	Malignant lung nodule diagnosis	AUC: 0.837 on five-fold cross-validation
	Shao et al.38	2022	12,360 participants	CT	Deep learning	Detection and risk stratification of lung nodules	AUC: 0.8516 on testing dataset
Subtype classification	Wu et al.39	2016	350 patients	CT	Radiomics	Histologic subtypes (LUAD vs. LUSC) prediction	The highest AUC: 0.72 on validation dataset
	Zhao et al.40	2018	651 nodules	CT	Dense Sharp Network	Tumor invasiveness prediction	ACC: 0.641 on testing dataset
	Hyun et al.41	2019	396 patients	18F-FDG PET/CT	4 clinical features (age, sex, tumor size, and smoking status) and 40 radiomic features	Histologic subtypes (LUAD vs. LUSC) prediction	AUC: 0.859 on testing dataset
	Han et al.42	2021	867 LUAD and 552 LUSC patients	PET/CT	10 feature selection techniques, 10 machine learning models, and the VGG16 deep learning algorithm	Histologic subtypes (LUAD vs. LUSC) prediction	The highest AUC: 0.903 on testing dataset
	Ren et al.43	2021	315 NSCLC patients	PET/CT	Clinico-biological features and FDG-PET/CT radiomic-based nomogram via machine learning	Histologic subtypes (LUAD vs. LUSC) prediction	AUC: 0.901 on validation dataset
	Wang et al.44	2021	1222 patients with LUAD	CT	Deep learning and radiomics	Adenocarcinoma subtype classification	AUC: 0.739–0.940 on internal validation dataset
	Choi et al.45	2021	817 patients with clinical stage I LUAD	CT	Deep learning	Prediction of visceral pleural invasion	AUC: 0.75 on temporal validation dataset
	Zhong et al.46	2022	3096 patients	CT	Deep learning	N2 metastasis prediction and prognosis stratification	AUC: 0.81 on prospective testing dataset

AI: Artificial intelligence; ACC: Accuracy; AUC: Area under the curve; CNN: Convolutional neural network; ¹⁸F-FDG PET/CT: Fluorine-18-fluorodeoxyglucose (FDG) positron emission tomography (PET) computed tomography (CT); FROC: Free-response receiver operating characteristic; IPNs: Indeterminate pulmonary nodules; LCP-CNN: Lung Cancer Prediction-Convolutional Neural Network; LIDC: Lung Image Database Consortium; IDRI: Image Database Resource Initiative; LUAD: Lung adenocarcinoma; LUNA16: Lung Nodule Analysis 16; LUSC: Lung squamous cell carcinoma; MSCS: Multi-scale cost-sensitive; NLST: National Lung Screening Trial; NSCLC: Non-small cell lung cancer.

Fig. 2.

Clinical process of CT-based AI models applied to lung cancer diagnosis and treatment. AI: Artificial intelligence; CT: Computed tomography.

Lesion detection

Detection of lesions is time consuming and tedious and the effectiveness of AI algorithms in lung nodule detection has been adequately documented.⁴⁷ In 2017, the Lung Nodule Analysis 16 (LUNA16) challenge solicited nodule detection and classification algorithms from around the world. To provide a fair comparison between various nodule detection and classification algorithms, the challenge used the largest publicly available Lung Image Database Consortium (LIDC)–Image Database Resource Initiative (IDRI) dataset. The leading CAD system named Combined LUNA16 used CNN networks with a sensitivity of 96.9%.⁴⁸ This system even updated the LIDC–IDRI reference standard by identifying nodules missed in the original LIDC–IDRI annotations.

Later, various deep learning models emerged. Lung nodule detection using a 3D deep CNN combined with a multiscale prediction strategy achieved a sensitivity of 0.9293 at per scan.³⁰ In order to apply to CT images of different layer thicknesses, an innovative multi-resolution CT lung nodule detection system was built with a sensitivity of 96.95% and 91.17% for lung nodules in thin and thick CT images, respectively.³¹ To date, several lung nodule detection products have been approved by the Food and Drug Administration (FDA) for practical use in clinical settings to enhance physician efficiency.²¹

Malignancy evaluation

The powerful combination of LDCT and AI has led to a significant increase in the efficiency of lung nodule detection. To reduce unnecessary invasive procedures, it is crucial to determine the malignancy of the nodules, although some non-invasive stratification models have been proposed, such as the lung imaging reporting and data system (Lung-RADS), PanCan model, and Mayo Clinic models, which rely on manual visual extraction of data on qualitative characteristics of nodules including nodule diameter, spiculation, and location with inter-grader variability.⁶ Computer Aided Nodule Assessment and Risk Yield (CANARY) is an image analysis software that has only been used in lung adenocarcinoma (LUAD) patients.⁴⁹

The Google AI team pioneered the CT-based end-to-end deep learning model for predicting the malignancy probability of pulmonary nodules from the National Lung Screening Trial (NLST) cohort with an area under the curve (AUC) of 0.944, increasing sensitivity by 0.052 and specificity by 0.116 compared to specialists.¹⁶ Lung Cancer Prediction-Convolutional Neural Network (LCP-CNN) focused on CT images of indeterminate pulmonary nodules (IPNs) from the NLST dataset and achieved a significantly improved AUC (0.921 on internal validation cohort) over the clinically validated risk models (Brock and Mayo) on risk stratification.³³ It further predicted malignant nodules with superior discriminatory properties and lower false negatives than the Brock model (AUC: 0.896 vs. 0.868) on the external validation dataset of 5–15 mm lung nodules.³⁴ European researchers developed deep learning algorithms for malignancy risk estimation of lung nodules detected by LDCT, and the algorithms were generalizable across different screening populations and protocols with algorithm performance comparable to that of chest radiologists.³⁵ The AI-aided diagnosis system for lung cancer screening based on mobile CT is successfully applied to community cohorts in resource-limited regions of western China. It realized automatic lung nodule localization, risk stratification (AUC: 0.8516), and consultation recommendations in a lung cancer screening cohort.³⁸ Recently, a deep learning model named Sybil was established to predict the risk of lung cancer occurring 1–6 years after the screening.⁵⁰ Surprisingly, it achieved decent performance from a single LDCT scan (the highest concordance index [C-index]: 0.81).

Several challenges occur during the development of these systems, such as unbalanced samples. Our team innovated Multi-Scale Cost-Sensitive Neural Networks (MSCS-Net) and used a three-dimensional CNN as the backbone of a lightweight model to extract pulmonary nodule features from CT images.³² The clipped multi-scale input could realize a sub-network of learning multi-level context features, and solve the problem of clinical small samples and class imbalance. Moreover, the semi-supervised deep transfer learning (SDTL) framework could be used for benign–malignant lung nodule diagnosis.³⁶ Specifically, a similarity metric function is employed in the network semantic representation space to progressively include a small group of samples without pathological findings to iteratively optimize the classification network to address the highly imbalanced distribution of benign and malignant nodules. Transfer learning (TL) and metadata evaluation introduced metadata of maximum standardized uptake value (SUVmax) and lesion size from PET/CT into the baseline CT model, improving the CT diagnostic performance of both the PET/CT model (AUC: 0.837 vs. 0.762) and the conventional CT model (AUC: 0.877 vs. 0.817).³⁷

Algorithm innovations require to be validated for practicality in clinical application.⁵¹ A large number of emerging AI models such as the DeepLN system simultaneously realize the detection and classification of pulmonary nodules.^52,53 To increase the explanatory capacity of the models, the prediction of common nodule features has been added to improve the homogeneity of the physicians and has been validated in different large-scale datasets.⁵⁴

Subtype classification

For nodules that have been diagnosed as malignant, a refined diagnostic classification is required to clarify the pathological subtype and stage for appropriate treatment. Various histologic subtypes such as LUAD and lung squamous cell carcinoma (LUSC) exhibit different growth and treatment patterns.³⁹ An exploratory study revealed that 53 out of 440 imaging features were associated with tumor pathology classification (LUAD and LUSC). After dimensionality reduction, the naïve Baye's classifier constructed with five features achieved the best performance in differentiation. Whereas based on clinical features and radiomic features from PET/CT images, the logistic regression model outperformed other machine learning classifiers in distinguishing LUAD from LUSC.⁴¹ In another study, deep learning networks outscored conventional machine learning approaches in terms of classification task performance.⁴² Furthermore, the model performance is enhanced by incorporating clinical and other information. An integrated clinico-biologico-radiological model, consisting of two clinical factors, two tumor markers, seven PET radiomics, and three CT radiomic parameters, yielded an AUC of 0.901 (95% confidence interval [CI]: 0.840–0.957) in the validation set for discriminating between LUAD and LUSC.⁴³ According to the latest WHO classification, LUAD is classified into adenocarcinoma in situ (AIS), minimally invasive adenocarcinoma (MIA), and invasive adenocarcinoma (IAC). Deep learning has made it possible to identify LUAD subtypes without invasive surgery and biopsy.^40,44 Several studies have demonstrated that radiomic features are closely related to histopathologic profiles of lung cancer and even surrogated for pathological types including the aggressive adenocarcinoma subtypes, namely visceral pleural, lymphatic, venous, and perineural invasion.^45,55,56

Tumor-Nodes-Metastasis (TNM) staging, the most extensively approved grading system, is essential for the diagnosis of lung cancer.⁵⁷ The pathological stage (early vs. late) of lung cancer patients has been accurately predicted (average precision score: 0.84 in LUAD patients) based on CT image features.⁵⁸ The evaluation of lymph node (LN) metastases requires complete LN dissection, which might cause additional trauma. Especially for stage I non-small cell lung cancer (NSCLC), deep learning signatures assist in accurately predicting N2 status with stable performance in prospective cohorts (AUC: 0.81).⁵⁹ Moreover, radiogenomics initially explored its biological basis, suggesting that deep learning features capturing N2 metastasis risk might be associated with epidermal growth factor receptor (EGFR) mutations, anaplastic lymphoma kinase (ALK) fusions, and potential metabolic pathways associated with cell proliferation and tumor progression.⁴⁶

AI application in CT imaging for precision treatment

AI is able not only to predict the microscopic morphology of tumor lesions but also to assess the gene mutation and molecular expression of patients, and predict prognosis, thereby providing personalized treatment advice to patients with lung cancer (Table 2).^17,60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81

Table 2.

Representative research of AI for lung cancer treatment based on CT images.

Application	Author	Year	Dataset/sample size	Imaging modality	Algorithm	Task	Performance
Gene mutation and molecular expression prediction	Wang et al.60	2019	844 LUADs	CT	Deep learning	EGFR mutation status prediction	AUC: 0.81 on independent validation dataset
	Mu et al.61	2020	681 NSCLC patients	PET/CT	Deep learning	Quantification of EGFR mutation status	AUC: 0.81 on external testing dataset
	Tian et al.62	2020	939 consecutive stage IIIB–IV NSCLC patients	CT	Deep CNN	Assessment of PD-L1 expression and ICI responses	AUC: 0.76 on testing dataset, C-index: 0.66
	Song et al.63	2021	937 patients	CT	Deep learning model and clinicopathological information	ALK fusion status prediction	AUC: 0.8481 on validation dataset
	Rossi et al.64	2021	109 patients	CT	Radiomics and SVM model	Detection of EGFR mutations	AUC: 0.85 on five-fold cross validation
	Zhang et al.65	2021	134 patients	CT	1672 radiomic features	Simultaneous identification of EGFR, KRAS, ERBB2, and TP53 mutations	AUCs: 0.78–0.87 on five-fold cross validation
	Wang et al.17	2022	18,232 patients	CT	FAIS	Prediction of EGFR genotype and targeted therapy response	AUCs: 0.748–0.813 on six testing datasets
	Wang et al.66	2022	3816 patients	CT	Multitask AI system	EGFR and PD-L1 status prediction	AUCs: 0.928 for EGFR mutated status, 0.905 for PD-L1 expression on testing dataset
	Shao et al.67	2022	1096 Patients	CT	MMDL system	Identification of multiple actionable mutations and PD-L1 expression	AUCs: 0.862 for 8 mutated genes, 0.856 for 10 molecular statuses on testing dataset
Treatment efficacy assessment	Song et al.68	2018	117 stage IV EGFR-mutant NSCLC patients	CT	CT-based phenotypic characteristics	Prediction of PFS with EGFR-TKI therapy	C-index: 0.718, 0.720 on two validation datasets
	Xu et al.69	2019	179 patients with stage III NSCLC treated with definitive chemoradiation	CT	Transfer learning of CNN with RNN using single seed-point tumor localization	Prediction of OS	AUC: 0.74 for 2-year OS on validation dataset
	Lou et al.70	2019	944 patients	CT	Deep learning	Predict treatment failure and hence guide the individualization of radiotherapy dose	C-index: 0.77 on independent validation dataset
	Khorrami et al.71	2020	139 patients	CT	Compared changes in the radiomic texture (DelRADx) of CT patterns both within and outside tumor	Prediction of OS and response to ICIs	AUCs: 0.81, 0.85 on two independent validation datasets
	Dercle et al.72	2020	Nivolumab, 92; docetaxel, 50; and gefitinib, 46	CT	Radiomics	Prediction of systemic cancer therapies response	AUC: 0.77 for nivolumab, 0.67 for docetaxel, and 0.82 for gefitinib on validation dataset
	Mu et al.73	2020	194 patients with stage IIIB–IV NSCLC	PET/CT	Radiomics	Prediction of ICIs benefit	AUC: 0.81 on prospective testing dataset
	He et al.74	2020	327 patients	CT images	Deep learning radiomics	Prediction of ICIs response	AUC: 0.81 on testing dataset
	Dissaux et al.75	2020	27, 29, and 8 patients treated with SBRT from three different centers	18F-FDG PET/CT	Radiomics	Prediction of local recurrence	AUC: 0.905 on testing dataset
	Deng et al.76	2022	570 patients with stage IV EGFR-mutant NSCLC treated with EGFR-TKIs and 129 patients with stage IV NSCLC treated with ICIs	CT	EfficientNetV2-based survival benefit prognosis (ESBP)	Survival benefit prediction of TKIs and ICIs	C-index: 0.690 on the EGFR-TKI external testing dataset
Survival prognosis prediction	Aerts et al.77	2014	1019 patients with lung or head-and-neck cancer	CT	440 features quantifying tumor image intensity, shape, and texture	Prediction of OS	C-index: 0.65 on validation dataset
	Hosny et al.78	2018	1194 NSCLC patients	CT	3D CNN	Mortality risk stratification	AUCs: 0.72 for 2-year OS from the start of respective treatment for radiotherapy and 0.71 for surgery on external validation dataset
	Arshad et al.79	2019	358 stage I–III NSCLC patients	FDG-PET/CT	Radiomics	Prediction of OS	C-index: 0.541–0.558 on testing dataset
	Jazieh et al.80	2022	133 patients with unresectable stage III NSCLC	CT	Radiomics risk score	Prediction of clinical outcomes	C-index: 0.77 for PFS, 0.77 for OS on testing dataset
	Huang et al.81	2022	1168 nodules of 965 patients	FDG-PET/CT scans	CNN	Prediction of OS	C-index: 0.737 for PET + CT + clinical ensemble model on testing dataset

AI: Artificial intelligence; ALK: Anaplastic lymphoma kinase; AUC: Area under the curve; CNN: Convolutional neural network; CT: Computed tomography; DFS: Disease-free survival; EGFR: Epidermal growth factor receptor; FAIS: Fully automated artificial intelligence system;¹⁸F-FDG PET/CT: Fluorine-18-fluorodeoxyglucose (FDG) positron emission tomography (PET) computed tomography (CT); ICI: Immune checkpoint inhibitors; LUAD: Lung adenocarcinoma; MMDL: Multi-label multi-task deep learning; NSCLC: Non-small cell lung cancer; OS: Overall survival; PD-L1: Programed death ligand-1; PFS: Progression-free survival; RNN: Recurrent neural networks; SBRT: Stereotactic body radiotherapy; SVM: Support vector machine; TKIs: Tyrosine kinase inhibitors; TMB: Tumor mutational burden.

Gene mutation and molecular expression prediction

With the development of targeted therapy and immunotherapy, the treatment of lung cancer has proceeded in the era of precision medicine. The identification of gene mutations and molecular expression status is an essential step in determining therapeutic options. However, existing assays such as polymerase chain reaction (PCR) and next-generation sequencing (NGS) for genetic testing, and immunohistochemistry (IHC) for molecular testing, are relatively expensive and challenging based on invasively acquired tissues to perform. However, the introduction of radiogenomics aids in analyzing the link between microscopic molecules and macroscopic imaging features to predict molecular states non-invasively.²⁶

EGFR is the most intensively studied gene in lung cancer radiogenomics, and EGFR-tyrosine kinase inhibitors (TKIs) are recommended for patients with positive mutations. A deep learning model based on CT images of 844 LUAD patients successfully achieved the prediction of EGFR mutation status with better performance (AUC: 0.81 in the independent validation cohort) than hand-crafted CT characteristics and clinical features prediction models.⁶⁰ Furthermore, the deep learning model built with ¹⁸F-FDG-PET/CT images accurately predicted EGFR mutation status with AUCs of 0.83 and 0.81 in the internal validation and external test cohorts, respectively.⁶¹ The predictive EGFR genotype performance of a fully automated artificial intelligence system (FAIS) was validated in six testing sets that mined whole-lung information from CT images to take complete advantage of peri-lesion changes.¹⁷ Radiomics identified 17 different radiological features in baseline CT scans associated with the subsequent development of T790M during EGFR inhibitor therapy.⁶⁴ However, due to the insufficient sample size, this finding required more extensive validation. Similarly, the non-invasive prediction of ALK fusion status has been investigated. After adding clinicopathological information, the AUC of the prediction model improved from 0.7754 to 0.8481 in the validation cohort.⁶³ In addition, the programed death ligand-1 (PD-L1) expression levels were graded as indicators of immunotherapy using immunohistology.⁶² Deep learning algorithms non-invasively measured the rank (PD-L1 expression signature <1%, 1–49%, and ≥50%) and explored the regions of interest in the network to assist in the clinical decisions of physicians.⁸²

Based on the single-gene status prediction, the demand for multi-molecular status assessment has become more urgent. As the most critical gene for precision therapy, EGFR, and the molecule PD-L1, several deep learning models that simultaneously predict both molecules were validated.^66,83 In multi-task AI systems, the joint module combining deep learning, radiomics, and clinical features predicted molecular states optimally. Four key genes of lung cancer (EGFR, KRAS, ERBB2, and TP53) were also successfully identified through machine learning with AUCs of 0.78 (95% CI: 0.70–0.86), 0.81 (95% CI: 0.69–0.93), 0.87 (95% CI: 0.78–0.95), and 0.84 (95% CI: 0.78–0.91), respectively.⁶⁵ Furthermore, a multi-label multi-task deep learning (MMDL) system achieved the simultaneous prediction of eight treatment-related genes recommended by the National Comprehensive Cancer Network (NCCN). After adding TP53 and PD-L1, this model achieved 10 molecular status prediction simultaneously.⁶⁷ These radiogenomic models have the potential to be an ancillary tool in conjunction with adjunctive testing to support precision treatment options.⁸⁴

Treatment efficacy assessment

Standard therapies for lung cancer include surgery, chemotherapy, radiotherapy, targeted therapy, and immunotherapy. AI techniques are extensively used to screen the beneficiary population and predict clinical outcomes.85, 86, 87 The prognosis of surgical patients varies greatly, and AI provides a novel way of prognostic assessment. The Cox model based on preoperative PET/CT signatures and clinical characteristics can effectively predict the disease-free survival (DFS) of NSCLC patients undergoing surgery.

Stereotactic body radiation therapy (SBRT) is the standard treatment for inoperable patients, and AI assisted physicians in accomplishing faster, finer, and more consistent tumor segmentation for radiation therapy.^88,89 However, there is a possibility of distant failure or local recurrence in radiotherapy. With PET and CT, a kerneled support tensor machine (KSTM)-based model was proposed to predict the distant failure in NSCLC treated with SBRT.⁹⁰ In another study, two PET features were confirmed to be associated with local recurrence in patients with lung cancer who received SBRT.⁷⁵ In patients treated with radiotherapy, three image features, namely the tumor volume, the maximum distance between involved nodes at baseline, and the change in tumor-to-involved nodes distance, were closely associated with PFS. However, the sample sizes of these studies were relatively limited (87 NSCLC patients from four centers, 110 NSCLC patients, and 82 patients with stage III NSCLC, respectively).75, 90, 91 To achieve individualized precision treatment, the Deep Profiler model was built on a retrospective dataset of 944 patients to predict time-to-event treatment failures (C-index of 0.77 on independent validation dataset).⁷⁰ In addition, a personalized radiotherapy dose index iGray incorporating Deep Profiler and clinical variables was recommended to estimate the probability of treatment failure below 5%.⁷⁰ In general, these AI technologies have accelerated the development of refinement of radiotherapy and chemotherapy regimens.

Targeted therapies and immunotherapies have revolutionized the treatment paradigm of lung cancer, but only a few patients have benefited from them. The image features mined through AI algorithms have the potential to become markers for classifying high-benefit patients. For instance, the CT radiomic model enables the prediction of PFS for EGFR-TKI therapy in NSCLC (C-index: 0.718 and 0.720 for two validation cohorts, respectively), improving management of TKI.⁶⁸ Similarly, PET/CT-based biomarkers are available to predict the durable clinical benefit (DCB) in NSCLC patients treated with checkpoint blockade immunotherapy (AUC: 0.81, 95% CI 0.68–0.92 in prospective test cohort).⁷³ Specifically, the radiomic risk score remained significantly associated with PFS for patients treated with durvalumab after chemotherapy (C-index: 0.77).⁸⁰ Moreover, deep learning features were developed to distinguish between high tumor mutational burden (TMB) and low-TMB patients and subsequently help the TMB radiomic biomarker to successfully evaluate the effectiveness of the immunotherapy (OS; hazard rate [HR]: 0.54, 95% CI: 0.31–0.95; P = 0.03; PFS; HR: 1.78, 95% CI: 1.07–2.95; P = 0.02).⁷⁴ Then an EfficientNetV2-based survival benefit prognosis (ESBP) system as a prognostic tool for EGFR-TKI and ICI therapies improved the performance of primary radiologists and oncologists to the expert level along with the potential to improve treatment-related labor and cost efficiency.⁷⁶

The non-invasive monitoring of lesion changes in the imaging images allows for a more accurate assessment of treatment outcomes during multiple follow-ups.^69,72 Machine learning could quantify the changes in the tumor nodules during immunotherapy, which distinguishes between responders with AUCs over 0.8.⁷¹ Overall, AI models have developed multiple approaches to identify appropriate tailored therapies for lung cancer patients.

Survival prognosis prediction

The prognosis of lung cancer patients is influenced by several complex factors, and imaging markers have become non-invasive biomarkers. Initially, radiomics extracted quantitative imaging features from CT images of lung cancer patients to predict prognosis with a C-index of 0.65.⁷⁷ Subsequently, the deep learning networks have extracted prognostic signatures from the CT images of patients treated with radiotherapy. Therefore, a transfer learning approach was applied to achieve the same prognosis prediction for surgery patients.⁷⁸ These models significantly divided the patients into low-risk and high-risk groups of mortality. Activation mapping indicated that areas both inside and outside the tumor in CT images contributed to the prognostic features highlighting the importance of peritumor tissue in patient stratification.

The combination of FDG-PET/CT provides anatomical and metabolic information about the lesion that is essential to establish an accurate prognosis and guide curative options.⁷⁹ Characteristics such as tumor volume were confirmed to correlate with prognosis. The random survival forests that ensembled the clinical factors and deep learning features of PET and CT images were constructed to predict the OS (C-index: 0.737).⁸¹ In future studies, using prospective validation will help the model provide recommendations for lung cancer treatment options and improve patient care.

AI application in histopathology images

Digitized WSIs have shifted from traditional histopathology to computational methods, giving rise to a broad range of applications for AI methods in histopathological analysis (Table 3).^18,27,92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103 These novel tools are used to assist pathologists in the accurate diagnosis of lung cancer, segmentation of different cell types, molecular expression assessment, and prognostic prediction (Fig. 3).

Table 3.

Representative research of AI for lung cancer treatment based on pathologic images.

Application	Author	Year	Sample size of WSIs	Algorithm	Task	Performance
Classification	Coudray et al.18	2018	1634	Deep CNN (Inception-V3)	Classification of LUAD, LUSC, or normal lung tissue and prediction of 10 most commonly mutated genes in LUAD	AUCs: 0.97 for classification, 0.733–0.856 for genomic prediction on testing dataset
	Khosravi et al.92	2018	12,139	CNN Inception-V1 and Inception-V3	Discrimination of LUAD and LUSC	AUC: 0.92 on testing dataset
	Wang et al.93	2020	939	Weakly Supervised Deep Learning	Classification of carcinoma types	Accuracy: 0.973 on testing dataset
	Yang et al. 94	2021	741, 318, and 212 from two hospitals, and 422 from TCGA	EfficientNet-B5- and ResNet-50-based deep learning methods	Classification of LUAD, LUSC, SCLC, pulmonary tuberculosis, organizing pneumonia, and normal lung tissue	AUCs: 0.918–0.978 on testing dataset
	Chen et al.95	2021	9662	Deep learning	Classification of lung cancer types	AUCs: 0.9594 for LUAD, and 0.9414 for LUSC on testing dataset
	Zheng et al.96	2021	4818	Graph-transformer	Classification of LUAD, LUSC, or normal lung tissue	Accuracy: 0.912 on five-fold cross-validation
	Wang et al.97	2019	1337 from TCGA, 345 from NLST, 102 from CHCAMS, and 130 from SPORE	CNN	Transformation of pathological images	Accuracy: 0.901 on independent testing dataset
Enhancement of Oncology Care	Yu et al.98	2016	2186 from TCGA, and 294 from TMA	Machine learning	Prognosis prediction	AUC: 0.81 on testing dataset
	Fu et al.99	2020	17,355 histopathology slide images from 28 cancer types	Inception-V4 deep learning and transfer learning	Classification of cancer types	Average AUC: 0.99 for 14 cancers on held-back validation dataset
	Kapil et al.100	2021	151	Deep learning	Assessment of PD-L1 expression and survival analysis	C-index: 0.93 on testing dataset
	Qaiser et al.101	2022	1122	Weakly supervised CNN	Prediction of disease outcome	C-index: 0.7033 on testing dataset
	Choi et al.102	2022	802	Deep learning	Assessment of PD-L1 expression	C-index: 0.902 for pathologists with AI assistance
	Lee et al.27	2022	3950 patients with kidney, breast, lung, and uterine cancers	Graph DNN	Prognosis prediction	C-index: 0.731 on NLST dataset, 0.709 on TCGA dataset
	Chen et al.103	2022	6592 gigapixel WSIs from 5720 patient samples across 14 cancer types from the TCGA	AMIL network for processing WSIs, SNN for processing molecular data features, and deep-learning-based MMF	Prognosis prediction	C-index: 0.578 for AMIL, 0.606 for SNN, and 0.644 for MMF on five-fold cross-validation

AI: Artificial intelligence; AMIL: Attention-based multiple-instance learning; AUC: Area under the curve; CHCAMS: Chinese Academy of Medical Sciences; CNNs: Convolutional neural networks; DNN: Deep neural networks; LSCC: Lung squamous cell carcinoma; LUAD: Lung adenocarcinoma; LUSC: Lung squamous cell carcinoma; MMF: Multimodal fusion; NLST: National Lung Screening Trial; NSCLC: Non-small cell lung cancer; PD-L1: Programed death ligand-1; SCLC: Small cell lung cancer; SNN: Self-normalizing network; SPORE: Specialized Programs of Research Excellence; TCGA: The Cancer Genome Atlas; TMA: Stanford tissue microarray; WSIs: Whole slide images.

Fig. 3.

Application of histology-based AI models to assist in lung cancer classification and prognosis. AI: Artificial intelligence; H & E: Hematoxylin & eosin; LUAD: Lung adenocarcinoma; LUSC: Lung squamous cell carcinoma.

Classification of pathological types

LUAD and LUSC differ in origin with distinct pathological features and treatment modalities.¹⁰⁴ The classification process typically requires the visual judgment of pathologists and AI models have become a powerful aid. A deep CNN based on Inception-V3 was proposed to classify WSIs into LUAD, LUSC, and normal tissues with an AUC of 0.97 achieving sensitivity and specificity close to that of a pathologist.¹⁸ Meanwhile, a DNN was innovatively presented to predict the 10 genes’ mutation in adenocarcinoma, and eventually six genes were successfully determined. This CNN architecture has been able to discriminate various cancer tissues, two subtypes of lung cancer (LUAD vs. LUSC), biomarkers of bladder cancer, and scores of breast cancer with accuracy of 1, 0.92, 0.95, and 0.69, respectively.⁹² To fulfill the requirement for accurate lung cancer diagnosis, a six-type classifier for histopathological classification of LUAD, LUSC, small cell lung carcinoma (SCLC), pulmonary tuberculosis, organizing pneumonia, and normal tissues had a correlation coefficient of 0.873 with the pathologists.⁹⁴

These aforementioned models must be validated for generalizability in large-scale datasets. The annotation of WSIs remains a critical step in building the dataset. Weakly supervised learning and annotation-free whole-slide training approach solved this difficulty to some extent.^93,95 These methods input image-level labels or coarse annotations, reducing the manual annotation effort. A graph-based vision transformer was further optimized to deal with this classification task holistically.⁹⁶ It incorporated the graph representation of pathology images and a vision transformer for processing WSI images to overcome the label noise of the patch-based methods, which outperformed current advanced methods.

In addition, several segmentation algorithms were applied to locate the distributions of lung cancer pathology images. A CNN called ConvPath automatically converted a WSI into a spatial map of tumor cells, stromal cells, and lymphocyte cells.⁹⁷ Another multi-resolution CNN named HookNet successfully split up tertiary lymphoid structures and germinal centers in lung cancer via a hooking mechanism.¹⁰⁵ In the ACDC@LungHP (Automatic Cancer Detection and Classification in Whole-slide Lung Histopathology) challenge, the best Dice coefficient of the segmentation results for lung cancer tissue reached 0.8372.¹⁰⁶ The multi-model methods that combined multiple networks performed significantly better than the single-model methods by effectively assisting physicians in locating suspicious areas.

Enhancement of oncology care

AI approaches are employed to identify previously unrecognized image features associated with patient prognosis and guide treatment decisions. A total of 9879 quantitative image features were automatically extracted from 2186 WSIs of LUAD and LUSC patients by machine-learning methods to predict NSCLC prognosis.⁹⁸ The method showed excellent results in distinguishing shorter-term survivors from longer-term survivors in the test set (P < 0.05). Recently, the image features extracted from hematoxylin & eosin (H&E) images based on weakly supervised survival CNNs were significantly associated with the prognosis of lung and bladder cancers in both univariate and multifactorial analyses.¹⁰¹

Simultaneously, the AI models predict lung cancer prognosis through the assessment of the PD-L1 expression. Currently, PD-L1 tumor proportion score (TPS) grade requires immunohistochemical staining and manual interpretation to recommend the appropriate treatment. However, this process is subject to biases such as errors in staining operations, differences in agents, and manual subjectivity. Deep learning models have standardized this process to coordinate with the pathologist with a concordance value above 0.9 and further predicted patient prognosis (HR for high vs. low automated tumor cell (TC) score group: 0.539, P = 0.004).¹⁰⁰ With the assistance of AI, the overall concordance rate among pathologists improved from 81.4% to 90.2%, with a reduced HR for overall patient survival and progression-free survival after ICI treatment.¹⁰²

Furthermore, several pan-cancer models are proposed to match clinical scenarios. A tumor-environment-associated context learning using graph deep learning (TEA-graph) trained on pathological images of kidney, breast, lung, and uterine cancers considered contextual features in gigapixel-sized WSIs.²⁷ The prognostic C-index of lung cancer in NLST and TCGA was 0.731 and 0.709, respectively. After correlating 17,355 pathological images from 28 cancer types with their matched genomic, transcriptomic, and survival data, the pan-cancer computational histopathology (PC-CHiP) analysis distinguished histopathological subtypes and highlighted prognosis-related areas such as necrosis or lymphocyte aggregation.⁹⁹ Interestingly, multimodal data fusion improved the predictive model performance. In 14 cancer types, most fusion models were able to distinguish well between high-risk and low-risk groups for survival prediction.¹⁰³ For instance, in LUAD, the prognostic prediction performance of the deep-learning-based multimodal fusion (MMF) algorithm was superior to that of the attention-based multiple-instance learning (AMIL) network for processing WSIs and the self-normalizing network (SNN) for processing molecular data features. Additionally, the quantitative assessment revealed increased lymphocyte and tumor infiltrating lymphocytes presence in low-risk patients. These explorations point the way for future multimodality development.

Challenges and opportunities

In spite of the current boom in AI for medical image prediction tasks, several challenges have been faced. However, these obstacles also represent opportunities to bridge the translation between scientific research and clinical applications (Fig. 4). The foremost dilemma is the collection and annotation of standardized data for model training, validation, and testing. For instance, varying institutions, different CT scanners, and diverse reconstruction methods affect the quality of CT images.¹⁰⁷ Therefore, uniform acquisition coefficients are required to reduce image capture variability. Advanced decentralized AI algorithms such as swarm learning and federated learning achieve comparable performance to models trained on merged datasets through shared training parameters and weights, promising to support multicenter research.¹⁰⁸

Fig. 4.

The translational gap from AI models to clinical application. AI: Artificial intelligence.

Subsequently, the segmentation and labeling of images become labor-intensive, especially in the feature extraction phase of machine learning. Weakly supervised or semi-supervised learning ameliorates this difficulty to a certain extent.¹⁰⁹ To deal with the challenge of acquiring labels at the input layer, the Semi-Supervised Medical Image Detection (SSMD) method was constructed to achieve the effect of automatically obtaining a large number of high-quality labels after a small amount of manual labeling.²⁸ The performance is verified in image segmentation and lesion detection by using the adaptive consistency principle to recognize unlabeled or weakly labeled images. A multi-view divide-and-rule (MV-DAR) model using fuzzy annotations to construct multiple views to predict the probability of malignancy of lung nodules is also available, providing interesting experimental ideas for label acquisition.¹¹⁰

On the regulatory side, the national US FDA has been overseeing these automated clinical decision systems. A number of CT-based lung nodule detection systems have been approved, but most systems remain in the research phase.¹¹¹ The reason hindering their clinical application is the feasibility of the algorithms. The amount of human expert intervention required for the AUC, accuracy, and 95% CIs as metrics for algorithm performance assessment is unclear. Thus, extensive prospective trials are required. In addition, the interpretability of AI models, especially deep learning models, might be a critical limitation for their clinical application.^112,113 Although there are attention maps indicating deep learning signals from specific regions of an image, the specific information predicted for that region is difficult to quantify yet and requires additional correlation studies.

Increasingly, it has been recognized that the occurrence of cancer is a complex biological behavior influenced by multiple factors. Medical data from different modalities provide information on patient care from specific perspectives, with both overlapping and complementary information. The fusion of multimodal data shows the path to the realization of accurate disease diagnosis.²⁹ Further, AI is a perfect fit with regard to the purpose of the integration of parallel information streams such as demographics, radiomics, pathomics, and genomics to improve predictive models for patient outcomes. A previous study have confirmed this observation that a combined model comprising deep learning, imaging histology, and clinical features performed optimally in measuring PD-L1 expression levels.⁸² Additionally, the multimodal fusion model of pathological images and molecular data features was superior to single models in predicting the prognosis of cancer patients.¹⁰³ Moreover, fusion models have revealed the association between multiple modalities. For example, certain mutations in adenocarcinoma correlated closely with specific changes in cell morphology.¹⁸ These cross-modal relationships can validate existing biomarkers or obtain non-invasive alternatives to support large-scale population screening or selection of patients with high benefits for treatment.

Conclusion

In summary, AI is an exciting development in precision diagnosis and treatment of lung cancer. Radiomics and deep learning models based on radiologic images are effective tools for lung cancer screening, non-invasive diagnosis and personalized management. Meanwhile, AI technology for pathological slides enables subtype identification and treatment decision making to improve the efficiency of physicians. This emerging field has the potential to serve as a novel arsenal of tools for clinicians, advancing cancer care into the era of precision medicine.

Conflicts of interest

No conflict of interest exists in the submission of this manuscript, and the manuscript is approved by all authors for publication. We would like to proclaim that the work described is an original study that has not been published previously, and is not under consideration for publication elsewhere, in whole or in part.

Edited by: Peifang Wei