Integrating artificial intelligence into radiological cancer imaging: from diagnosis and treatment response to prognosis

Cancer poses a serious threat to human health worldwide and is a leading cause of death¹. The analysis of radiological imaging is crucial in early detection, accurate diagnosis, effective treatment planning, and ongoing monitoring of patients with cancer. However, several challenges impede the effectiveness of cancer imaging analysis in clinical practice. One difficulty is that healthcare professionals’ immense clinical workloads can result in time constraints and increase pressure, thereby hindering their ability to maintain high accuracy and thoroughness in image analysis. Additionally, subjective variability among radiologists can lead to inconsistent interpretations and diagnoses. Because this variability is often influenced by personal biases, standardized assessments are often difficult to achieve. Moreover, the inherent complexity of cancer imaging necessitates extensive clinical experience; this aspect can also be a limiting factor, particularly if expertise or resources are limited. The application of artificial intelligence (AI) can alleviate these problems by enhancing the accuracy, objectivity, and efficiency of cancer imaging analysis while assisting physicians. Therefore, the advancement of AI research is crucial for achieving progress in radiology.

With the development of computing resources, AI technologies have been widely applied in the field of radiological cancer imaging. Numerous AI models, through continuous refinement, have achieved performance comparable to or even surpassing that of radiologists in identifying various types of lesions. In recent years, AI has been effectively used in the detection of pulmonary nodules, breast cancer, and colon cancer^2–5. These successful applications have prompted the evaluation of AI approaches in more complex decision-making tasks, including cancer diagnosis, predicting treatment responses, and assessing disease prognosis. Herein, we compare mainstream AI methods used in the field of radiology and illustrate their applications in tumor imaging analysis. We also discuss the current limitations of these AI methods and explore potential directions for future AI advancements, to better integrate AI into clinical practice.

AI technologies applied in radiology

Currently, three primary types of AI approaches are widely used for analyzing cancer imaging in radiology: machine learning with radiomics, deep learning, and large models. These AI technologies can leverage imaging data to identify biomarkers for diagnosis, response prediction and prognosis, thereby offering a non-invasive, tissue-preserving method that is compatible with existing clinical workflows (Figure 1).

Figure 1.

Overview of AI-driven methods in radiological cancer imaging. This schematic illustrates the foundational concepts of machine learning with radiomics, deep learning, and large models, and their roles in cancer diagnosis, treatment response, and prognosis prediction.

Among these approaches, machine learning with radiomics involves extraction of predefined features from radiological cancer images through data characterization algorithms. These features capture various aspects of tumoral patterns, such as intensity-based metrics; texture; shape; peritumoral characteristics; first-order statistics; and tumor heterogeneity, volume, and vascular features. An initial step in the radiomics workflow is feature selection, wherein a broad array of features is refined to a smaller, task-specific subset. This process is aimed at enhancing predictive accuracy, minimizing feature redundancy, or improving robustness and stability. The selected features are then fed into machine learning models, such as logistic regression or random forest, for outcome prediction. Developing radiomics models generally does not require extensive training data or high computational resources. Moreover, the model features are derived from fixed mathematical formulas and have interpretable definitions. Accurately delineating tumor boundaries or regions of interest is essential for feature extraction, but this step is normally labor intensive for radiologists.

Deep learning, a type of machine learning, uses neural networks to process data through multiple layers of nonlinear transformations. Each layer creates more abstract features, thus aiding in recognition of patterns in the data. These features are then used to produce outputs such as predicted treatment outcomes or tumor subtype classification. Building a deep learning model involves several steps comprising data collection and preprocessing; selection of a network architecture; and splitting of the data into training, validation, and test sets. During training, the model is continually adjusted according to validation data results. Once optimized, the model can be deployed to evaluate its performance in real-world settings. Unlike radiomics, which requires manual feature extraction, deep learning automatically learns patterns from the data through convolutional operations. However, it is often considered a “black box” because of the lack of explanation for its decision-making process. Furthermore, deep learning requires more computational resources and larger datasets than radiomics for training; however, it can be trained with less detailed manual annotation and can flexibly work with 2D and 3D data. Deep learning is also adaptable in addressing data problems such as data imbalance and hard sample learning, by using approaches including data augmentation, custom loss functions, and optimized models. These approaches enable deep learning to effectively analyze cancer imaging findings.

Large models, often referred to as foundational models, are deep learning architectures characterized by vast numbers of parameters and complex structures. These models have emerged in areas such as natural language processing and computer vision^6,7. Their design, aimed at enhancing both expressive power and predictive accuracy, enables these models to handle highly complex tasks. Compared with smaller models related to radiomics or standard deep learning, large models have better performance in identifying intricate patterns and generalizing output for previously unseen data. However, because building such models normally requires substantial computational resources and large datasets, they are typically developed in industrial settings. In contrast, adapting general-purpose models to specific tasks often requires less data. Researchers typically fine-tune or pretrain models on task-specific data to improve performance and effectiveness in specialized scenarios. To further clarify the differences among these 3 AI approaches, we provide a comparison of their characteristics is provided in Table 1.

Table 1.

Comparison of the 3 main AI methods applied in cancer imaging

Characteristics	Machine learning with radiomics	Deep learning	Large models
Data requirement	Moderate	Adequate	Enormous
Hardware requirement	Moderate	High	Very high
Annotation	Manual delineation	Flexible	Flexible
Image features	Predefined	Learned automatically	Learned automatically
Performance	Moderate	High	Very high
Explainability	Good	Poor	Poor

AI for cancer diagnosis

Accurate interpretation of imaging is crucial in cancer diagnosis. In cancer imaging analysis, tumor heterogeneity and the diverse imaging features of tumors are key factors influencing diagnostic accuracy. By leveraging AI and imaging technologies, radiologists can effectively extract multidimensional tumor information that can aid in patient stratification, molecular diagnostics, metastasis prediction, radiology report generation, answering medical questions, and assessing malignancy.

To predict and stratify pathological low- and high-grade bladder cancer according to CT images, Zhang et al. have developed a radiomics-based logistic regression model by using data from 108 patients; this model achieved an area under receiver operating characteristic curve (AUC) of 0.86 on a validation set of 37 patients⁸. Additionally, Kniep et al. have used a random forest model with radiomics features to predict tumor type from MRI images of brain metastases⁹. Their model was trained on 526 brain metastases and achieved an AUC ranging from 0.64 for non-small cell lung cancer to 0.82 for melanoma, in 132 metastases. Furthermore, Fan et al. have trained and tested a logistic regression model using radiomics features extracted from CT scans of 119 patients with stage II colorectal cancer; this model predicted microsatellite instability status with an AUC of 0.75¹⁰. In the field of deep learning, researchers have developed and validated convolutional neural networks for estimating malignancy risk, by using 16,077 lung nodules from CT scans in the National Lung Screening Trial. This algorithm showed a high AUC of 0.93 on 883 nodules in the Danish Lung Cancer Screening Trial, achieving a performance comparable to that of thoracic radiologists¹¹. Beyond image-based diagnosis, large AI models also facilitate the generation of diagnostic reports. Researchers have explored the effectiveness of the large language model GPT4 in generating radiology reports for various anatomical locations. This model was found to accurately generate structured reports based on free-text PET/CT reports for 131 patients with breast cancer, particularly regarding primary lesion size (accuracy: 89.6%) and metastatic lesion details (accuracy: 96.3%)¹². Other researchers have developed large models based on both vision and language information, which have achieved promising results in radiology visual question answering, report generation, and summarization tasks for ultrasound and chest X-ray images^7,13.

Although deep learning or large models have shown satisfactory results in cancer imaging diagnosis, their decision-making process remains opaque. Unlike machine learning with radiomics, which provides interpretable features, deep learning and large models must improve the interpretability of their extracted features to enable more effective integration into clinical practice. Techniques such as gradient-weighted class activation mapping (Grad-CAM) and SHapley Additive exPlanations (SHAP) can provide insights into the decision-making process by highlighting the most relevant features or regions contributing to the predictions of deep learning models^14,15. For instance, Song et al. have applied Grad-CAM to identify the image regions that convolutional neural networks considered significant in distinguishing between benign and malignant thyroid nodules in ultrasound images¹⁶. Similarly, Islam et al. have used SHAP to highlight important heatmap regions contributing to the diagnosis of lung abnormalities by using a transformer-based model¹⁷. By integrating these visualization techniques, AI models can enhance transparency and interpretability, thereby fostering trust among healthcare professionals and ensuring reliable decision-making in cancer management.

AI for treatment response evaluation

Treatment responses describe the direct reactions of the body to therapeutic interventions. These responses include specific indicators such as tumor reduction, symptom improvement, and changes in biomarker levels. Evaluating these responses can aid in determining the effectiveness of a patient’s current treatment plan and potentially influence decisions regarding subsequent therapies.

To distinguish controlled from progressive disease, Colen et al. have created an XGBoost model with radiomics to predict pembrolizumab response in 57 patients with advanced rare cancers enrolled in a phase II clinical trial of pembrolizumab. After application of the least absolute shrinkage and selection operator for feature selection on pretreatment contrast enhanced CT scans, the model achieved high accuracy, sensitivity, and specificity (94.7%, 97.3% and 90%, respectively), as assessed according to RECIST criteria¹⁸. Furthermore, Antunovic et al. have developed and validated logistic regression models with radiomics to predict pathological complete response to neoadjuvant chemotherapy in 79 patients with locally advanced breast cancer on PET/CT¹⁹. The models yielded AUC values of 0.70–0.73 and indicated that HER2⁺, triple negative patients were more likely to have a pathological complete response than those with the luminal subtype. The aforementioned studies used imaging data to predict treatment response. Incorporating additional clinical information, such as demographic details, racial and ethnic background, molecular subtypes, and laboratory results, may further improve the accuracy of pathological complete response prediction²⁰. Compared with machine learning approaches with radiomics, deep learning excels in handling complex, multimodal datasets; consequently, deep learning offers greater potential for accurate response prediction. For instance, by integrating CT imaging, histopathologic, and genomic features, Vanguri et al. have successfully built a multimodal deep learning model for assessing immunotherapy response to PD-(L)1 blockade, by using data from 247 patients with advanced non-small cell lung cancer²¹. Their model achieved an AUC of 0.80 and outperformed investigated unimodal models.

For treatment response evaluation, AI models might have reasonable performance on internal validation but frequently face a problem of performance decline in the external validation of data from diverse sources. This problem is attributable to variations in population characteristics, imaging parameters, or image quality. Using large-scale data, transfer learning, and self-supervised learning techniques for model training might help improve model generalizability in real-world clinical settings.

AI for cancer prognosis

Cancer prognosis refers to the expected future health outcomes and disease progression after cancer treatment. Physicians rely on prognostic assessments to guide the development of personalized treatment plans, evaluate treatment efficacy, and discuss strategies for managing disease. However, prognosis is influenced by factors including cancer type, cancer stage, treatment modality, patient age, overall health, and specific tumor biomarkers. In clinical practice, physicians face challenges in providing accurate prognostic assessments based on imaging.

AI technologies offer new avenues for determining cancer treatment prognosis from medical imaging findings. For example, Zheng et al. have trained a survival prediction model on CT data and clinical information for 189 patients with stage I-IIIA non-small cell lung cancer who received stereotactic body radiation therapy²². This multimodal model effectively stratified low- and high-risk patients, and achieved an AUC of 0.76 on the internal validation set comprising 81 patients and an AUC of 0.64 on the Maastro test set comprising 228 patients. Moreover, Leger et al. have used a radiomics-based random forest model trained on 48 patients to improve the prediction of overall survival in patients with head and neck cancer undergoing CT imaging during treatment²³. Second-week CT scans in 30 patients presented a higher C-index of 0.79 than the value of 0.65 observed in pretreatment scans, as confirmed with Kaplan-Meier analyses. Similarly, Zhou et al. have developed a logistic regression model based on radiomics features from arterial- and portal venous-phase CT scans to predict early recurrence of hepatocellular carcinoma. The model, trained and tested on data from 215 patients, achieved an AUC of 0.82²⁴. Wei et al. have conducted a study in a cohort of 94 patients to assess whether a logistic regression model with radiomics features could predict 3-year recurrence of advanced ovarian cancer before surgery²⁵. Their radiomics nomogram, using pretherapeutic contrast material-enhanced CT of the abdomen and pelvis for feature extraction, achieved an AUC of 0.85 on the validation set comprising 39 patients. Focusing on stage II and III colorectal cancer, Badic et al. have extracted radiomics features from 136 contrast-enhanced CT scans for model training, and used a random forest model to predict recurrence after surgery; the model achieved an AUC of 0.79 on a test cohort of 57 patients²⁶.

Numerous studies have focused on using radiomics instead of deep learning or large models for prognostic research, primarily because obtaining prognostic data is challenging, and radiomics can create models by using datasets of moderate size. However, radiomics models encounter clinical challenges because of inconsistencies in tumor segmentation, thus resulting in variable feature extraction and ultimately affecting model performance. When model development or validation involves tumor region segmentation, the large model Segment Anything can be applied or fine-tuned for automated segmentation²⁷. Consequently, reproducibility can be increased, and the integration of radiomics and deep learning algorithms can be accelerated. Furthermore, AI in prognosis research has yet to establish quantifiable diagnostic markers and remains largely qualitative. Phrases such as “poor prognosis” or “unfavorable outcome” lack specificity in offering clear prognostic insights. Thus, a need persists to develop AI models that can further quantify the effects of factors or imaging features on prognosis, and provide more specific imaging biomarkers before treatment.

Potential of AI beyond clinical practice

Beyond cancer diagnosis, treatment response, and prognosis, AI has the potential to revolutionize the radiological cancer imaging field beyond direct clinical applications, by addressing operational challenges. AI can optimize the allocation of medical resources by predicting patient demand and streamlining imaging protocols²⁸, thereby preventing unnecessary examinations and increasing cost efficiency²⁹. Workflow automation is another critical area in which AI excels, particularly in automating repetitive tasks, extracting unstructured data, and summarizing the literature in radiology analysis^30,31. These capabilities not only decrease radiologists’ workload but also minimize turnaround time, thus ensuring timely patient care³². Furthermore, AI can contribute to quality improvement by introducing standardized and consistent methods. For example, by automating complex image processing tasks, AI can minimize the influence of subjective judgment and consequently decrease interobserver variability³³. Additionally, AI systems can identify subtle errors that can potentially arise during manual workflows, such as minor discrepancies in lesion measurements or inaccurate use of radiology terms in report writing^34,35. These capabilities are particularly valuable in high-volume imaging centers, where the risk of human error increases with the workload. Moreover, AI can serve as a quality control tool by cross-referencing imaging findings with radiology reports and flagging inconsistencies for further review³⁶. This proactive approach has the potential to not only efficiently identify missed findings but also markedly expedite radiology quality assurance programs. These advancements would collectively enhance the efficiency, accuracy, and reliability of cancer imaging processes, and might pave the way to more sustainable and high-performing healthcare systems.

Conclusions and future perspectives

Herein, we provided an overview of the major AI technologies and their applications in radiological cancer imaging. Although AI has shown promising results in various clinical tasks, substantial room remains for performance improvement.

The success of high-performing AI models depends on access to high-quality datasets. To provide such datasets, standardized acquisition protocols and rigorous imaging quality control processes should be carefully designed and implemented, to ensure generation of consistent, low-noise images that accurately represent clinical conditions. Additionally, multiple experienced clinicians should participate in the data annotation process, including tasks such as lesion delineation and imaging diagnosis. These efforts would enable AI models to effectively learn reproducible and clinically relevant imaging features, while minimizing the influence of noisy labels and reader variability during training.

Beyond using high quality datasets for modeling, integrating and analyzing heterogeneous, multidimensional clinical data—including pathology, proteomic, and genomic data—is also essential to effectively enable AI-powered cancer diagnosis, treatment response evaluation, and disease prognosis³⁷. The combination of these diverse data modalities might provide a more comprehensive understanding of the underlying biological mechanisms, thus enhancing the predictive accuracy of AI models. Integrating genomic data can reveal tumor-specific mutations, whereas proteomic profiles can uncover dynamic changes in protein expression levels, both of which are highly relevant to individualized treatment planning and outcome prediction. To support this integration, efforts should focus on establishing unified frameworks for feature reduction, alignment, and merging, to efficiently use different types of data.

Additionally, addressing the technical bottlenecks that AI models might encounter in real-world clinical applications is crucial. For instance, ensuring the transferability and generalizability of AI models across diverse data sources remains a major challenge, because of variations in imaging protocols, equipment, and patient demographics. To overcome this challenge, robust domain adaptation techniques must be developed, and standardized benchmarks for cross-institutional validation must be established. Moreover, the development of AI should prioritize privacy protection, by using advanced encryption techniques and federated learning frameworks, to minimize risks associated with data breaches. In addition, addressing data imbalance is essential, to avoid biased model performance and diminished model reliability in underrepresented patient groups or conditions. Furthermore, enhancing real-time processing and inference speed with efficient architectures is also important to ensure that AI models can meet the demands of time-sensitive clinical scenarios, and enable faster, more effective decision-making in critical care and emergency settings.

The integration of AI with advanced imaging modalities such as 5T MRI and spectral CT has immense potential for improving diagnostic accuracy and clinical decision-making^38,39. For 5T MRI, AI can enhance image reconstruction, reduce noise, and extract novel quantitative biomarkers from high-resolution data, thus providing deeper insights into subtle pathological changes. Similarly, in spectral CT, AI can leverage multi-energy data to achieve precise tissue characterization, material decomposition, and improved lesion differentiation. By harnessing the capabilities of AI, these cutting-edge imaging techniques can be further optimized, to facilitate more accurate diagnoses, personalized treatment planning, and advancements in precision medicine.

To effectively integrate AI into clinical practice, radiologists must gain a foundational understanding of AI, including proficiency in data interpretation and algorithmic processes, as well as understanding the limitations of AI tools, to critically assess their reliability and applicability. Training in machine learning basics, programming, and data science would enhance collaboration with AI developers, whereas knowledge of ethical considerations, such as bias and data privacy, would ensure responsible implementation. Additionally, radiologists should adapt to new workflows that incorporate AI, focusing on tasks requiring human judgment, such as correlating AI findings with clinical context and improving patient communication. These skills would position radiologists as essential intermediaries between AI tools and patient care, in maximizing the potential benefits of the technology for clinical use.

In the future, the advancement of AI in radiology will depend on close collaboration between radiologists and technical experts. Promoting partnerships between these professionals is necessary to ensure that developed AI systems are both technically robust and clinically meaningful.

Funding Statement

This study was funded by grants from the National Natural Science Foundation of China (Grant Nos. 82171932 and 82302180), the Ministry of Science and Technology of China (Grant No. 2024ZD0520002), the Chinese National Key Research and Development Project (Grant Nos. 2021YFC2500402 and 2021YFC2500400), the National Health Commission Capacity Building and Continuing Education Center (Grant No. YXFSC2022JJSJ011), the Tianjin Key Medical Discipline (Specialty) Construction Project (Grant No. TJYXZDXK-010A), and the Scientific Developing Foundation of Tianjin Education Commission (Grant No. 2024KJ182).

Conflict of interest statement

No potential conflicts of interest are disclosed.

Author contributions

Conceived and designed the analysis: Sunyi Zheng, Xiaonan Cui, and Zhaoxiang Ye.

Wrote the paper: Sunyi Zheng and Xiaonan Cui.