Challenges and opportunities for artificial intelligence in oncological imaging

Abstract

Imaging plays a key role in oncology, including the diagnosis and detection of cancer, determining clinical management, assessing treatment response, and complications of treatment or disease. The current use of clinical oncology is predominantly qualitative in nature with some relatively crude size-based measurements of tumours for assessment of disease progression or treatment response; however, it is increasingly understood that there may be significantly more information about oncological disease that can be obtained from imaging that is not currently utilized. Artificial intelligence (AI) has the potential to harness quantitative techniques to improve oncological imaging. These may include improving the efficiency or accuracy of traditional roles of imaging such as diagnosis or detection. These may also include new roles for imaging such as risk-stratifying patients for different types of therapy or determining biological tumour subtypes. This review article outlines several major areas in oncological imaging where there may be opportunities for AI technology. These include (1) screening and detection of cancer, (2) diagnosis and risk stratification, (3) tumour segmentation, (4) precision oncology, and (5) predicting prognosis and assessing treatment response. This review will also address some of the potential barriers to AI research in oncological imaging.

Clinical needs in oncological imaging and opportunities for artificial intelligence

Imaging plays an integral role in oncology, including detection and diagnosis of cancer, changes in tumours with treatment or with disease progression, and complications of the disease or treatment. Traditionally, the role of imaging has been limited to predominantly qualitative assessment of tumours for diagnosis or crude size-based measurements of tumours for assessment of disease progression or treatment response (such as RECIST criteria for clinical trials); however, advances in the understanding of cancer biology has led to improved understanding of different tumour subtypes, the pathophysiology of tumours, and the response of different subtypes of treatment. It is being increasingly recognised that imaging may hold significant information about the phenotype of cancer that is not currently being used for characterisation or treatment.

Artificial intelligence (AI) in oncological imaging has the potential to improve the efficiency and accuracy of the traditional clinical roles of imaging as well as introducing new roles not currently part of routine clinical imaging using quantitative techniques.^1,2 Due to increased computational power and advances in AI research, techniques are now available for inputting large amounts of digital data and training algorithms to recognise complex patterns.² Algorithms can be trained on either structured or unstructured data with the possibility of integrating imaging and non-imaging (including clinical, histopathological, or genomics) data. AI algorithms can be determined based on predefined quantitative features that are based on expert knowledge or using deep learning techniques without predefined features, but typically requiring large datasets.^2,3

AI techniques could potentially be used to segment tumours and efficiently measure tumours and compare to previous studies in a precise and systematic manner. Efficiency is crucial in the use of imaging in clinical practice due to increasing volume demands on radiologists. Even relatively straightforward size-based criteria such as Response Evaluation Criteria In Solid Tumors (RECIST) used for measuring treatment response are often not routinely used in clinical practice due to lack of automation and the time and effort that is required by radiologists to manually measure and compare to previous studies in a systematic manner. AI techniques could also be potentially used to identify imaging features that can classify tumours based on tumour features either for improved diagnosis or tumour subtype classification (corresponding to pathological or genomic patterns that may have either prognostic or treatment-related implications).

Radiologists can play an important role in helping to direct AI research by identifying potential features based on known pathology or imaging characteristics as well as identifying important clinical questions. Radiologists can also play an important role in translating AI research to clinical practice. The aim of this review is to outline several major areas in oncological imaging where there may be opportunities for AI technology. Specifically, this will include the use of AI in (1) screening and detection of cancer, (2) diagnosis and risk stratification, (3) tumour segmentation, (4) precision oncology, (5) predicting prognosis and assessing treatment response, and (6) potential barriers to AI research (Tables 1 and 2).

Table 1.

Summary of opportunities for artificial intelligence (AI) technology in oncological imaging.

	Current clinical practice	Potential for AI technology
Screening/detection	Predominantly manual detection	May use automated or semi-automatic techniques to increase accuracy or efficiency
Diagnosis and risk stratification	Predominantly clinical “rules” based (TI-RADS, Lung-RADS, PI-RADS, etc.)	May improve accuracy
Tumour segmentation	Not typically done as it is highly labour intensive	May improve efficiency and reliability May allow for feature extraction for other uses of AI in oncological imaging
Precision oncology	Not typically done	May subtype tumours into known or new histopathology or genomic subtypes that can be targeted for therapy
Predicting prognosis and assessing treatment response	Tumours stage based on anatomy (local invasion and metastases) Treatment response typically size-based criteria (e.g., RECIST), limited by pseudoprogression or pseudoresponse	May subtype tumours by tumour biology/grade to determine appropriate treatment May use other features in addition to tumour size including tumour morphology to determine response and to distinguish from treatment effects

TI-RADS, Thyroid Imaging Reporting and Data Systems; Lung-RADS, Lung Reporting and Data Systems; PI-RADS, Prostate Imaging Reporting and Data Systems; RECIST, Response Evaluation Criteria In Solid Tumors.

Table 2.

Summary of barriers to artificial intelligence (AI) research in oncology.

Barriers to AI research in oncology
Difficulty in obtaining large, good-quality, multi-institutional data sets
Inconsistent or poor-quality imaging
Labour-intensive tumour segmentation and annotation
Translation into real clinical settings
Multidisciplinary approach
Streamlined processes compatible with clinical workflow
Legal and ethical challenges

Screening and detecting of cancer

One of the key roles of current oncological imaging is to screen patient populations and to detect malignancy at an early stage. AI techniques may have potential to act as a detection aid or as a “second reader” for radiologists, which may help to improve both their efficiency and accuracy.

In breast cancer screening programmes, various forms of computer-assisted diagnosis (CAD) for mammography have been in development since the 1990s. These initial CAD programs used classification systems with rules-based approach.^4,5 Unfortunately, early forms of CAD did not lead to improved diagnostic accuracy as there was often increased detection (slightly higher sensitivity), but poor specificity leading to false positives and increased recall rates and biopsy^4,5; however, more recent studies using deep-learning techniques may have improved specificity of the detection task, although this remains an area of ongoing research. One study of 3,228 patients showed that the performance of deep neural networks was similar to radiologists for diagnosis of malignancy in a screening population, although radiologists were slightly less sensitive and more specific.⁶ Other studies have looked at other screening-related clinical problems including breast cancer risk models and reducing recall rates. For example, a large study of 39,571 women used hybrid mammography images and traditional risk factors to develop a deep-learning model for breast cancer risk with improvement over traditional Tyrer–Cuzick risk models.⁷ Another study looked at decreasing recall rates in 5,147 patients with BIRADS 4 lesions using a deep-learning based model that integrates mammography images and medical reports with seemingly good results (100% sensitivity and 74% specificity and AUC of 0.93).⁸ Several large studies from the UK have demonstrated that single reading with CAD may be equivalent to double reading with two radiologists, although with higher recall rates and with slightly different types of missed cases.^9,10 A single reader with CAD is now considered an acceptable alternative to double reading in the UK where double reading is recommended and considered standard practice.

In thoracic imaging, computed tomography (CT) of the chest for lung cancer screening is becoming more accepted. Automatic lung nodule detection on CT chest using CAD approaches has been present since the early 2000s.¹¹ As with breast imaging, initial approaches involved complex classification algorithms with high false-positive rates and failed to gain widespread clinical acceptance.¹¹ In more recent years, large databases of CT chest studies have been available due to lung cancer screening trials, such as the NELSON trial.¹² Using data from these trials and deep-learning techniques, more recent models appear to demonstrate significantly improved specificity. For example, one study using 400 CT examinations selected randomly from the NELSON trial demonstrated high sensitivity with 96.7% for CAD compared to 78.1% for radiologists with double reading. The false-positive rate of CAD was higher than the consensus of double expert readers (3.7% for CAD versus 0.5% for readers), although this was reduced when small nodules <50 mm³ were excluded (1.9% for CAD and 0.1% for readers).¹³ This study was limited by relatively small number of subsolid lesions. In other studies, it has been shown that the usefulness of commercially available CAD technology is less sensitive for subsolid nodules.¹⁴

CAD approaches to early detection of cancer are also being developed beyond the breast cancer and lung cancer screening programmes. In abdominal imaging, there have been some CAD approaches to detection of polyps and colorectal cancer on CT colonography, although these studies are smaller and less well developed than those in breast or lung cancer screening. As with breast and thoracic imaging, initial CAD approaches involved complex classification based on predefined features and similarly appear to have improved sensitivity at the cost of reduced specificity.^15,16 Further studies with newer techniques may yield better results. In prostate cancer, AI techniques appear to demonstrate mild improvement in the detection of cancer on magnetic resonance imaging (MRI) particularly within the more difficult central gland or transition zone lesions, and have been shown to have reasonably consistent results across institutions.^17–19 Importantly, some of these more recent studies have demonstrated high specificity, which had been a challenge of older techniques.

Other potential areas for AI research in cancer detection include detecting metastatic disease in patients with known primary cancer, including common sites of metastases, such as lung, liver, bone, or peritoneum.²⁰ The challenge is similar to detection of primary malignancy, as early studies appear to offer potential for high sensitivity, but is limited in specificity. For example, one study using a machine learning based CAD software to detect melanoma lung metastases on CT chest, found additional nodules in 54.3% of patients compared to radiologists and led to altered follow-up in 30%²¹; however, none of these lung nodules turned out to be malignant or otherwise clinically significant on follow-up.²¹

Improving detection of cancer for screening patient populations is a significant area of AI research in oncological imaging. In theory, AI has the potential to improve efficiency and accuracy of radiologists in this area. As noted above, the early CAD programs using rules-based classification systems had the tendency to result in high sensitivity but with low specificity leading to increased unnecessary intervention and additional cost. For this reason, these early systems failed to gain widespread clinical adoption and change in clinical practice; however, newer techniques and newer studies demonstrate promise for improving specificity. If this can be achieved, then AI technology may play a significant role in aiding radiologists in the area of cancer detection. With the exception of CAD programs for double reading in breast imaging, most CAD programs are not currently incorporated into clinical workflow. Improvements in workflow integration is also required in order to gain widespread clinical acceptance.

Diagnosis and risk stratification of cancer

Once a lesion is detected, the next task is to determine if it represents malignancy. AI has the potential to improve the accuracy of radiologists in distinguishing benign from malignant lesions or determining the aggressiveness of tumours, particularly in cancers where low-grade tumours are treated conservatively and high-grade tumours are treated aggressively, such as in thyroid or prostate cancer. There may even be a potential for AI to help determine which patients may be at risk for developing cancer, such as deep-learning techniques to discriminate breast density in mammograms, which may imply increased cancer risk.⁷ The use of AI in the diagnosis of cancer has been explored for a variety of techniques including ultrasound, CT, and MRI.

Ultrasound is the primary technique for diagnosis of thyroid cancer. Thyroid nodule classification is an important clinical problem because the prevalence of thyroid nodules in the general population is high and the prevalence of thyroid cancer is low. AI has the potential to improve clinical management by risk stratifying patients for invasive thyroid biopsies. Several larger studies have developed AI algorithms that appear to have accuracy similar to radiologists using American College of Radiologists (ACR) TI-RADs (Thyroid Imaging Reporting and Data Systems) guidelines.^22–25 Other AI studies using ultrasound images have investigated the use of AI for diagnosis of malignant breast, liver, and ovarian masses, although the literature in these areas is more limited.^26,27

On CT, a large number of studies have investigated the use of AI techniques for diagnosis of a wide variety of tumours, including primary lung, liver, adrenal, and renal lesions as well as metastases.^28–33 For example, on CT chest, predetermined texture features, deep learning, or hybrid techniques have been explored as methods for distinguishing benign versus malignant lung nodules either as an aid for radiologists or for standalone diagnosis, with promising results.^34–36 Some studies seem to suggest accuracy equivalent to or possibly even better than radiologists using clinical criteria such as Lung-RADS.^36–38 On liver CT, AI techniques appear to show promise for diagnosing focal liver lesions both in cirrhotic and non-cirrhotic patients.^29,30

On MRI, an area of active research is the use of AI for diagnosis of clinically significant prostate cancer. One of the unique challenges of prostate cancer is that low-grade cancers can often be clinically insignificant and may be amendable to conservative surveillance approaches whereas high-grade cancers can have high morbidity and mortality. Therefore, AI research in prostate MRI involves not only detecting cancers, but also predicting grade and aggressiveness.^39–43 Additional areas of research include diagnosis of brain tumours, breast tumours, and liver lesions.^44–47

Tumour segmentation

Segmentation of tumours is important both clinically and for research purposes. In terms of clinical applications, segmentation is needed to determine the volume of cancers. Even for measuring the long or short axis of tumours, an implicit segmentation is needed for delineation of the boundary of lesions. Tumour segmentation may also be required for surgical or radiation planning.⁴⁸ For research purposes, segmentation of tumours is needed in order to extract image features from these lesions (Fig 1). With the exception of some deep-learning techniques, most AI research in oncological imaging requires initial tumour segmentation. This is typically done manually by an expert radiologist as a reference standard; however, it is highly labour intensive, and annotating and segmenting tumours may be a significant rate-limiting step for AI research. Manual segmentation is also limited by inter-reader variability.

Figure 1.

A 49-year-old female patient with brain metastases from breast cancer. Model input included (top row, left to right) contrast-enhanced BRAVO, pre- and post-contrast CUBE, and T2W CUBE FLAIR. Bottom row shows the predictions (probability maps), generated by the neural network, and delineated metastases (yellow circle) overlaid the contrast-enhanced BRAVO image. Reprinted with permission from the 2018 ISMRM Machine Learning Workshop.

AI techniques that can automatically or semi-automatically segment tumours may significantly reduce time as well as improve reliability. Given these benefits and the necessity of segmentation for AI research in oncology, attempts at automatic or semi-automatic segmentation has been performed for most types of tumours with active AI research.^49–53 In imaging techniques with multiple sequences such as MRI, registration between segmentations on different sequences is also an active area of research.

Precision oncology

In the last decade, there has been an explosion of research in the area of histopathology and genomics in oncology to provide more precise characterisation of tumour biology and to provide more specific guidance as to likely effective therapies, especially with the advent of immunotherapies. In many different types of cancers, histopathology or genomics features can predict the aggressiveness and prognosis of tumours. Many targeted chemotherapy agents treat specific subtypes of tumours leading to the possibility of “personalised” or “precision” medicine.⁵⁴ Traditional oncological imaging that focuses solely on tumour morphology has limited ability to distinguish subtypes for most cancers. Obtaining histopathology or genomics to characterise tumours requires tissue samples, typically obtained from biopsy or surgery, both of which are invasive. Biopsy is limited due to issues of intra- and inter-tumoural variability and sampling error as the sampled portion of the tumour may have different histopathology or genomic characteristics compared to the rest of the tumour and this can change over time as the disease progresses.^55–57 Tumour heterogeneity is a major problem that may be one of the major causes of treatment failure or treatment resistance.^55,58–60 Sampling error may be a problem even after surgical treatment, as the histopathology or genomic characteristics of tumours can change over time, and it may also be ideal to know the subtype of the tumour prior to surgery.⁵⁷ Theoretically, if this information can be obtained through non-invasive imaging techniques, this could significantly impact clinical management. Imaging techniques may also be less prone to sampling error as information could be obtained on the whole tumour. AI technology in the field or radiogenomics may be a way of obtaining this information from cancer imaging data.⁶¹

One of the most active areas of AI research in tumour subtyping is in neuro-oncology. Gliomas and other primary brain tumours are now subtyped on histopathology by aggressiveness using WHO grading.⁶² There have been significant developments in understanding how the aggressiveness of these tumours is related to molecular markers. This has led to fertile ground for AI models to predict subtypes based on MRI data on these patients.^44,63 Studies have investigated radiomics and deep learning using different sequences (T1, T2, fluid attenuation inversion recovery [FLAIR], diffusion, etc.) or a combination of sequences to predict a range of molecular subtypes include the presence of IDH1 mutation, 1p/19q co-deletion, p53 status, and Epidermal growth factor receptor (EGFR), and Vascular endothelial growth factor (VEGF), status.^64–67 AI studies have also looked at radiogenomics to characterise regional genetic variation in glioblastoma as intra-tumoural heterogeneity is believed to be a significant cause of disease progression and account for decrease in response to treatment.⁶⁸ Although these results are promising and have the potential to be useful clinically, there is currently insufficient data to determine the robustness of the AI models due to variation in imaging equipment and parameters across different institutions.⁶⁹ These problems currently limit clinical application.

Targeted chemotherapy agents in breast cancer have led to significant improvements in morbidity and mortality. Radiogenomics research may enable radiologists to detect breast cancer subtypes on breast MRI earlier and in a non-invasive manner.^70–74 AI techniques may also be helpful to predict clinically significant pathological features or molecular status in other cancers including Gleason score in prostate cancer, histopathology of primary liver cancer, EGFR, or KRAS status in non-small cell lung cancer on CT or integrated positron-emission tomography (PET)/CT, and KRAS mutation in colorectal cancer.^75–79

Predicting prognosis and assessing treatment response

For many cancers, including common cancers such as breast, prostate, and colorectal cancer, the aggressiveness and prognosis of patients is widely variable. It is increasingly understood that the individual biology of tumours is highly variable. The ability to estimate the cancer prognosis and to predict the likelihood of responding to various types of treatments may significantly change management. Current imaging techniques are typically used for staging of cancer, including determining local invasion of adjacent structures or the presence of distant metastases. These are predominantly anatomically based and are generally limited in the ability to characterise tumour grade or biology. AI technology may enable radiologists to estimate the prognosis of tumours based on imaging. This may be done in the form of subtyping tumours by predicting known histopathological or genomics features as described in the section above or by predicting prognosis based on imaging features directly.

Some studies have looked at AI technology for prognostication, although this remains limited as research requires fairly large datasets with high-quality outcome data, as many confounders may affect outcomes. For example, in non-small cell lung cancer, some studies have developed AI algorithms that can predict mortality after radiotherapy or surgery and other studies have developed algorithms that can predict response to anti-PD1 immunotherapy agents.^80,81 Other studies have looked at AI algorithms to predict prognosis and response to chemotherapy in a variety of other cancers, including survival outcomes in glioblastoma multiforme based on pre-treatment brain MRI, response to transarterial chemoembolisation in HCC based on pretreatment liver MRI, response to chemoradiation on pre-treatment rectal MRI, and pre-therapy PET/CT to predict chemotherapy response in patients with neuroendocrine tumours or with lymphoma.^82–85

Traditional radiology methods of determining treatment response is based on crude size-based criteria such as RECIST⁸⁶; however, it is known that size-based criteria can have significant limitations. Tumours may respond favourably to treatment despite increase in size (so-called “pseudoprogression”), a finding commonly seen with immunotherapy agents.⁸⁷ Conversely, cancer may be progressing or increasing in aggressiveness, but initially decrease in size (so-called “pseudoresponse”), a finding commonly seen with anti-angiogenesis agents. Tumours may also stay the same size but change in morphology with less tumour burden due to fibrotic or necrotic content or increase in aggressiveness by having more solid high-grade tumour content. Therefore, other more accurate methods of determining response to treatment are required. Some morphologically based criteria have been developed such as mRECIST for hepatocellular carcinoma and the Choi response criteria for gastrointestinal stromal tumours and renal cell carcinoma.^88,89 These have been shown in certain settings to be more predictive than size-based RECIST criteria; however, their application has not been widely established and RECIST remains the predominant assessment criteria for clinical trials.

Additionally, it can be difficult to distinguish treatment effects from residual or recurrent tumour, particularly in the setting of chemotherapy, radiation, or ablative therapies where fibrosis can mimic tumour. In tumours such as gliomas that are deeply infiltrative with associated oedema or local treatment effects, accurate assessment of tumour response may be very difficult. AI may play a role in increasing accuracy of treatment response assessment or in determining treatment response at an earlier time.

Early works suggest that AI techniques may helpful in this area. For example, in neuro-oncology, several studies seem to show promise for distinguishing radiation and other post-treatment changes with residual or recurrent tumour and predict time to progression or outcomes.^90,91 Some studies appear to distinguish radiation treatment from residual or recurrent tumours in soft-tissue sarcomas, prostate cancer, and lung cancer.^92–94 Other studies have demonstrated ability to predict incomplete versus complete treatment response to chemotherapy in bladder, rectal, and other tumours.^83,95

Use of AI for quantitative imaging may be helpful not only for treatment management, but also for clinical trials. Development of precise and accurate quantitative measurements of size, tumour characteristics, or other imaging features could lead to more accurate, precise, and reliable results in clinical trials and improvement in the quality of these trials.⁹⁶

These early studies on the use of AI in predicting prognosis and treatment response are promising; however, further research is required in order to determine the exact patient population on which these algorithms can applied and the validity of these results across institutions. In addition, further research is also required in order to determine how this information can be appropriately used to dictate patient management including decision making for different types of therapy.

Potential barriers to AI research in oncology

As discussed in this review, there are many potential areas of AI research in oncological imaging; however, several major barriers exist in AI research in oncological imaging (Table 2).

AI research, particularly deep-learning techniques, requires good-quality large datasets. Multi-institutional attempts at acquiring datasets include The Cancer Genome Atlas (TCGA) database, which has some imaging as well as clinical and linkages to genomics data.⁹⁷ In addition, data from large clinical trials obtained from other purposes, such as the NELSON trial for CT lung cancer screening, may be useful for research.¹² For cancers and clinical problems where these datasets are not available, researchers need to make a consorted effort for collaborative efforts to create appropriate datasets. AI algorithms that show promise on small single institutional data needs to be validated over different populations and different institutions.

Good imaging quality is also vital to AI research as data analysis can only be as good as the input data.⁹⁸ Tumour detection, segmentation, and feature analysis is highly dependent on input imaging quality. This may be particularly challenging in heterogeneous, multicentre datasets where image quality may be highly variable. There have been some attempts to score and assess image quality that may be helpful for standardising images for analysis^99,100

Most AI research in oncological imaging requires segmentation and annotation of the image data. This is highly time consuming and labour intensive. Development of semi-automatic or automatic segmentation techniques may allow for improved quantity of data required for analysis and increase inter-rater reliability.

Once promising AI solutions are obtained for important oncological problems, it is important to determine if the results are valid and feasible in real clinical settings. For example, early CAD solutions to detection of cancers were limited in specificity with high false-positivity rates in real clinical scenarios, which limited their uptake.^4,5,11

Legal and ethical issues surrounding the use of AI in oncological imaging are also potential barriers to research as well as implementation. Most AI research requires researchers to use large healthcare datasets, both imaging and clinical data. This leads to legal and ethical questions surrounding who “owns” the data and who has a right to use that data, particularly when there may be a commercial value. Although patient consent for data use would be ideal, this may not be practical given the number of patients in large datasets particularly in the retrospective setting as AI is a relatively new field and consent to use of data was often not performed at the time scan.¹⁰¹ Even if researchers are given the right to use anonymised or de-identified data, deidentification of imaging data can be technically challenging as identifying information can sometimes be embedded in the image or images can be reconstructed in such a way to uniquely identify a patient’s features.¹⁰² The rights of individuals for data privacy must also be balanced with the societal benefits of medical research.¹⁰³ Other ethical issues include the potential for AI to exacerbate existing discrimination or disparities in healthcare both in the context of research or in implementation.¹⁰⁴ Clear legal and ethical frameworks for using AI in oncological imaging with input from all stakeholders (patients, hospital, research institutions, government, laws, etc.) is required.¹⁰⁵

Translation of results to clinical practice requires collaborative efforts between researchers, radiologists, surgeons, radiation oncologists, oncologists, and other healthcare practitioners as well as policy- and lawmakers in order to determine how they can be appropriate and safely used to improve patient outcomes. It will also require work to integrate AI technology into the existing clinical PACS and clinical workflow. Without seamless integration, it is unlikely that technology will be adapted into busy clinical practices. Research translation will also require significant effort to gain buy-in from the clinical oncological community.

Summary

AI technology has the potential to improve efficiency and accuracy of the traditional roles of the radiologist in detection, diagnosis, and determination of treatment response. AI techniques may also help define new roles for radiologists in the areas of prognostication or classification of tumours in histopathological and genomics subtypes. AI research may also lead to automatic or semi-automatic segmentation techniques that can be used for research as well as clinical purposes. Advances in technology, including new techniques and increased computing power, have led to an explosion of AI research in the last decade. Promising studies suggest that AI technology in imaging may lead to significant advances in cancer care; however, significant work is required to increase the quantity and quality of data available for research, validating promising results across institutions and in real clinical settings, and translating the work into clinical practice.