Artificial intelligence in Cancer Imaging: Clinical Challenges and Applications

Abstract

Judgement, as one of the core tenets of medicine, relies upon integration of multi-layered data with nuanced decision-making. Cancer offers a unique context for medical decisions given not only its variegated forms with evolution of disease, but the need to take into account the individual condition of patients, their ability to receive, and their responses to treatment. Challenges remain in the accurate detection, characterization, and monitoring of cancers despite improved technologies. Radiographic assessment of disease most commonly relies upon visual evaluations, whose interpretations may be augmented by advanced computational analyses. In particular, artificial intelligence (AI) promises to make great strides in the qualitative interpretation of cancer-imaging by expert-clinicians, including volumetric delineation of tumors over time, extrapolation of tumor genotype and biological course from its radiographic phenotype, predicting clinical outcome, and assessing the impact of disease and treatment on adjacent organs. AI may automate processes in the initial interpretation of images and shift the clinical workflow of radiographic detection, management decision on whether or not to administer an intervention, and subsequent observation to a yet to be envisioned paradigm. We review the current state of AI as applied to medical imaging of cancer and describe advances in four tumor types (lung,brain,breast,prostate) to illustrate how common clinical problems are being addressed. While most studies evaluating AI applications in oncology to date have not been vigorously validated for reproducibility and generalizability, the results do highlight an increasingly concerted efforts in pushing AI technology to clinical use and to impact future directions in cancer-care.

INTRODUCTION

Cancer, as a self-sustaining and adaptive process that interacts dynamically with its microenvironment, continues to thwart patients, researchers, and clinicians despite significant progress in understanding its biological underpinnings. Given this complexity, dilemmas arise at every stage of cancer management, including reliable early detection, accurate distinction of pre-neoplastic and neoplastic lesions, determination of infiltrative tumor margins during surgical treatment, tracking of tumor evolution and potential acquired resistance to treatments over time, and prediction of tumor aggressiveness, metastasis pattern, and recurrence. Technological advances in medical imaging and minimally invasive biomarkers hold promise in addressing such challenges across the spectrum of cancer detection, treatment, and monitoring. However, the interpretation of the large volume of data that is generated by these advancements presents a barrage of new potential challenges.

As we learn more about the disease itself, we are learning more about the power of tools that are already available to us, but which may be used in unprecedented ways. When a neoplastic lesion is initially detected, it needs to be distinguished from non-neoplastic mimickers and classified based on its predicted clinical course and biological aggressiveness in order to optimize the type and intensity of treatment. The widespread availability of computed tomography (CT) and magnetic resonance imaging (MRI) have fueled incidental detection of lesions within the body with unclear clinical significance, which then initiates a cascade of observation, further testing, or empiric intervention. With treatment, which includes cytoreduction through surgery, elicitation of direct and indirect mechanisms of tumor kill through radiation, and pharmacotherapies, cancers may adapt to the stressors imposed, evolve, and recur. With the radiographic appearance of a lesion that increases in size following treatment, distinction has to be made between neoplasm or tissue response to injury. On recurrence, neoplastic lesions have been shown to harbor new molecular aberrations distinct from the primary tumor, which may confer resistance to medical or radiation therapies. This is compounded by the innate intratumoral heterogeneity of cancers at the time of initial diagnosis, which is increasingly demonstrated by research but difficult to capture in routine clinical pathological sampling and profiling. The demand for non-invasive imaging, as the most common method to track response to treatment and to infer critical information about tumors themselves, has never been greater.

Traditional radiographic imaging evaluation of tumor relies upon largely qualitative features such as tumor density, pattern of enhancement, intratumoral cellular and acellular composition including presence of blood, necrosis, mineralization, regularity of tumor margins, anatomical relationship to the surrounding tissues and impact on these structures. Size and shape based measures of the tumor can be quantified in one, two, or three dimensions. These qualitative phenotypic descriptions are collectively termed “semantic” features. In comparison, a rapidly evolving field called radiomics is enabling digital decoding of radiographic images into quantitative features, including descriptors of shape, size, and textural patterns.¹Recent advances in artificial intelligence (AI) methodologies have made great strides in automatically quantifying radiographic patterns in medical imaging data. Especially deep learning, a subset of AI, is a promising method that automatically learns feature representations from sample images, and has shown to match and even surpass human performance in task specific applications ^2,34. Despite requiring large datasets for training, deep learning has been demonstrated relative robustness against noise in ground truth labels⁵ among others. The automated capabilities of AI offer the potential to enhance the qualitative expertise of clinicians, including precise volumetric delineation of tumor size over time, parallel tracking of multiple lesions, translation of intratumoral phenotypic nuances to genotype implications, and outcome prediction through cross-referencing individual tumors to databases of potentially limitless comparable cases. Furthermore, deep learning approaches promise greater generalizability across diseases and imaging modalities,⁶robustness to noise,⁷and reduction of errors - eventually leading to earlier interventions and significant improvements in diagnosis and clinical care. While these studies remain largely in the pre-clinical research domain, continued development of such automatic radiographic ‘radiomic’ biomarkers, like blood-based biomarkers, may highlight clinically actionable changes in tumors and drive a paradigm shift in the discrimination of cancer over time.

At the dawn of this exciting technological transformation, we review the current evidence and future directions for AI approaches as applied to medical imaging in four common cancer types: lung, brain, breast, and prostate cancer. We describe clinical problems and limitations in cancer detection and treatment, how current methods are attempting to address these, and how AI can impact future directions.

AI APPLICATIONS IN CANCER IMAGING

The desire to improve the efficacy and efficiency of clinical care continues to drive multiple innovations into practice, including artificial intelligence. With the ever increasing demand for healthcare services and the large volumes of data generated daily from parallel streams, optimization and streamlining of clinical workflows have become increasingly critical. AI excels at recognizing complex patterns in images, and thus, offers the opportunity to transform image interpretation from a purely qualitative and subjective task to one that is quantifiable and effortlessly reproducible. Additionally, AI may quantify information from images that is not detectable by humans, and thereby complement clinical decision making. AI can also enable the aggregation of multiple data streams into powerful integrated diagnostic systems spanning radiographic images, genomics, pathology, electronic health records, and social networks.

Within cancer imaging, AI finds great utility in performing three main clinical tasks: detection, characterization, and monitoring of tumors (Figure 1). Detection refers to the localization of objects of interest in radiographs, collectively known as Computer-Aided Detection (CADe). AI-based detection tools can be used to reduce observational oversights and serve as an initial screen against errors of omission.⁸Formulated within a pattern recognition context, regions with suspicious imaging characteristics are highlighted and presented to the reader. CADe has been used as an auxiliary aide to identify missed cancers in low-dose CT screening,⁹ detect brain metastases in MRI to improve radiology interpretation time while maintaining high detection sensitivity,¹⁰ locate microcalcification clusters in screening mammography as an indicator of early breast carcinoma,¹¹ and more generally, has improved radiologist sensitivity for detecting abnormalities.¹²

Figure 1. AI applications in medical imaging, as applied to common cancers.

AI tools can be conceptualized to apply to three broad categories of image-based clinical tasks in oncology: (1) detection of abnormalities; (2) characterization of suspected lesion by defining its shape or volume, histopathologic diagnosis, stage of disease, or molecular profile; and (3) determination of prognosis or response to treatment over time during monitoring.

Characterization broadly captures segmentation, diagnosis, and staging of tumors. It can also extend to include prognostication based on a given ailment, as well as outcome prediction based on specific treatment modalities. Segmentation defines the extent of an abnormality. This can range from basic two dimensional measurements of the maximal in-plane tumor diameter to more involved volumetric segmentations where the entire tumor, and possible surrounding tissues, are assessed. Such information can be utilized in subsequent diagnostic tasks as well as dosage administration calculations during radiation planning. In current clinical practice, tumors are typically manually defined, with associated limitations including inter-rater bias,¹³ inconsistent reproducibility even among experts,^14,15 and consumption of time and labor. While manually traced segmentations is frequently used as the basis of judging accuracy of automated segmentation algorithms, this has the potential of neglecting subclinical disease and restricting the region of analysis to human bias. AI has the potential to dramatically increase the efficiency, reproducibility, and quality of tumor measurements with automated segmentation. Finally, with rapid expansion of computing speed and increased efficiency of AI algorithms, it is likely that future analysis of cancer lesions will not require a separate segmentation step and whole body imaging data could be evaluated directly by AI algorithm. A whole body approach can also allow analysis of organ structures that may be pathological but not apparent to human vision.

On radiologic data, subsequent diagnosis of suspicious lesions as either benign or malignant ultimately results in a visual interpretation by radiologists. Clinically, human experience and expertise are applied to solving such problems using subjective, qualitative features. In comparison, Computer-Aided Diagnosis (CADx) systems employ the systematic processing of quantitative tumor features allowing for more reproducible descriptors. CADx systems have been used to diagnose lung nodules in thin-section CT,¹⁶ as well as prostate lesions in multiparametric MRI¹⁷ where inconsistencies in interpretation among human readers have been observed.^18,19 Characterization also includes staging where tumors are classified into predefined groups based on differences in their cancer’s appearance and spread, informative for expected clinical course and treatment strategies. The most widely used cancer staging system is the TNM classification^20,21 with other schemes applied for specific organs like the central nervous system (CNS). Recent studies have extended systems to perform staging by assessing tumor extent and multifocality in breast MRI²² while others have developed automated lesion volume measurement tools in contrast-enhanced magnetic resonance mammography (MRM).²³

An additional level of characterization interrogates the biological characterization of tumors. The emerging field of ‘imaging-genomics’ correlates radiographic imaging features with biological data, including somatic mutations, gene expression, chromosome copy number, or other molecular signatures. The maturity of genomics analyses, from a data-standpoint, provides synergistic opportunities for AI-based imaging efforts.²⁴

Lastly, AI can play increasing roles in monitoring changes in a tumor over time, either in natural history or in response to treatment. Traditional temporal monitoring of tumors have often been limited to predefined metrics including tumor longest diameter measured through the established Response Evaluation Criteria in Solid Tumors (RECIST) and World Health Organization (WHO) criteria for estimating tumor burden and determining treatment response. In addition to being criticized as oversimplifying the complex tumor geometry captured through sophisticated imaging instruments,²⁵ the generalizability and efficacy of such criteria has been questioned as in the case of osseous lesions where chemotherapy - which has proven to improve survival - does not result in radiographic responses as measured by RECIST.²⁶ AI-based monitoring, however, is able to capture a large number of discriminative features across images over time that go beyond those measured by human readers. While the seemingly disparate constituents of computer-aided monitoring are active areas of research: computer-aided registration of temporal images, segmentation, and diagnosis, the field is still in its infancy with applications yet to surface.

In addition to imaging, other minimally invasive biomarkers are also being developed for cancer diagnosis and longitudinal tracking of disease. Most notably, liquid biopsies, or the analysis of circulating tumor DNA (ctDNA) released from tumor cells, provide a window into the current and dynamic state of a cancer,²⁷ and allows tracking of disease progression or regression and monitoring for the emergence of targetable or resistance-associated cancer mutations in near real-time.^28–30 It is thus conceivable that liquid biopsies combined with radiomics profiling may significantly improve cancer treatment through non-invasive characterization of cancer biology for more accurate assessment of prognosis and real-time disease monitoring for precision medicine purposes.

Within the clinic, the aforementioned AI interventions are expected to augment their respective current standard of care counterparts (Figure 2). In addition to supporting clinicians with assistive information, multiple efforts have also demonstrated the utility of AI in the clinical decision making phases of the workflow.³¹ With AI-based integrated diagnostics, combining molecular and pathological information with image-based findings will add rich layers of intelligence to the findings, eventually leading to more informed decision-making.

Figure 2. Near future impression of an enhanced clinical workflow with AI interventions.

The traditional paradigm for tumor patients entails initial radiologic diagnosis of a mass lesion, a decision to treat or observe based on clinical factors and patient preference, definitive histopathologic diagnosis only after obtaining tissue, molecular genotyping in centers with such resources, and determination of clinical outcome only after the passage of time. In contrast, AI-based interventions offer the potential to augment this clinical workflow and decision-making at different stages of oncological care. Continuous feedback and retraining from measured outcomes may further improve AI systems.

LUNG CANCER IMAGING

Lung cancer is a leading cause of cancer-related death among men and women globally.³² Despite improvements in survival over the last several decades for most cancer types, lung cancer is falling behind, mainly because the cancer is often well advanced with limited treatment options by the time it is detected.³³ The fact that the majority of patients who are diagnosed with lung cancer will die from their disease can be attributed to the late stage at diagnosis. Medical imaging and AI are expected to play an important role in improving early detection and characterization of lung cancer by differentiating benign from malignant nodules. As early stages are often curable, this could drastically improve patient outcomes, minimize overtreatment, and even save lives. Furthermore, AI can also enhance lung cancer staging and characterization for treatment selection, as well as monitoring treatment response (Table 1).

Table 1.

Summary of key studies on the role of AI in imaging of lung cancer, as applied to detection, diagnosis and characterization, and predicting prognosis and treatment response.

Year	First Author	Citation	Tumor(s) studied	Application	No. cases	Imaging modality	Machine Learning Algorithm	Imaging Feature Type	Type of validation	Results
Cancer Detection
2016	Hawkins et al.	J Thorac Oncol. 2016 Dec;11(12):2120–2128.	NSCLC	Risk of lung cancer in screening/early detection	600	CT	Random forests classifier	Predefined Radiomic Features	independent validation within ACRIN 6684	AUC 0.83
2017	Liu et al.	Clinical Cancer Research. 23 (6): 1442–49.	NSCLC	Predict lung cancer in indeterminate pulmonary nodules	172	CT	Multiple supervised technique	Semantic	independent validation with single center data	AUC 0.88, ACC 81%, Sn 76.2%, Sp 91.7%
2017	Ciompi et al.	Sci Rep. 2017 Apr 19;7:46479	Benign vs Malignant lung lesions	Predict lung cancers in screening	1411	CT	SVM	Deep Learning Radiomics	independent validation with multi-center data	ACC 73%
Diagnosis and Characterization
2014	Yamamoto et al.	Radiology. 2014 Aug;272(2):568–76.	NSCLC	Discriminate ALK+ from non-ALK tumors	172	CT	Random forests classifier	Semantic	independent validation with multi-center data	Discriminatory power for ALK+ status: Sn 83.3%, Sp 77.9%, ACC 78.8%
2015	Maldonado et al.	Am J Respir Crit Care Med. 2015 Sep;192(6):737–44.	Lung adenocarcinoma	Differentiate indolent vs. aggressive adenocarcinoma	294	CT	Previsously build CANARY model	Semantic (Canary)	independent validation with single center data	Progression-free survival curve HR (P < 0.0001).
2017	Grossmann et al.	Elife. 2017 Jul;6.	NSCLC	Predict molecular and cellular pathways	351	CT	SVM	Predefined Radiomic Features	independent validation with multi-center data	Autodegration pathway prediction (AUC 0.69, p<10–4). Prognostic biomarkers combining radiomic, genetic, and clinical information (CI 0.73, p<10–9)
2017	Rios Valazquez et al.	Cancer Res. 2017 Jul;77(14):3922–3930.	NSCLC	Predict mutational status	763	CT	Random forests classifier	Predefined Radiomic Features	independent validation with multi-center data	EGFR+ and EGFR- cases (AUC 0.69); EGFR+ and KRAS+ tumors (AUC 0.80)
Predicting Treatment Response and Prognosis
2014	Aerts et al.	Nat Commun. 2014 Jun 3;5:4006	NSCLC, Head and neck cancer	Prognostic biomarkers	1019	CT	Regression	Predefined Radiomic Features	independent validation with multi-center data	Prognostic CI 0.70, CI 0.69
2015	Coroller et al.	Radiother Oncol. 2015 Mar;114(3):345–50.	Lung adenocarcinoma	Predict distant metastasis	182	CT	Regression	Predefined Radiomic Features	independent validation with single center data	CI 0.61, p=1.79 ×10−17.
2018	Sun et al.	Lancet Oncol. 2018 Sep;19(9):1180–1191.	NSCLC	Predict the immune phenotype of tumours and clinical outcomes	491	CT	Regression	Predefined Radiomic Features	independent validation with multi-center data	AUC 0·67; 95% CI 0·57–0·77; p=0·0019

Legend: ACC, accuracy; AUC, area under curve; CI, Concordance Index; HR, hazard ratio; NSCLC, non-small cell lung cancer; Sn, sensitivity; Sp, specificity; SVM, support vector machine;

Clinical Applications of AI in Lung Cancer Screening.

Until recently, a method to detect early stage lung cancer has been elusive even among high-risk populations. The National Lung Screening Trial (NLST) found that screening with low-dose CT (LDCT) was associated with a significant 20% reduction in overall mortality among high-risk current- and former smokers.³³ Lung cancers identified at early stage, whether by LDCT screening or incidentally, are more amenable to surgical cure and improved survival outcomes compared to lung cancers that are detected upon presentation with clinical symptoms, which are more frequently at a later stage of disease.³⁴ Though the emergence of immune checkpoint inhibitors and targeted therapies have demonstrated durable long-term survival in subsets of patients, not all patients benefit from such treatment modalities; thus, early detection has the benefit of improving patient survivals and may limit the need for extensive treatment. Based on these NSLT findings, annual LDCT is now recommended for high-risk individuals, and is second only to primary prevention (smoking cessation) for mitigating lung-cancer mortality, especially for those who have quit smoking but remain at risk. Although the NLST demonstrated a clear benefit for reducing all-cause mortality, many limitations are associated with early detection of lung cancer that could be enhanced with advanced computational analyses.^33,35–38 In the following sections, we describe current problems and limitations in lung cancer screening, how conventional methods are attempting to overcome these limitations, and how AI can improve these areas.

Lung cancer screening frequently identifies large numbers of indeterminate pulmonary nodules of which only a fraction are diagnosed as cancer (Figure 3). In the NLST, 96.4% of the pulmonary nodules identified in LDCT screens were not cancerous. At present, there are no established approaches to classify whether these nodules are cancerous or benign. Another potential harm of lung cancer screening is overdiagnosis of slow growing, indolent cancers that that may pose no threat if left untreated. As such, it is imperative that overdiagnosis needs to be recognized, identified and significantly reduced.³³ Next, if a nodule is detected, clinical guidelines provide for the evaluation and follow-up of nodules,³⁹ but do not offer decision tools for diagnostic discrimination and to predict risk and probability of future cancer development. Though conventional biostatistics and machine learning approaches have utilized to address many of limitations in lung cancer screening, AI has the potential supplant such approaches to identify biomarkers that to reduce imaging false positives and more accurately differentiate between benign and cancerous nodules. This can lead to a more quantitative prediction of lung cancer risk and incidence, leading to robust, better-defined clinical decision guidelines.

Figure 3. Clinical applications of AI in lung cancer screening on detection of incidental pulmonary nodules.

Imaging analysis show promise in predicting the risk of developing lung cancer on initial detection of an incidental lung nodule and in distinguishing indolent from aggressive lung neoplasms. Abbreviations: ROC, Receiver operating characteristic; PFS, Progression-Free Survival

The majority of indeterminate pulmonary nodules are incidentally detected; i.e., they are not encountered during screening, but in routine cross-sectional imaging for other diagnostic indications, such as CT angiography,⁴⁰ and pose a dilemma to patients and their providers. Annually, over 1.5 Americans are diagnosed with an incidentally detected nodule⁴¹; while most of these nodules are benign granulomas, up to 12% may be malignant ⁴². The Fleischner Society⁴³ and the ACR Lung CT Screening Reporting and Data System (Lung-RADS)⁴⁴ provide recommendations for the follow-up and management of these incidentally detected nodules, which usually requires follow-up imaging between 3 to 13 months to confirm growth prior to intervening with more invasive diagnostics (e.g., biopsy). These systems are “semantic” in that they describe features that are commonly used in the radiology lexicon to describe regions of interest by human experts. Because they are scored manually, there is high potential for large inter-reader variability.⁴⁵ In a recent study, a model incorporating four quantitatively scored semantic features (short axis diameter, contour, concavity, texture) conferred an accuracy of 74.3% to distinguish malignant from benign nodules in the lung cancer screening setting⁴⁶ A separate study was conducted to identify semantic features from small pulmonary nodules less than 6 mm to predict lung cancer incidence in the lung cancer screening setting and revealed final model yielded an AUROC of 0.930 based on total emphysema score, attachment to vessel, nodule location, border definition, and concavity.⁴⁷ While there was an imbalance between malignant and benign nodules in these aforementioned analyses, these studies provide compelling evidence for the utility of semantic features in lung cancer screening. As with nodules detected in the lung cancer screening setting, the standard of care for incidental pulmonary nodules lacks accurate decision tools in predicting malignancy versus benign disease and indolent versus aggressive behavior. Thus, appropriate management of incidental pulmonary nodules is dictated by the probability of cancer and the potential aggressiveness of its behavior. Prediction of the nature of a nodule may justify diametrically opposing strategies, such as biopsy versus observation. Erroneous prediction carries significant consequences, including risk of premature mortality from delayed intervention on one hand and morbidity and mortality resulting from invasive testing on the other lung cancer screening also detects cancers that exhibit a wide spectrum of behaviors, with some being clinically indolent and others being aggressive mandating prompt treatment. One study estimated that more than 18% of all lung cancers detected by LDCT in the NLST seem to be indolent.⁴⁸

In 2017, the Arnold Foundation supported a $1 million prize for the automated lung cancer detection and diagnosis challenge. In this challenge, thousands of annotated CT images from the NCI Cancer Imaging Archive (TCIA) were provided to the community to train and validate their models. All of the top teams used convolutional neural networks (CNNs) to both automatically detect and diagnose lesions and the winners had to make their network model publicly available.⁴⁹ The winning team reported a high performance (log loss=0.399; where a perfect model would have a log loss of 0). While this is encouraging, it is notable that the winning networks require more detailed evaluation of their performance in clinical settings. Furthermore, there was a significant bias with a 50% cancer prevalence in this challenge was higher than the 4% prevalence in a screening population with indeterminate nodules. Although this challenge identified promising methods, it is likely that significant fine-tuning is required before they can have any clinical use.

Though incidence of lung cancer is declining in the United States and most Western Nations,⁵⁰ lung cancer will remain a major public health burden for decades to come. Even after smoking cessation, former smokers remain at increased risk, especially compared to lifetime never smokers, of developing lung cancer. As such, improvements in lung cancer screening will be remain relevant and important to improve patient outcomes of this disease. As lung cancer imaging research as evolved from conventional biostatics, to machine learning, to deep learning, we contend that AI could emerge next to develop clinically adoptable approaches, precisely identify those at-risk, improve risk prediction of future cancer incidence, discriminate malignant from non-malignant nodules, and distinguish indolent tumors versus biologically aggressive cancers.

Characterizing Lung Cancers using Imaging.

Lung cancers exhibit a wide spectrum of behaviors, with some being clinically indolent and others being aggressive mandating prompt treatment. Although there are prognostic factors associated with better survival (such as female gender, tumors harboring an EGFR mutation, early stage disease, no regional lymph node involvement, and good performance status)⁵¹ as well as factors associated with poor survival (e.g., poor pulmonary function, presence of cardiovascular disease, male gender, current smoking status, advanced age, and late stage tumor)^{52–56,57–61} these factors have limited clinical utility to address the heterogeneous, dynamic nature of cancer as a “moving target”. Specifically, a tumor lesion is constantly evolving and diversifying, modifying its phenotype and genomic composition, and through metastatic spread, even its location. This is even more true when subjected to the selection pressure of therapeutic intervention, where cancer evolution rapidly explores and exploits resistance mechanisms, potentially even aided by the mutagenic nature of systemic cytotoxic chemotherapy, leaving the treating oncologist chasing a constantly changing disease.^62–64

Image-based biomarkers, on the other hand, can non-invasively and longitudinally capture the radiographic phenotype and characterize the underlying pathophysiology of a tumor. Due to the ease of clinical implementation, size-based measures, such as the longest diameter of a tumor (such as RECIST and WHO), are widely used for staging and response assessment. However, sized-based features and stage of disease have limitations as these metrics are associated with marked variability in outcomes and response. As such, research efforts have been successful to identify semantic features and automatic radiomic features to predict lung cancer patient outcomes.^1,65–68 For instance, the CANARY tool ⁶⁹ offers an semantic-based risk stratification to identify a potentially vulnerable subset of lung adenocarcinomas that harbor a more aggressive course. Preliminary work has shown that AI can quantify radiographic characteristics about the tumor phenotype automatically, and that this information is significantly prognostic in several cancer types including lung cancer (p<3.53×10–6),⁷⁰ as well as is associated with distant metastasis in lung adenocarcinoma (p=1.79×10(−17)),⁷¹ tumor histological subtypes (p-value=2.3 × 10(−7),⁷² and with underlying biological patterns including somatic mutations⁷³ and gene-expression profiles.⁷⁴

Assessing intratumor heterogeneity through medical imaging.

Medical imaging can also play an important role in quantifying intratumor characteristics of lung cancer. Sequencing studies where multiple independent samples from the same tumor have been analyzed have shown that intratumor heterogeneity (ITH) is a common feature among solid tumor cancers.⁷⁵ A tumor consists of billions of independent cancer cells. Low levels of DNA damage or changes in epigenetic regulation are introduced at each cell division, causing slight changes to the cancer cell genome that increase over time. When a change induces a selective advantage in a particular microenvironment, clonal expansion can give rise to a cancer subclone with all the cancer cells sharing a single recent common ancestor. Genomic intratumor heterogeneity, defined as the coexistence of independent cancer subclones within the same tumor, is associated with poor prognosis in non-small cell lung cancer (NSCLC) and clear cell renal cancer (ccRCC).^75–78 However, tumor subclones may be spatially separated and can carry significantly different mutation loads, ranging from highly homogeneous to more than 8,000 heterogeneous mutations differing between individual regions in the same tumor.^76–79

Intratumoral heterogeneity analysis has shown that while targetable somatic alterations may appear clonal in a single tumor biopsy, they may be entirely absent in additional biopsies from different regions of the same tumor.^76,80,81 This evidence that phenotypic diversification exists within tumors has ramifications for the application of precision medicine techniques based on molecular characterization of tissue from single region biopsies. Because targets found in single tumor biopsies may be subclonal, therapies against them would only be effective against a subset of the cancer cells, leaving cancer subclones without the target unharmed (Figure 4). Different strategies have been proposed to quantify ITH in the clinical setting, including multi-region sequencing of the primary tumor, analysis of circulating tumor DNA (ctDNA), and utilizing medical imaging data.^27,75,82 Unfortunately, while multi-region sequencing provides improved measures of the extent of ITH compared to single-sample analysis;^83,84 it requires a high quality tumor specimen of sufficient size and remains subjected to potential sampling bias with the potential of missing important cancer subclones due to incomplete sampling of the tumor in its entirety.

Figure 4. Applications of non-invasive monitoring during the course of cancer evolution.

Cancers share a common theme in developing intratumoral heterogeneity during their natural history. The presence of subclones (represented by different colors) confers significant implications in the response to treatment and may be difficult to capture through standard biopsies. Imaging and blood biomarkers during the disease monitoring offers a potential technological solution to detect the presence of intratumoral heterogeneity through space and time, and thereby, perhaps direct change in therapeutic strategies accordingly.

Medical imaging can play an important role in quantifying intratumor characteristics of lung cancer and improve the ability to capture and quantify intratumor heterogeneity. Further, as evolutionary fitness is contextual and dependent on the particular microenvironment, it is likely that these environments can be identified by imaging.⁸⁵ As with most tumor-based biomarkers, there are many limitations as they can be subjective to sampling bias due to the heterogeneous nature of tumors, the requirement of tumor specimens for biomarker testing, and, often, the assays can be timely, expensive, and require large amounts of tissue or tissue analytes.⁸⁶ In contrast, image-based features, guided by AI, are available in real-time from standard-of-care images, do not require timely and often expensive laboratory assay testing, are not subject to sampling bias and artifact, and, of course, are non-invasive. And image-based features represent the phenotype of the entire tumor in 3D and not just the portion that was subjected to biomarker testing (i.e., from biopsy).⁸⁶

AI for assessing response to targeted and immunotherapies.

The success of quantitative imaging endpoints based on the RECIST criteria paved the way for the development of AI in oncology because the widespread adoption of these endpoints as early indicators of survival in clinical trials generated large datasets of CT images with clinical meta-data. Retrospective analysis of these clinical trial datasets has been invaluable in meeting AI’s need for big data to enable training and validating AI algorithms, which otherwise might have been prohibited by the expense and effort necessary to generate these datasets from scratch. In part because of the success of RECIST, quantitative CT analysis is now the workhorse of contemporary oncology,⁸⁷ creating immediate translational potential for AI predictive models.

The strengths of AI are well suited to overcome the challenge posed by the current generation of targeted and immunotherapies, which can produce clear clinical benefit that is poorly captured by endpoints based on RECIST. These endpoints rely on the assumption that successful response to therapy will be reflected by tumor shrinkage, and in particular the measurement of response based on tumor diameter assumes that tumors are spherical and undergo uniform spatial change following treatment. Targeted and immunotherapies lead to novel patterns of response which confound current RECIST-based endpoints, and may contribute to the high failure rate of clinical trials and the cost of drug development. The ability of AI to quantify biological processes associated with response other than size thus answers an urgent need in the field.

Currently, response prediction for targeted and immunotherapies is based on biomarkers for immunogenic tumor microenvironment (e.g., PD-L1 expression) and mutational status (e.g., EGFR). These are acquired via biopsy, which is invasive, difficult to perform longitudinally, and limited to a single region of a tumor. The predictive value of PD-L1 expression may also be limited. For example, in the KEYNOTE-189 clinical trial, immunotherapy with pembrolizumab in combination with standard chemotherapy produced a survival benefit in all patients regardless of PD-L1 expression, even among those with a PD-L1 tumor proportion score of less than 1% which should indicate small chance of benefit.⁸⁸

A growing body of evidence suggests that AI could assess response to immunotherapy through recognition of radiomic biomarkers associated with response. Imaging phenotype was associated with overall survival (OS) in NSCLC following second-line treatment with anti-PD1 (nivolumab). In this study, OS was significantly (p = 0.005) predicted by two radiomics features at baseline, regions dissimilarity (HR = 0.11, 95% CI 0.03–0.46, p = 0.002) and entropy (HR = 0.20, 95% CI 0.06–0.67, p = 0.009), which indicate more heterogeneous primary lung tumor with irregular patterns of intensities on contrast enhanced CT. Another lung cancer immunotherapy study showed that prognosis of overall survival was improved by adding genomic and radiomic information to a clinical model, leading to an increase from CI = 0.65 (Noether p=0.001) to CI = 0.73 (p=2×10−9), and that the inclusion of radiomic data resulted in a significant increase in performance (p = 0.01).⁸⁹ These findings indicate that radiomic and genomic biomarkers are complementary, creating a potential role for artificial intelligence to elucidate predictive associations between their combined data. While machine learning has been deployed to genetically classify lung cancer based on the identification of patterns in microarray gene expression⁹⁰, its use to detect radiomic-genomic correlations predictive of outcome remains understudied ⁹¹

AI analysis of quantitative imaging data may also improve the assessment of response to targeted therapy. A decrease in FDG uptake by NSCLC tumors treated with bevacizumab, a monoclonal antibody against vascular endothelial growth factor, identified more patients who responded to treatment than conventional CT criteria (73% vs. 18%); in this study, neither PET nor CT was associated with overall survival (PET p = 0.833; CT p = 0.557) ⁹² Currently, predicting response to targeted therapy is driven largely by biopsy to assay the status of the mutation being targeted. AI predictive models could supplement this by identifying imaging phenotypes which are associated with mutational status. This approach has the advantage of being able to repeatedly and non-invasively characterize the mutational status of all tumors, not merely at the site of the biopsy, which can avoid the lack of predictive power associated with intra-tumoral heterogeneity and the emergence of distinct acquired resistance mechanisms in separate metastases within the same patient. Support for this approach comes from a quantitative imaging study of NSCLC patients treated with gefinitib which found that EFGR mutation status could be significantly predicted by the radiomic feature Laws-Energy (AUC = 0.67, p = 0.03) ⁹³

Biomarkers must be objectively and reproducibly measurable in order to serve as criteria for response assessment. AI affords high objectivity through its ability to characterize complex patterns within tumor images without the inter-observer variability associated with visual assessment by human experts. Understanding the measurement error of radiomic features is important to establish the reproducibility of AI predictive models based on them. Different tumor segmentation algorithms introduce variance known to affect the calculation of radiomic features and thus perhaps the performance of AI techniques which require semi-automatic segmentation.⁹⁴ Imaging settings including CT scanners, slice thickness, and reconstruction kernels also affect the calculation of radiomic features.^95,96 Variation in these settings exists within clinical practice and clinical trials and may affect the power and reproducibility of biomarkers developed by AI. The training and validation of CNNs may reduce this effect by selecting predictive features which are reproducible and discarding those which vary between image sets, but this needs to be proven. There is also a tension between the rapid pace of development in the AI field and the need for clinical trial endpoints to maintain historical consistency and achieve validation in large data warehouses before criteria are updated (e.g., RECIST 1.0 to 1.1). Continued progress in the size and appropriateness of public domain cancer datasets is necessary to meet the latter requirement.

CENTRAL NERVOUS SYSTEM TUMOR IMAGING

Central nervous system (CNS) tumors span a broad spectrum of pathologies, perhaps more diverse than any other organ system in the body. Among tumors arising from or seeding in brain parenchyma, metastases from systemic cancers and gliomas predominate. Additionally, a multiplicity of tumors arising from non-neural tissues that abut the brain are commonly encountered and must be considered within CNS tumors, including meningiomas, pituitary tumors, schwannomas, and lesions of the skull. This variegated diorama of diagnoses pose unique demands on clinicians for accurate assessment of imaging.

Three main challenges exist at present during the evaluation of radiologic studies for CNS tumors. First, accurate diagnosis of the type and extent of disease is tantamount to clinical decision making. Second, reliable tracking of neoplastic disease over time, especially following treatment with its associated effects on surrounding neural tissue which may acquire signal characteristics difficult to distinguish from tumor. Third, the ability to extract genotype signatures from the phenotypic manifestation of tumors on imaging, as the impact of molecular taxonomy becomes increasingly appreciated in influencing tumor behavior and clinical outcome.

The traditional paradigm for tumor patients entails initial radiologic diagnosis of a mass lesion, a decision to treat or observe based on clinical factors and surgeon or patient preference, definitive histopathologic diagnosis only after obtaining tissue, molecular genotyping in centers with such resources, and determination of clinical outcome only after the passage of time (Figure 2). Accurate extrapolation of pathologic and genomic data from imaging data itself, as being developed in the field of imaging-genomics, would disrupt this classic paradigm to improve guidance of tumor patients with more informed data upfront. Imaging-genomics may also shed light into reasons for treatment success and failure across a patient population and in multi-institutional clinical trials across heterogeneous populations. Furthermore, in locations around the globe with scarce access to expert neuroradiologists, limited encounter with rare CNS tumors, or lack of molecular profiling, computational analysis of imaging through shared network algorithms offers a potentially valuable resource to improve care to all brain tumor patients.

Diagnostic dilemmas in CNS.

Imaging plays an important role in the initial diagnosis of brain tumors and is a routine part of both initial and subsequent evaluation. The complex imaging features of brain tumors, as well as the frequent genetic heterogeneity within tumors types and the invasive nature of the procedures needed to obtain tissue diagnosis gives rise to diagnostic dilemmas in this field (Table 2).

Table 2.

Summary of key studies on the role of AI in imaging of CNS tumors, as applied to diagnosis, biological characterization, monitoring treatment response, and predicting outcome.

Year	First Author (Last name)	Citation	Tumor(s) studied	Application	No. patients	Imaging modality	Machine Learning Algorithm	Imaging / Radiomic Feature Type	Type of validation	Performance
Diagnosis
2015	Fetit et al.	NMR Biomed. 2015;28(9):1174–84	Medulloblastoma, pilocytic astrocytoma, ependymoma	Classification of CNS tumor subtype	48	MRI conventional	Multiple supervised techniques	Texture	leave-one-out cross validation single center	AUC 0.91–0.99
2017	Coroller et al.	PLOS One. 2017;12(11):e0187908	Meningioma	Differentiate grade I vs grade II-III	175	MRI conventional	Random forest	Radiomic and semantic features	independent validation with single center data	AUC 0.76–0.86
2017	Zhang et al.	Oncotarget. 2017;8(29): 47816–30	Glioma (WHO grade 2–4)	LGG (WHO grade 2) vs HGG (grade 3–4)	120	MRI conventional, perfusion, diffusion, permeability maps	SVM	Histogram, texture	leave-one-out cross validation	ACC 0.945
2018	Zhang et al.	Eur Radiol. 2018;28(9):3692–3701	Pituitary adenoma	Null cell adenoma vs other subtypes	112	MRI conventional	SVM	Intensity, shape, size, texture	independent validation with single center data	AUC 0.804
2018	Kang et al.	Neuro Oncol. 2018;20(9):1251–1261	Glioblastoma, lymphoma	Classify glioblastoma vs lymphoma	198	MRI conventional, perfusion, diffusion maps	Multiple supervised techniques	Volume, shape, texture	independent validation with multi-center data	AUC 0.946
Biological characterization
2016	Korfiatis et al.	Med Phys. 2016;43(6):2835–2844	Glioblastoma	MGMT methylation status prediction	155	MRI conventional	SVM, Random forest	Texture	cross validation single center	AUC 0.85, Sn 0.803, Sp 0.813
2017	Zhang et al.	Neuro-Oncology. 2016;19(1):109–17	Glioma (WHO grade 3–4)	IDH1/2 mutant vs wild-type	120	MRI conventional, apparent diffusion maps	Random forest	Histogram, texture, shape	independent validation with single center data	ACC 89%, AUC 0.923
2017	Zhou et al.	Neuro-Oncology. 2017;19(6): 862–70	Glioma (WHO grade 2–3)	1p/19q chromosomal status, IDH1/2 mutation status	165	MRI conventional	Logistic regression	VASARI features	boot strap validation single center	AUC 0.86
2018	Chang et al.	Clinical Cancer Research. 2018;24(5):1073–81	Glioma (WHO grade 2–4)	IDH1/2 mutant vs wild-type	496	MRI conventional, apparent diffusion maps	Deep learning Resnet	Histogram, texture, shape	independent validation with multi-center data	ACC 89%, AUC 0.95
Monitoring Treatment Response
2015	Larroza et al.	J Magn Reson Imaging. 2015;42(5):1362–8	Brain Metastases	Classify tumor vs radiation necrosis	73	MRI conventional	SVM	Texture	cross validation single center	AUC >0.9
2016	Tiwari et al.	AJNR Am J Neuroradiol. 2016;37(12):2231–2236	Glioma and brain Metastases	Classify tumor vs radiation necrosis	58	MRI conventional	SVM	Intensity, texture	independent validation with multi-center data	ACC 80%
2017	Kim et al.	Oncotarget. 2017;8(12):20340–20353	High-grade glioma	Classify tumor vs radiation necrosis	51	MR diffusion, perfusion, susceptibility weighted maps	Regression	Intensity, histogram	single center prospective trial without validation	Sn 71.9% Sn, Sp 100%, ACC 82.3% ACC
2017	Kebir et al.	Oncotarget. 2017;8(5):8294–8304	High-grade glioma	Classify tumor vs radiation necrosis	14	FET PET	unsupervised consensus clustering	Texture	single center retrospective trial without validation	Sn 90%, Sp 75% for detecting true progression, 75% NPV
Predicting Treatment Response and Survival
2016	Chang et al.	Neuro Oncol. 2016;18(12):1680–1687	Glioblastoma	Predict OS	126	MRI conventional, diffusion	Random forest	Shape, intensity histogram, volume, texture	single center data split into training/testing	HR 3.64 (P<0.005)
2017	Grossmann et al.	Neuro Oncol. 2017;19(12):1688–1697	Glioblastoma	Predict PFS and OS	126	MRI conventional	unsupervised principle component feature selection, random forest supervised training	Shape, volume, texture	multi-center phase 2 clinical trial data split into train/testing	OS (HR=2.5; P=0.001); PFS (HR=4.5; P=2.1×10–5)
2017	Liu et al.	Neuro Oncol. 2017;19(7):997–1007	Glioblastoma	Predict OS	117	MRI perfusion	Unsupervised consensus clustering	histogram of intensity	single center data split into training/testing	HR >3, P<0.01

Legend: ACC, accuracy; AUC, area under curve; HR, hazard ratio; NPV, negative predictive value; OS, overall survival; PFS, progression-free survival; Sn, sensitivity; Sp, specificity; SVM, support vector machine; WHO, World Health Organization;

Validation categories: cross-validation = within own dataset, independent validation with single center data, independent validation with multi-center data

In the setting of gliomas, the most common malignant primary brain tumors in adults, cross-sectional imaging technique such as CT and MRI provide high resolution spatial information as well as tissue contrast allowing radiologists to characterize different glioma subtypes and grades. AI can improve the utility of current standard diagnostic imaging techniques by refining preoperative classification of gliomas beyond what human experts can provide. For example, AI has been applied in the research setting to preoperative MRI to distinguish between low- and high- grade tumors, as well as individual WHO grades by training machine learning classifiers using image texture features obtained from spatially co-registered multi-modal MRIs (Table 2).⁹⁷ Furthermore, clinically-relevant molecular subtypes of gliomas such as the presence of an isocitrate dehydrogenase (IDH) mutation can be identified using machine learning methods including deep convolutional neural network (CNN) trained on conventional MR images.^98,99,100

Subtype classification problems are not unique to adult gliomas, however. Conceptually similar work has been done on other brain tumors where it has been shown that classification algorithms trained on radiomics features extracted from conventional MRI can generate predictive models for pituitary adenoma subtypes,¹⁰¹ and pediatric brain tumors (Table 2).¹⁰²

Diagnostic ambiguity can also arise when distinguishing between different tumor types as well. One key clinical dilemma is when differentiating between primary central nervous system lymphoma (PCNSL) and glioblastoma, which can have similar imaging phenotypes. Radiomics models, using image-based texture features, have been shown to enhance the differences between GBM and PCNSL.^103,104 Interestingly, a similar diagnostic dilemma often arises when evaluating histopathology slides of these same two disease processes; as CNN are being increasingly applied to histopathology image classifications in research studies across the globe,¹⁰⁵ we expect robust predictive models to emerge addressing this problem as well.

To date, most research applications of AI in brain tumors focus on addressing challenges in distinguishing between histopathologic and molecular subtypes of brain tumors.^100,104,106 To accomplish this, AI algorithms are trained using pre-selected patient populations with the specific tumor subtypes. This approach makes it challenging to integrate diagnostic models into clinical workflow since the model’s diagnostic accuracy can be consistent only when the testing population resembles that of the training data. With sufficient training data based on more general patient populations, it is likely that the diagnostic capability of AI will expand to include accuracy differentiation among multiple tumor types as well as non-tumor mimickers.

Tumor detection and delineation.

Synergistic with accurate diagnostic differentiation between tumor subtypes is the ability of computational algorithms to automatically detect the presence and extent of tumor itself. On MRI, the most common modality of delineating CNS neoplasms, tumors may manifest with variable levels of contrast-enhancement or none at all; be associated with peritumoral edema or hemorrhage; and blur in margins from adjacent bone, blood vessels, fat, or surgical packing materials. Furthermore, neural response to treatment, also known as pseudoprogression, contributes an additional layer of complexity in discerning tumor from non-tumor, as detailed below. While these features challenge the automatic detection of CNS tumors, the need to develop robust volumetric algorithms for analysis of tumor and its adjacent microenvironment remains vital.

An escalating cascade of studies and methodologies for semi-automatic and automatic detection of CNS tumors have been published in recent years, largely applied to conventional MR imaging, but also for PET and ultrasound images.^107–114 While these are most frequently used in the exploratory and research setting, semi-automatic algorithms have been applied to treatment planning for stereotactic radiosurgery,¹¹² quantitating volume of residual tumor after surgery,¹¹⁴ and tracking tumor growth over time.¹¹⁰ One can envision the benefits of a robust automatic tumor detection algorithm in the assessment of patients with numerous intracranial lesions, such as in the setting of CNS metastases, and their differential growth rate or response to treatment over time. Likewise, in skull base lesions, which are often irregularly shaped and extend across intracranial and extracranial compartments, automatic volumetric reconstruction may detect sensitive changes in growth that are missed by the casual observer.

The near universal accessibility of computational tools for image analysis and sharing of open source code by a number of researchers promises to accelerate the pace of advancement in this field.¹¹⁵ Additionally, publicly available imaging databases offer powerful resources for hypothesis testing and validation, including the Multimodal Brain Tumor Image Segmentation challenge (BRaTS) data from the Medical Image Computing & Computer Assisted Intervention (MICCAI) group, The Cancer Imaging Archive (TCIA), and the Ivy Glioblastoma Atlas Project.¹¹⁶ Ultimately, the fruit of such efforts will hopefully develop tools that minimize inter-observer variability in tracking tumors across time and treatments, and extract deeper layers of data beyond radiographic phenotype from routine imaging for CNS tumors.

Monitoring response to treatment.

In 20–30% of patients with glioblastoma receiving standard upfront radiation with adjuvant temozolomide, enlargement of contrast-enhancing lesion(s) that stabilize or resolve without changes in treatment are observed and termed pseudoprogression.¹¹⁷ Similarly, approximately 25% of CNS metastases develop necrosis within the irradiated field manifesting as enlarging enhancement that mimics recurrent tumor following stereotactic radiosurgery of brain metastasis.^117,118 Despite many conventional or advanced imaging techniques that have been investigated to distinguish true tumor from treatment-related changes, it remains challenging to characterize spatially heterogeneous tissues that often contains both viable tumor and treatment-related changes. Combining multiple imaging features using machine learning approaches can improve the ability to construct an accurate tissue classifier that can account for the heterogeneity of treated tumor. Texture features extracted from conventional MRI have been identified to distinguish radionecrosis from recurrent brain tumors.^119,120 Perfusion-weighted and susceptibility weighted MRI sequences can also be combined to differentiate recurrence from radionecrosis in high-grade glioma patients.¹²¹ Texture analysis has also been applied to amino acid PET imaging to diagnose pseudoprogression.¹²² To provide a more direct historical correlation of tumor and necrotic tissues, voxel-based evaluation of MRI coregistered to sites of stereotactic biopsy has resulted in a parametric model that correlates with cell counts of the biopsied specimens.¹²³ Overall, most of this research are in the phase of moving from pilot data to validation in clinical trials. Only upon more rigorous proof of the clinical utility of such technology can regulatory approval and commercialization be achieved and then dissemination into widespread clinical use.

Biologic characterization of CNS tumors: prospects and promise.

A molecular taxonomy is being defined for most common CNS tumors with the wide availability and decreasing cost of next-generation sequencing. Furthermore, molecular signatures are found to confer prognostic implications beyond standard histopathologic classifications, including for adult and pediatric gliomas, meningiomas, pituitary tumors, craniopharyngiomas, medulloblastomas, and ependymomas. These molecular imprints increasingly guide frequency of surveillance imaging for a tumor, patient consultation for clinical outcome and recurrence risk, and decisions on the type of treatment to administer (e.g. radiation or observation).^124–126 However, such information is largely determined only after tissue sampling of the tumor after an intervention. Additionally, as with systemic cancers, brain tumors harbor incredible molecular heterogeneity within an individual tumor and on recurrence using multi-focal sampling and single cell sequencing strategies. Such heterogeneity likely contributes to the limited effectiveness of current pharmacotherapeutics against brain tumors and the perceived acquired resistance after period of apparent disease control. Therefore, a non-invasive method of tracking tumor genotype over time which can capture the entire landscape of tumor heterogeneity offers appeal.

Radiomic analysis of CNS tumor imaging has the potential to characterize the phenotype of the entire tumor, rather than a core of the tumor as is frequently sampled for molecular analysis, and provides a non-invasive window into the internal growth pattern of the tumor. Previous works have reported significant connections between imaging features, molecular pathways, and clinical outcomes across brain tumors. The behavior of gliomas is significantly associated with their molecular alterations, especially alterations in isocitrate dehydrogenase 1/2 (IDH1/2), epidermal growth factor receptor (EGFR), O⁶-methylguanine-DNA methyltransferase (MGMT), and chromosomes 1p and 19q. The World Health Organization recognized the significance of molecular stratification in gliomas in its 2016 update on the classification of gliomas.¹²⁷ Machine learning algorithms trained on preoperative MR images have been able to distinguish each of these features with 80–95% sensitivity and specificity, including prediction of glioblastoma subtypes and survival,¹²⁸ IDH mutation status in high and low-grade gliomas,^98,100 presence of chromosome 1p and 19q loss in low-grade gliomas,^98,129 MGMT methylation state,¹³⁰ EGFR amplification status,¹³¹ and presence of EGFR receptor variant III¹³² as well as EGFR extracellular domain mutations (Figure 5). Moreover, unsupervised deep learning methods are showing promise in discerning molecular subgroups in glioblastoma with differential prognoses.¹³³

Figure 5. Grad-Cam visualizations (Selvaraju et al. 2017) for a convolutional neural network (Chang et al. 2018) applied to 2 examples of IDH1/2 wild type glioblastoma and 2 examples of IDH1 mutant glioblastoma.

Color maps are overlaid on original gadolinium enhanced T1-weighted MRI with red color weighted to the discriminative regions for IDH status classification.

In meningioma, benign variants (grade I) most commonly carry a mutation in one of several putative oncogenic drivers while high-grade variants (grade II-III tumors) harbor a variable number of chromosomal alterations. Radiomic analysis of pre-operative MRI from meningioma patients revealed the ability of computer-extracted imaging features to strongly associate with meningioma grade ¹⁰⁶ and also certain genomic features (Bi, Aerts, et al, unpublished data). Additionally, quantitative radiomic features could discern subtleties, such as the number of atypical features associated with grade I meningiomas, beyond the capacity of qualitative radiologist-rated imaging features.¹⁰⁶

Similar radiomic analysis are developing for pituitary tumors,¹⁰¹ craniopharyngiomas, chordomas, and other CNS tumors. Beyond single tumor subtype analysis, future efforts need to improve the accuracy and sensitivity such that such methods can be applied to the clinical setting with confidence, derive more nuanced molecular signatures beyond that of a single or dual marker, and accommodate the artifacts associated with recurrent and post-treatment disease states to allow for truly longitudinal application of radiomics throughout the course of CNS tumor patients.

Clinical trial applications.

Predictive biomarkers can have important roles in clinical trials due to their ability to select patients who are more likely to respond to treatment and thereby, improve the chance of detecting clinical benefit and lower the risk of drug toxicity from ineffective therapies. The best known predictive biomarkers for treatment of glioblastoma is MGMT promoter status, in which methylated tumor subtypes have shown greater response to alkylating agents.^134,135 Recently, antiangiogenic treatment of newly diagnosed glioblastoma has been evaluated in two phase III clinical trials where bevacizumab, an antibody targeting vascular endothelial factor (VEGF), did not result in improved overall survival when added to the standard treatment.^136,137 Currently, there is no clinically useful molecular marker predictive of treatment response for antiangiogenic therapy. Imaging based biomarkers for treatment response prediction for newly-diagnosed recurrent glioblastoma have been investigated using both conventional and advanced modalities. Radiomic imaging predictors of response based on conventional imaging features have been identified using a retrospective single-center data set of patients with recurrent glioblastoma receiving bevacizumab treatment. In a retrospective evaluation of single and multi-institutional single-arm data sets, radiomic models were constructed using conventional and diffusion MRI features to differentiate long-term from short-term survivors.^138,139 Unsupervised clusters of radiomic features based on non-parametric parameters of preoperative perfusion MRI were first extracted independently from two data sets of patients with glioblastoma and subsequently the feature clusters were combined and evaluated for association with patient survival outcome.¹⁴⁰ The radiomic cluster that was associated with poor survival (HR > 3) was associated with mutations in the angiogenesis and hypoxia pathways. These preliminary investigations were based on patients receiving therapy without availability of a control treatment arm, and therefore only establish the prognostic values of these imaging markers.

There are several advantages of using clinical trial data, both retrospectively and prospectively, to screen and validate radiomic biomarkers. Since these patient populations are relatively uniform (including treatment regimens including type, dose and duration, as well as imaging assessment timing and frequency during pre-treatment and on-treatment periods), the predictive accuracy for patient outcome will likely improve. The predictive models constructed in this setting can more readily applied to future prospective trials that uses similar protocols. Recent effort in standardizing imaging acquisition protocol for brain tumor trials should also increase the generalizability of radiomic models to different clinical trials as well as to actual clinical implementation ¹⁴¹.

BREAST CANCER IMAGING

Breast cancer is the most commonly diagnosed cancer and the second most common cause of cancer death in U.S. women ¹⁴². The 5-year survival rates for breast cancer have improved tremendously since the 1980’s; this likely due to significant uptake of mammographic screening as well as improvements in breast cancer treatment. Breast cancer is a heterogeneous disease and tumors vary with respect to etiology, prognosis, and response to therapy. Presence of the estrogen receptor (ER) is important for response to specific treatments (e.g., tamoxifen for ER+ disease) and prognosis (poorer outcomes for ER- disease), and may define etiologic subtypes. Triple negative breast cancers (TNBC) are ER-, progesterone receptor negative (PR-), and HER2-. They do not present with the typical signs of malignancy on standard mammography,¹⁴³ are more likely to be detected as interval and high grade tumors, and have a poor 5-year survival rate.

Advances in both imaging and computers have synergistically led to a rapid rise in the potential use of AI in various tasks in breast imaging, such as risk assessment, detection, diagnosis, prognosis, and response to therapy (Table 3).

Table 3.

Summary of key studies on imaging characterization of breast lesions, including detection, diagnosis, biological characterization, and predicting prognosis and treatment response.

Year	First Author	Citation	Application	No. cases	Imaging modality	Machine Learning Algorithm (if applicable)	Imaging / Radiomic Feature Type	Results
Detection
1994	Zhang et al.	Medical Physics 1994; 21: 517–524	microcalcification detection	34	Mammography	Convolutional neural networks	Deep learning characterization followed by conventional image analysis	AUC 0.91
2006	Karssemeijer et al.	Br J Radiol 2006; 79:S123–26	mass lesions	500	Mammography	Engineered Algorithms	Engineered Algorithms	similar performance to radiologists
2006	Reiser et al.	Medical Physics 2006; 33:482–91	mass lesions	21	breast tomosynthesis	Engineered Algorithms	Engineered Algorithms	Sn 90%
2012	Sahiner et al.	Medical Physics 2012;39:28–39	microcalcifications	72	breast tomosynthesis	Engineered Algorithms	Engineered Algorithms	Sn 90%
Diagnosis
1998	Gilhuijs et al.	Medical Physics 1998;25:1647–1654	mass lesions	27	DCE-MRI	Engineered Algorithms	size, shape, kinetics	AUC 0.96
1999	Jiang et al.	Academic Radiology 1999;6: 22–33	microcalcifications	104	Mammography	Engineered Algorithms	size and shape of individual icrocalcifications and clusters	AUC 0.75
2007	Chen et al.	Magnetic Resonance in Medicine 2007;58: 562–571	mass lesions	121	DCE-MRI	Engineered Algorithms	uptake heterogenity in cancer tumors via 3D texture analysis	3D better compared to 2D analysis
2010	Bhooshan et al.	Radiology 2010; 254: 680–690	Differentiate benign vs DCIS vs IDC	353	DCE-MRI	Bayesian neural networks	size, shape, margin morphology, texture (uptake heterogenity), kinetics, variance kinetics	AUC 0.79–0.85
2010	Jamieson et al.	Medical Physics 2010;37: 339–351	mass lesions	1126	multi-modality: mammography, breast ultrasound, and breast DCE-MRI	t-SNE followed by Bayesian neural networks	multi-radiomic features in non-supervised data mining	AUC 0.88
2011	Nielsen et al.	Cancer Epidemiol. 2011; 35: 381–387	breast cancer risk	495	Mammography	−	Texture analysis	AUC 0.57–0.66
2016	Huynh et al.	J Medical Imaging 2016;3(3), 034501	mass lesions	219	Mammography	Deep learning	Feature extracted from transfer learning from pre-trained CNN	AUC 0.81
2017	Antropova et al.	Medical Physics online 2017	mass lesions	1125	multi-modality: mammography, breast ultrasound, and breast DCE-MRI	Deep learning	Fusion of human-engeineered computer features and those feature extracted from transfer learning from pre-trained CNN	DCE-MRI: AUC 0.89, FFDM: AUC 0.86; ultrasound: AUC 0.90
Biological characterization
2014	Gierach et al.	Breast Cancer Research 2014; 23: 16: 424.	BRCA1/2 mutation status	237	Mammography	Bayesian artificial neural network	Texture analysis	AUC 0.68–0.72
2016	Li et al.	npj Breast Cancer 2016;2:16012	Molecular subtype classification	91 (from TCGA)	DCE-MRI	Engineered Features, Linear discriminant analysis	Multi-radiomic tumor signature including size, shape, margin morphology, texture (uptake heterogenity), kinetics, variance kinetics	AUC 0.65–0.89
2017	Li et al.	J Med Imaging 2017; 4(4), 041304	BRCA1/2 mutation status	456	Mammography	Convolutional neural networks, computerized radiographic texture analysis, SVM	Texture analysis and deep learning	AUC 0.73–0.86
Predicting Treatment Response and Prognosis
2018	Drukker et al.	Cancer Imaging 2018;18:12	Prediction of recurrence-free survival	284 (from ACRIN 6657)	DCE-MRI	−	most-enhancing tumor volume	HR 2.28–4.81

Legend: ACC, accuracy; AUC, area under curve; DCE-MRI, dynamic contrast-enhanced MRI; DCIS, ductal carcinoma in situ; FFDM, Full Field Digital Mammography; HR, hazard ratio; IDC, invasive ductal carcinoma; Sn, sensitivity; Sp, specificity; SVM, support vector machine; TCGA, The Cancer Genome Atlas

Breast Cancer Screening: BIRADS analog to digital.

Computer-aided detection (CADe) and computer-aided diagnosis (CADx) in breast cancer imaging have been under development for decades.^144–146 CADe systems specifically for screening mammography interpretation have been in routine clinical use since the late 1990’s.^146,147 Detection of cancer by radiologists is limited by the presence of structure noise (camouflaging normal anatomical back­ground), incomplete visual search patterns, fatigue, distractions, the assessment of subtle and/or complex disease states, vast amounts of image data, and the physical quality of the breast image itself. In computer-aided detection, the computer aims to locate suspect lesions, leaving the classification to the radiologist.

While CADe continues to be developed for screening mammography, investigators have also looked to automate the detection of breast lesions on 3D ultrasound, breast MRI, and breast tomosynthesis images by incorporating pre-defined algorithms as well as novel deep learning methods.^148–151 The motivation for computerized detection on 3D breast images arose with the arrival of 3D ultrasound and MRI for use as adjunct imaging for screening women with dense breast tissue.¹⁵²

CNN’s have been utilized in medical image analysis in the early 1990’s for the detection of microcalcifications in digitized mammograms.^153,154,153 as well as for distinguishing between biopsy-proven masses and normal tissue on mammograms.¹⁵⁵. More recently, deep learning methods have allowed for the computer-aided detection of breast lesions in breast MRI, ultrasound, and mammography ^148–151.

Breast Cancer Risk Assessment: Density and Parenchyma.

Computer vision techniques have been developed to extract the density and characteristics of the parenchyma patterns on breast images to yield quantitative biomarkers for use in breast cancer risk prediction, and ultimately in personalized screening regimes.

Both area-based and volumetric-based assessments of density are used to estimate mammographic density since increased density serves as a breast cancer risk factor as well as providing a masking effect that obscures lesions.^156–158 Breast density refers to the amount of fibroglandular tissue in the breast relative to the amount of fatty tissue. In Full Field Digital Mammography (FFDM), these tissue types are distinguishable since fibroglandular tissue attenuates X-rays much stronger than does fat. Given that FFDMs are two-dimensional (2D) projections of the breast, 3D percent density values are estimated.

Besides breast density, there is also evidence that the variability in parenchymal patterns (e.g. characterizing the spatial distribution of dense tissue) are also related to breast cancer risk. Using radiomic texture analysis, investigators have characterized the spatial distribution of the gray-scale levels within regions on FFDM where a skewness measure was incorporated into the analysis of mammograms to describe the density variation.¹⁵⁶ Others have used texture analysis and deep learning to discriminate BRCA1/BRCA2 gene mutation carriers (or women with breast cancer in the contralateral breast) from women at low risk for breast cancer, and using almost 500 cases, found that women at high risk for breast cancer have dense breasts with parenchymal patterns that are coarse and low in contrast (AUC~0.82).^{159–161,159,160} Further efforts have applied texture analysis to breast tomosynthesis images to characterize the parenchyma pattern for ultimate use in breast cancer risk estimation, with preliminary results indicating that texture features correlated better with breast density on breast tomosynthesis (p=0.003 in regression analysis) than on digital mammograms.¹⁶²

Additionally, the characterization of the breast parenchymal pattern has also been extended to Breast Parenchymal Enhancement (BPE) on dynamic contrast-enhanced MRI.^163,164,165 On a limited dataset of 50 BRCA1/2 carriers, quantitative measures of BPE were found to be associated with the presence of breast cancer and that relative changes in BPE percentages were predictive of breast cancer development post risk-reducing salpingo-oophorectomy (p<0.05).¹⁶⁶ Deep learning methods are increasingly being evaluated to assess breast density as well as parenchymal characterization, an example of which includes the performance assessment of transfer learning in the distinction between women at normal risk of breast cancer and those at high risk based on their BRCA1/2 status.¹⁶⁷

AI to improve breast cancer diagnosis.

Since the 1980’s, various investigators have been developing machine learning techniques for computer-aided diagnosis (CADx) in the task of distinguishing between malignant and benign breast lesions.¹⁶⁸ These AI methods for CADx involve the automatic characterization of a tumor, which is indicated initially by either a radiologist or a computer. The computer characterizes the suspicious region or lesion and/or estimates its probability of disease, leaving the patient management to the physician.

With the application of AI methods to breast image data, characteristics of tumor size, shape, morphology, texture, and kinetics can be quantitatively obtained. For example, use of the dynamic assessment of contrast uptake on breast MRI allows investigators to quantify cancers in terms of heterogeneity, yielding phenotypes of spatial features and dynamic characteristics.^169,170 For example, entropy is a mathematical descriptor of randomness and gives information on how heterogeneous is the pattern within the tumor, thus, describing the heterogeneous pattern of the vascular system uptake (i.e., contrast uptake) within tumors imaged on contrast-enhanced breast MRI. Such analyses could potentially reflect the heterogeneous nature of angiogenesis and treatment susceptibility as shown by the NCI TCGA Breast Cancer Phenotype Group.¹⁷¹

With CADx, both pre-defined and deep-learned algorithms have been evaluated. It is interesting to note that investigators have shown that use of either pre-engineered or deep learning features perform well in the classification of breast lesions in the task of distinguishing between malignant and benign lesions, and that the “fusion” of the two methods can yield statistically significant improvement in performance.^172,173 Across all three breast imaging modalities (690 DCE-MRI cases, 245 FFDM cases, and 1125 ultrasound cases), the “fusion” classifier performed best indicating the potential for the complimentary use of both pre-engineered and deep learning tumor features in diagnostic breast cancer workup: DCE-MRI [AUC = 0.89 (se = 0.01)], FFDM [AUC = 0.86 (se = 0.01)], and ultrasound [AUC = 0.90 (se = 0.01)])¹⁷⁴. Other investigators have used transfer learning with CNNs pre-trained on 2282 digitized screen-film and FFDMs for use in characterizing tumors on 324 breast tomosynthesis volumes, which demonstrated the ability to transfer the knowledge of the imaged patterns between the imaging modalities.¹⁷⁵

Predictive image-based biomarkers.

Beyond CADe and CADx,⁴ other AI applications in breast imaging include assessing molecular subtypes, prognosis, and therapeutic response by yielding predictive image-based phenotypes of breast cancer for precision medicine. A major interest area within breast cancer research is the attempt to understand relationships between the macroscopic appearance of the tumor and its environment. These relationships can be extracted from clinical breast images, and the biological indicators of risk, prognosis, or treatment response. Such effective development of biomarkers benefits from the integration of information from multiple patient exams, i.e., clinical, molecular, imaging, and genomic data, i.e., the other –omics that are often obtained during diagnostic workup and subsequent biopsies.

In one collaborative effort through the NCI’s TCGA Breast Phenotype Group, multi-disciplinary investigators phenotypically characterized 84 solid breast tumors in order to gain image-based information on the underlying molecular characteristics and gene expression profiles (Figure 6).¹⁷⁶ Statistically significant associations were found between enhancement texture (entropy) and molecular subtypes (normal-like, luminal A, luminal B, HER2-enriched, basal-like), even after controlling for tumor size (P = 0.04 for lesions ⩽2 cm; P = 0.02 for lesions 42 to ⩽ 5 cm). MRI-genomic associations were also unveiled, furthering the understanding of genetic mechanisms that regulate the development of tumor phenotypes.^171,174

Figure 6. Significant associations between Genomic Features and Radiomic Phenotypes in breast carcinoma imaged with MRI.

Gene-Set Enrichment Analysis (GSEA) and linear regression analysis were combined to associate genomic features, including miRNA expressions, protein expressions, and gene somatic mutations among others, with six categories of radiomic phenotypes. In this figure, each node represents a genomic feature or a radiomic phenotype. Each line is an identified statistically significant association, while non-significant associations are not visualized. Node size is proportional to its connectivity relatively to other nodes in the category. Reused with permission from Maryellen L. Giger, Chicago University ¹⁷¹.

With regards to predicting a patient’s response to a particular therapeutic treatment, for example, the semi-manual delineation of functional tumor volume (FTV) from breast MRI (141 women: 40 with arecurrence, 101 without) has been shown to be a predictor of recurrence-free survival on patients undergoing neoadjuvant therapy in the ACRIN study 6657,¹⁷⁷ with demonstrated automation potential.¹⁷⁸

PROSTATE CANCER IMAGING

Prostate cancer is the most frequently diagnosed non-cutaneous male malignancy and the second leading cause of cancer-related mortality among men in the United States.³² Statistics of prostate cancer frequency, morbidity, and mortality can be examined in many different ways. It is very common cancer as it is a “tumor of aging”, but has a very low disease specific mortality, all of which reinforce that it is a complex public health concern that impacts a large population. While prostate cancer is a serious disease, most men diagnosed with prostate cancer do not die from it.¹⁷⁹ The 5-year survival rate of prostate cancer patient ranges from about 30% in patients with metastatic disease to 100% in patients with localized disease. The key clinical problems in prostate cancer diagnosis today include: (1) overdiagnosis and overtreatment resulting from inability to predict the aggressiveness and risk of a given cancer and (2) inadequate targeted biopsy sampling, leading to misdiagnosis and to disease progression in men with seemingly low-risk prostate cancer. In a meta-analysis,¹⁸⁰ the rate of over-diagnosis of non-clinically significant prostate cancer is reported to be as high as 67%, leading to unnecessary treatment and associated morbidity. Due to this range of clinical behavior, it is necessary to differentiate clinically significant tumors (those with biopsy Gleason score GS≥7 and/or pathological volume ≥ 0.5 cc)¹⁸¹ as candidates for therapy, from clinically insignificant tumors that can safely undergo active surveillance. It has been noted that potential survival benefits from aggressively treating early-stage prostate cancer are undermined by harm from unnecessary treatment of indolent disease.

The biological heterogeneity of prostate cancer leads to different clinical putomes, ranging from indolent to highly aggressive tumors with high morbidity and mortality, and differences in therapy planning, therapy response, and prognosis of patients. This is reflected by incorporation of genomic profiling in the National Comprehensive Cancer Network (NCCN) guidelines, including Decipher, Oncotype DX Prostate, Prolaris, and others. In parallel with molecular characterization, AI also has the potential to empower clinicians in the detection, localization, characterization, staging, and monitoring of prostate cancer. There are no widespread multi-center trials as yet, so much of the initial work is limited to single center, single algorithm analyses and on small data sets. However, some groups, such as NIH and MICCAI, are developing infrastructure to allow for larger, well-annotated data sets to become available for AI development.

Computational methods mostly based on supervised machine learning have been successfully applied to imaging modalities such as MRI and ultrasound to detect suspicious lesions and differentiate clinically significant cancers from the rest. Recent application of deep learning in prostate cancer screening and aggressive cancer diagnosis has produced promising results.

Multi-parametric Magnetic Resonance Imaging (mpMRI) provides the required soft tissue contrast for detection and localization of suspicious clinically significant prostate lesions and gives information about tissue anatomy, function, and characteristics. Importantly, it has superior capabilities to detect the so-called “clinically significant” disease- one with Gleason pattern 4 or higher (Gleason score 7 or higher) and/or tumor volume > 0.5 cm3. Recent years have seen a growth in the volume of mpMRI examination of prostate cancer due to its ability to detect these lesions and allow targeted biopsy sampling. A large population study from the UK suggested that use of mpMRI as a triage before primary biopsy can reduce the number of unnecessary biopsies by a quarter and decrease overdiagnosis of clinically insignificant disease.¹⁸² This was further validated in the and on smaller data sets than would be optimal. In the multinational PRECISION study of 500 patients,¹⁸³ men randomized to mpMRI prior to biopsy experienced a significant increase in detection of clinically significant disease over the current standard of care, which employs a 10–12 core transrectal ultrasound guided biopsy (38% vs 26%).

The growing trend towards mpMRI has introduced a demand for experienced radiologists to interpret the exploding volumes of oncological prostate MRIs. Furthermore, reading challenging cases and reducing the high rate of interobserver disagreements on findings is a remaining challenge for prostate MRI. In 2015, the European Society of Urogenital Radiology, American College of Radiology and AdmeTech foundation published the second version of Prostate Imaging Reporting and Data system (PI-RADS™). These provide guidelines for radiologists in reading and interpreting the prostate mpMRI, which aim to increase the consistency of interpretation and communication of mpMRI findings. Over the past ten years, AI models have been developed as CADe and CADx systems to detect, localize and characterize prostate tumors.¹⁸⁴ In conjunction with PI-RADS, accurate CAD systems can increase the inter-rater reliability and improve the diagnostic accuracy of mpMRI reading and interpretation.¹⁸⁵ In preliminary analyses, it has been shown that addition of a CADx system can improve the performance of radiologists in prostate cancer interpretation.^186,187

Preliminary work in mpMRI CADx systems focused primarily on classic supervised machine learning methodologies, including combinations of feature extractors and shallow classifiers. In this category of AI systems, feature engineering plays a central role in the overall performance of the CAD system. Combinations of CADe and CADx systems have been reported that use intensity, anatomical, pharmacokinetic (PK modeling), texture and blobness features.¹⁸⁸ Pharmacokinetics are the detailed metrics which can be extracted from a time signal analysis of IV contrast passing through a given volume of tissue. They include parameters such as wash-in and wash-out. Texture features are also signal based and depend heavily on the imaging techniques. Others used intensity features calculated from mpMRI sequences, including T2W, ADC, high b-value DWI and a T2-estimation map by proton density image,¹⁸⁸ or only using features extracted from PK analysis and DTI parameter maps.¹⁸⁹ Similar image-based features were included into CAD systems^190–193 and many of these systems use support vector machines (SVMs) for classification.^{189,191,194–196}

In the past years, deep learning networks, and in particular convolutional neural networks (CNNs), have been revolutionizing investigative research into prostate cancer detection and diagnosis. These methods use different modality types, CNN architectures, and learning procedures to train deep networks for prostate cancer classification and have achieved state-of-the-art performance. Some investigators use CNNs to classify MRI findings with auto-windowing mechanism to overcome the high dynamic range of MR images and normalization,¹⁹⁷ while others use different combinations of mpMRI images by stacking each modality as a 2D channel of RGB images and used them as training examples.^198,199 Furthermore, 3D CNN can be designed that use specific MRI based parameters such as Apparent diffusion coefficient (ADC) high b-value and K^Trans modalities.²⁰⁰

Deep learning systems have been applied to localize and classify prostate lesions at the same time.²⁰¹ Both de novo training ^198,200,201 and transfer learning of pre-trained models¹⁹⁹ have been successful for training CNNs for prostate cancer diagnosis in MRI. Explicit addition of anatomically-aware features to the last layers of CNNs has been used successfully to boost the performance of CNNs.^197,200 In addition to MRI, AI techniques have achieved promising results by incorporating ultrasound data, specifically radio frequency (RF) for prostate cancer classification. Here again, both classic machine learning approaches^202,203 and deep learning²⁰⁴ have been used to train classifiers to grade prostate cancer in temporal ultrasound data.

The results of the ongoing research in the use of AI for detection and characterization of prostate cancer are promising and demonstrate ongoing improvement. The recent body of research in prostate cancer image analysis shows a transition from feature engineering and classic machine learning methods towards deep learning and use of large training sets. Unlike lung and breast cancers, clinical routines in prostate cancer have not yet adopted regulated CAD systems. However, the recently achieved results of deep learning techniques on mid-size datasets such as the PROSTATEx benchmark are promising. As it is now evident there has been a rapid growth in prostate MR exam volumes worldwide and increasing demand for accurate interpretations. Accurate CAD systems will improve the diagnostic accuracy of prostate MRI readings which will result in better care for individual patients, as fewer patients with benign and indolent tumors (false positives) will need to undergo invasive biopsy and/or radical prostatectomy procedures which can lower their quality of life. On the other hand, early detection of prostate cancer improves the prognosis of patients with clinically significant prostate cancer (Gleason pattern 4). Computer-assisted detection and diagnosis systems of prostate cancer help clinicians by potentially reducing the chances either missing or overdiagnosing suspicious targets on diagnostic MRIs, although this merits additional validation in trials before routine clinical incorporation.

CHALLENGES AND FUTURE DIRECTIONS

Despite the reported successes of AI within cancer imaging, several limitations and hurdles must be overcome before widespread clinical adoption. With the increasing demand for CT²⁰⁵ and MR²⁰⁶ imaging, care providers are constantly generating large amounts of data. Standards including the Picture Archiving and Communication System (PACS) as well as the Digital Imaging and Communications in Medicine (DICOM) have ensured that these data are organized for easy access and retrieval. However, such data is rarely curated in terms of labelling, annotations, segmentations, quality assurance, as well as fitness for the problem at hand. The curation of medical data represents a major obstacle in developing automated clinical solutions as it requires trained professionals, making the process expensive in both time and cost. These issues are exacerbated with data-hungry methods including deep neural networks. Unsupervised²⁰⁷ and self-supervised²⁰⁸ methods do not require explicit labelling and hence promise to alleviate some of these issues, while synthetic data²⁰⁹ can potentially enable a faster route towards curation, address the inevitable class imbalance, as well as mitigate patient privacy concerns. Standardized benchmarking is of particular importance in the medical domain especially given the multitude of imaging modalities, anatomical sites, as well as acquisition standards and hardware. The research community has yet to reach consensus on specific datasets that can be used for comparing and contrasting efforts in terms of performance, generalizability, and reproducibility - although the amount of medical data being made public is an encouraging move forward.²¹⁰ Furthermore, access to available datasets should be improved to promote intellectual collaboration. Institutional, professional, and government groups should be encouraged to share validated data to support development of AI algorithms, which requires overcoming certain fundamental technical, legal, and perhaps ethical concerns.²¹¹ For example, NIH recently has shared chest x-ray and CT repositories to help AI scientists.²¹² Such efforts bear expansion to a much wider audience across disease states.

Another limitation includes the interpretability of AI, the ability to interrogate such methods for reasons behind a specific outcome, as well as the anticipation of failures. While the current state of research has prioritized performance gains over explainability and transparency, interpretability of AI is an active area of research.²¹³ The benefits of trust and transparency in AI systems will differ based on their performance: allow for identifying failures when AI is sub-human and consequently transform super-human AI into a learning resource. From a legal standpoint, policy makers have taken note: Discussions around improving AI accountability through explanations have recently been debated in the EU General Data Protection Regulation (GDPR),²¹⁴ and continue to surface in sensitive applications where explanation is currently required under the law - or anticipated to be in the near future.

From an ethical perspective, a question poses itself: What would a hippocratic oath for clinically deployed AI systems read and how would it be enforced. While data curation and modelling practices are bias in nature as they take into account specific patient cohorts, a conscious effort must be put in understanding who exactly will be the ultimate beneficiaries and stakeholders of such technology. Algorithms can be unethical by design, ²¹⁵ and might exacerbate the already existing tension between providing care and turning profits. Additionally, a safeguard against “learned helplessness” must be employed as a means to curb high reliance on automation and the ultimate abandonment of common sense. Finally, automated systems might also challenge the dynamics of responsibility within the doctor-patient relationship, as well as the expectation of confidentiality.²¹⁶

In terms of regulatory aspects, the US Food and Drug Administration (FDA) has been regulating automated clinical decision making systems since the 1990s.²¹⁷ With the advent of new prediction techniques including deep learning, predictive models seeking approval must be further scrutinized in terms of the ground truth data utilized in training them, their intended use cases, their generalizability and robustness against edge cases, as well as their life-long learning aspects as they are continuously updated with more learning and more data. It is likely that AI application software will need to meet rigorous testing that is mandated for new submissions for regulatory approval, including quality control and risk assessment. As cloud computing and virtualization are being increasingly used to process medical data, healthcare information technology is gradually becoming part of the “big data” revolution.²¹⁸ This offers a fertile environment for incorporating state-of-the-art AI systems that are often distributed. Nevertheless, it raises data security and privacy concerns as maintaining Health Insurance Portability and Accountability Act (HIPAA) compliance is essential. Current cyber security research starts to offer solutions, including cryptonets where homomorphic encryption allows neural networks to run training and inference on encrypted data.²¹⁹

Today’s diagnostic paradigm in medicine focuses on the detection of visually recognizable findings that suggest discrete pathologies in images. However, the focus of such detection of singular disease processes may miss concurrent conditions in an individual as a whole. Imaging methods, from simple x-rays to advanced cross-sectional imaging methods such as magnetic resonance imaging or computed tomography, provide the opportunity to assess cancer in context of its surrounding organ system.

With the integration of AI, complex assessments of biologicals network may have a profound impact on the assessment of response and prognosis and treatment planning. Beyond the finding of neoplasm, imaging may detect changes in the adjacent or distant organs beyond the tumor that alters patient susceptibility to systemic morbidities, which ultimately can contribute to mortality. This may occur as a byproduct of disease progression itself or as a byproduct of treatment, such as radiation or chemotherapy. For example, in patients undergoing treatment for thoracic or breast cancer, chemotherapy may lead to myocardial damage while radiation therapy promotes advanced coronary atherosclerosis; patients who survive cancer also suffer from a high rate of cardiovascular events. Collectively, these cardiotoxicities may confer signal on routine imaging during the monitoring of cancer and be detected in earlier stages of development with comprehensive analytical systems that capture the diorama of disease processes. Initial concepts to apply AI to this clinical scenario stemmed from the fact that thoracic cancers and cardiovascular pathology are adjacent to each other and may be detected simultaneously, i.e. coronary calcification or pericardial fat on chest CT. Development of automated AI-based detection and quantification algorithms would therefore enable assessment of cardiometabolic markers without the need for additional imaging. In this manner, the role of AI can be extended to screening by simultaneously evaluating additional risks from the same data source. Since healthcare ultimately aims to prevent disease, generation of accurate risk models is essential in guiding actionable risk modification strategies.

While AI can detect incidental findings that may be clinically beneficial, these findings may be also clinically irreverent and, if not carefully framed in the correct clinical context, may increase patient stress, healthcare costs, and undesired side effects from treatment. It is likely that during the early phase of AI where human experts will continue to play key roles in gatekeeping the AI’s output, and therefore the majority of incidental findings detected by AI will still be evaluated by humans to discern if they are clinically significant or not the same manner when humans detected incidental findings. Over time, as AI systems mature, these incidental findings may become part of standard data evaluation and reporting the same way primary lesions are evaluated and reported in patient’s clinical context.

Additionally, imaging is not an isolated measure of disease. Increasingly, the molecular signature of cancers, including non-invasive blood biomarkers of tumor, socioeconomic status, and even social networks are appreciated to impact the outcome of cancer patients. Sources of data are also rapidly expanding beyond direct medical testing and include input from wearables, mobile phones, social media, unstructured electronic health records, and other components of the digital age. AI is well suited to integrate parallel streams of information--biological, demographic, and social--over time to improve predictive models for patient outcome.

As the power and potential of AI is increasingly proven, multiple directions remain for AI to transition into routine clinical practice. For imaging analysis, the accuracy and predictive power of AI methodologies need significant improvement and demonstration of comparable efficacy, or better, than human experts in controlled studies if it is to be poised to supplant clinician workflows. This shows initial promise in several disease conditions, but requires additional proof of clinical utility in prospective trials and education of physicians, technologists, and physicists to incorporate into widespread use.^220,221 While there will likely always be a “black box” for human experts in viewing AI generated results, data visualization tools are increasingly available to allow some degree of visual understanding of how algorithms make decisions.²²²

Curation of comprehensive datasets and outcomes that incorporate both disease-related and unrelated elements will also help train and expand AI systems to account for risks beyond cancer itself. In global settings with limited access to expert clinicians or exposure to uncommon pathologies, AI may offer a repertoire of “expert” experiences in disease interpretation. Conversely, strategies that predict outcomes without a ground truth provided by human experts may disrupt the traditional workflow familiar to clinicians and patients today.²²³ Furthermore, increased incorporation of AI in monitoring health resources and outcome will likely improve efficiency and reduce cost. As with any new innovative technology, the possibilities for development reside beyond current imagination.