Artificial Intelligence: reshaping the practice of radiological sciences in the 21st century

Abstract

Advances in computing hardware and software platforms have led to the recent resurgence in artificial intelligence (AI) touching almost every aspect of our daily lives by its capability for automating complex tasks or providing superior predictive analytics. AI applications are currently spanning many diverse fields from economics to entertainment, to manufacturing, as well as medicine. Since modern AI’s inception decades ago, practitioners in radiological sciences have been pioneering its development and implementation in medicine, particularly in areas related to diagnostic imaging and therapy. In this anniversary article, we embark on a journey to reflect on the learned lessons from past AI’s chequered history. We further summarize the current status of AI in radiological sciences, highlighting, with examples, its impressive achievements and effect on re-shaping the practice of medical imaging and radiotherapy in the areas of computer-aided detection, diagnosis, prognosis, and decision support. Moving beyond the commercial hype of AI into reality, we discuss the current challenges to overcome, for AI to achieve its promised hope of providing better precision healthcare for each patient while reducing cost burden on their families and the society at large.

Introduction

Humans have been always fascinated in the development of intelligent machines that can emulate their cognition and serve their needs.¹ Homer, in the Greek mythology wrote about Hephaestus, a blacksmith, who manufactured mechanical servants to help their masters. Al-Jazri, a 13th century artisan and mathematician is accredited with building the first humanoid (robot) that performed music and washed hands among other human-like functions.² However, artificial intelligence (AI) extends beyond robotics into processing information and invoking cognitive computing systems. For instance, the Turing machine, is a computing device that can simulate an algorithmic logic much like a human. Hence, it has long been argued that an intelligent machine, a machine that can think like humans, needs to pass the Turing test; i.e. when the responses from the computer and the human become indistinguishable.³ Although it remains argumentative whether such a test is indicative of the cognitive implications of true human-like intelligence, the notion of AI is satisfactory, as it indicates an aiding tool that can support human needs and improve their quality of life, thus dispelling fictional fears of a machine takeover! This is particularly true in the areas for practitioners of radiological sciences (diagnostic and therapy), who have been the vanguards of new technologies in medicine including early adoption of AI development and implementation as will be highlighted in this BJR 125th anniversary article, which is quite fulfilling since AI has currently become front and center in radiological sciences.

What is artificial intelligence?

Formally, AI as a field was conceived at the 1956 Dartmouth Summer Research Project (Dartmouth College, Hanover, 1956) by John McCarthy and colleagues, who coined the term “artificial intelligence.” According to McCarthy “AI is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable.⁴” Hence, AI currently can be thought of as representing a wide range of technologies that involve developing machines that can perform tasks that are characteristic of human’s (or other beings’) intelligence, e.g. playing a boardgame or autonomously driving a vehicle. The subcategory of AI that is primarily dedicated to the development of intelligent computer programs (algorithms) is denoted as machine learning (ML). The term was coined by another noted attendee of the Dartmouth workshop, IBM engineer Arthur Samuel, whom described ML as “a field of study that gives computers the ability to learn without being explicitly programmed.⁵” Traditionally, ML has relied on feeding computer-extracted human-engineered patterns (features) derived from the raw data, e.g. using computer vision methods, to an algorithm to perform a designated learning task; a process colloquially referred to now as shallow learning (SL). This is in contrast to a special subcategory of ML that allows for combined data representation (e.g. feature extraction) and task learning (e.g. classification or detection) known as deep learning (DL). Conceptually, DL comprises representation learning methods that are fed with raw data and can automatically discover the features needed for detection or classification using any ML approach.⁶ However, pragmatically this has been shown by Hinton and colleagues to be most successfully implemented currently with deep neural networks methods,^7,8 in which the frontend layers of the neural network learn the data representation and then pass this information into backend layers to perform the classification or detection tasks, all in one framework. Hence, in the case of a standard convolutional neural network (CNN), one of the most commonly used architectures for DL, the convolutional layers are tasked with learning the appropriate pattern representation (features) from raw image data and passing these features into the fully connected layers to perform the classification task.^7,8 A schematic of the relationship between AI, ML, and DL is shown in Figure 1. For instance, if the goal is to develop a computer-aided diagnosis (CADx) system that is comparable to a radiologist for distinguishing a benign from a malignant nodule on an imaging scan, then the hardware and software infrastructure of the CADx as a whole would consitute the AI-aid system. The collective computer set of rules, i.e. the algorithm, that is used to learn from annotated data, which nodule is benign or malignant would be the ML algorithm (classifier) that is being trained to classify these nodules. If the ML algorithm design requires extraction of human-engineered features, the learning process would be designated as an ML/SL procedure while if the algorithm operates directly on the raw imaging data and is able to automatically learn the data representation, then the learning process is designated as an ML/DL procedure. ML/DL has the intrinsic advantage of mitigating the statistical bias of human-engineered feature selection processes that are associated with ML/SL, leading to superior performances in many applications, e.g. imaging analytics.

Figure 1.

Venn diagram of the relationship between AI, ML, and DL. AI, artificial intelligence; DL, deep learning; ML, machine learning.

Furthermore, ML algorithms that require labeled data (annotations) for training, i.e. (input, output) pairs, undergo what is denoted as supervised learning, i.e. learning with a teacher. In contrast, if only unlabeled data are used as input data, this ML training is referred to as unsupervised learning, i.e. inferring patterns within the data. ML training in which part of the data is labelled and the rest unlabeled is referred to as semi-supervised learning and is generally thought to be more representative of human learning. If the algorithm learns to map inputs into optimized actions, this is denoted as reinforcement learning, i.e. goal-oriented tasks. These algorithms currently represent the main categories of ML, with supervised learning being the most common type in radiological sciences with applications ranging from detection, to diagnosis, to therapeutic interventions. However, several techniques are emerging to relieve the burden and cost of data labeling in supervised learning including: the semi-supervised approach mentioned above, transfer learning (using knowledge from other domains, such as natural images when learning medical ones), active learning (an interactive approach with human being involved), and more recently weak supervised learning, where the labels are assumed to be imprecise or noisy.⁹ Unsupervised learning is typically used for clustering or data reduction tasks while reinforcement learning is applied for optimizing sequential decision-making processes, in clinical management for instance. A summary of some of the common applications of these different ML/DL categories in radiological sciences is given in Table 1.

Table 1.

Sample radiological applications of ML algorithms is radiological sciences

ML category	Applications	References in text
Supervised learning	Detection of objects in images	10–22
	Auto contouring and segmentation	23–46
	Diagnosis of cancer lesions	11,12,14,47–62
	Image prognosis	21,63–65
	Pathological analysis	66–71
	Image analysis for risk assessment	72–75
	Image quality	76–81
	Quality and assurance	82–86
	Treatment planning and management	87–102
	Response prediction	101,103–113
Weak supervised learning	EMR and imaging integration	114,115
Unsupervised learning	Clustering for tumor segmentation	116,117
	Dimensionality reduction for classification	102,118
	Error detection and quality assurance	119
	Image registration	120
Reinforcement learning	Treatment adaptation	121,122
	Planning optimization	123

ML, machine learning.

Historically, applications of ML in medicine demonstrated decision-support capabilities in detecting disease onset or analyzing the relationship between treatment and response. Ledley and Lusted in their seminal 1959 paper, anticipated the role of a probabilistic logic-based approach to understand and support physicians’ reasoning¹²⁴. An early major AI initiative was the MYCIN project at Stanford in the 1970s, which was a rule-based system to identify bacteria types that may cause infectious diseases¹²⁵, achieving an acceptability rating of 65% from a panel of experts¹²⁶. However, several radiological applications in medical imaging preceded MYCIN, with Winsberg et al. reporting, in 1967, on computer detection of radiographic abnormalities in mammograms with computer analysis¹⁰ and with Lodwick et al. presenting a roentgenograms concept for analyzing bone and lung cancer images^47,48 and Meyers et al. developing an automated computer analysis of cardiothoracic ratios.⁷²

Through pioneering efforts at the University of Chicago in the 1980s, computer-aided diagnosis (CAD) became a major subject in medical imaging and diagnostic radiology, providing radiologists with computer output as a “second opinion” to aid in making final decisions.^{11–13,49–51} These CAD systems utilized image feature-based analysis for the detection of microcalcifications in mammogram images¹⁴ and lung nodules in digital chest radiographs.⁵² It is important and interesting to note that the subtle change from computer readers to computer aids allows for a more acceptable approach to incorporating AI into clinical practice, possibly leaving the final decision to the human clinician. However, current issues surrounding physician liability and medical AI use are complex and many challenges are yet to be resolved. Therefore, new guidance on assigning liability for injuries that may arise from interaction between algorithms and practitioners (including appropriate use of the AI system by the human) is needed.^127,128

These CAD applications were extended in the 1990s by application of artificial neural networks (ANN) generally¹⁵ and convolutional neural networks (CNNs) specifically^16,17,53 as reviewed in the literature.^{11–13,50,51,73} In the 2000s, CAD applications mainly utilized ML methods based on support vector machines (SVM) algorithms^18,54,55 paralleling advances in machine learning at the time. A similar trend was noted in therapeutic radiology, with applications in mid 1990s using ANNs for automating treatment planning evaluation,⁸⁷ beam orientation customization,⁸⁸ and standardization (knowledge-based planning).⁸⁹ In the 2000s, these applications focused on predicting radiotherapy response with ANNs^103–105 and SVMs.^106–108

The promise of AI in medicine in general and radiological sciences in particular have been hyped in the past, but, despite progress, failed on predictions in the 1970s that by the year 2000 computers would act as a powerful extension of the physician's intellect.^129,130 This may be partly due to clinicians not using AI as intended or as FDA approval may have originally required. Although AI had its fair share of boom and busts, denoted as AI winters, applications of AI in medicine are witnessing a new but more cautious spring with applications spanning all aspects of practice. A summary of publications of AI in radiology and radiation oncology in the past 30 years is shown in Figure 2 using PubMed with search terms (“Radiology” OR “Radiation Oncology”) AND (“Artificial Intelligence” OR “Machine Learning”). The figure highlights the rapid increase in AI consideration, especially in more recent years.

Figure 2.

AI related publications in radiology and radiation oncology over the past30 years. AI, artificial intelligence.

Radiological science practitioners, have been at the forefront utilizing AI, leading its implementations, particularly in areas related to diagnostic imaging and therapy. This anniversary article highlights the current status of AI in the radiological sciences, noting its achievements and existing challenges. It is intended to provide a realistic vision of its role, not as a competitor to human cognition but as a partner in delivering improved healthcare by combining AI/ML software with the best human clinician knowledge (i.e. human-in-the-loop) to permit delivering care that outperforms what either can do alone and thus improving both credibility and performance.^131–134

AI today for radiological sciences

In the past, the U.S. Food and Drug Administration (FDA) has approved AI systems for clinical use, including the R2 ImageChecker system, which received FDA approval in 1998¹⁹ for use as an aid in the detection of breast abnormalities on screening mammograms (CADe), i.e. as a second reader. Interestingly, the R2 ImageChecker system included a CNN.^17,19 The approval launched the first phase of FDA-approved systems for detection of lesions. More recently, the FDA is approving multiple platforms that incorporate AI, signaling a new era in medicine and radiology.¹³⁵ Among these, the first system cleared was the QuantX Advanced system to aid in breast cancer diagnosis (CADx), developed originally by Giger and colleagues.^{56,57,63,116,136} Today, with the success of deep convolutional neural networks in the classification of natural objects in digital photograph images, the field of diagnostic imaging with its digital workflow and petabytes of digital medical images is becoming a natural target for AI studies. CT and MRI scans typically involve hundreds of images per exam; reviewing the sheer volume of these images imposes limitations on the productivity of practicing radiologists. Thus, a natural application of AI is to assist the radiologist in the detection and classification of abnormalities on medical images to improve productivity without loss of diagnostic accuracy. Furthermore, there is a focus on quality and efficiency in healthcare systems due to rising costs. Radiologists are highly skilled and expensive professionals, so any tool that can maximize quality and efficiency is of great interest. ML/DL also can improve image quality by improving current image reconstruction and filtering of modalities such as MRI,⁷⁶ CT⁷⁷ and ultrasound⁷⁸ as well as potentially improve their spatial resolution. Finally, precision medicine requires derivation of reproducible imaging biomarkers.¹³⁷ There is potential of ML/DL to help in this task as well as will be discussed in the following.

Applications in computer-aided detection, diagnosis (CADe,x) and decision support

Applications in diagnostics radiology have witnessed steady growth and demonstrated remarkable progress in this new era of AI spanning its workflow as shown in Figure 3.^66,138–141 In the following, we highlight some representative cases. Note that along this workflow, AI can aid in setting acquisition parameters, reconstructing images, assessing image quality of acquired images, preprocessing and “hanging” images for interpretation, along with the decision-making tasks of detection, diagnosis, patient management, and treatment response.

Figure 3.

Permission has been obtained to reproduce this image. The schematics highlights AI role in radiology.¹³⁸

Case triage is an example of how ML can improve radiologists’ workflow, i.e. CADt. An AI algorithm identifies exams with a higher likelihood of containing critical findings and puts them at the top of a reporting list for the radiologist to reduce turnaround and potentially improve outcomes. Chilamkurthy et al. recently demonstrated using deep Residual Network (ResNet), a popular CNN architecture that can have deeper layers than standard CNNs without being subject to degradation by preserving identity connection (short-cuts) through skipping some layers, an area under the ROC curve of greater than 0.9 for a variety of critical findings such as haemorrhage and calvarial fractures on unenhanced head CT in a retrospective study involving 313,318 scans from about 20 centers.²⁰

Initial applications of ML in medical image segmentation have focused on unsupervised learning techniques such as clustering methods.^67,116 More recent studies have shown the promise of supervised learning with CNN’s in segmentation of organs largely empowered by transfer learning from natural images databases (e.g. Google’s ImageNet), and no doubt we will see further development in this arena.^23,24,117 The most commonly used DL algorithm for biomedical image segmentation is based on the U-Net architecture, which consists of two symmetric CNNs, one is an encoding (contracting) path and the other is a decoding (expanding) path, and hence the U shape.²⁵ Fatty infiltration of the liver and the potential for liver damage related to non-alcoholic steatohepatitis is an important heath concern. In a recent study Graffy et al. used a modified three-dimensional U-Net for liver segmentation to assess the distribution of fatty infiltration of the liver in 11,669 CT scans in 9552 adults and published the distribution hepatic steatosis, demonstrating excellent performance compared to manual assessment.⁶⁸ The feasibility of routine addition of liver steatosis measurement using a fully automated tool can thus be envisioned to add value to routine CT reports and development as a useful population screening tool.

When dealing with different medical imaging modalities (CT, MR, positron emission tomography, single-photon emission CT, ultrasound, etc.), an important task is to bring these images into the same frame of reference, a process denoted as image registration. Varying algorithms for rigid and defomarable registration have been proposed in the literature using intensity or feature-based approaches^142,143 and more recently ML/DL algorithms.¹⁴⁴ For example, de Vos et al., developed an unsupervised learning technique to train CNNs for image registration using intensity-based metrics. They demonstrated its performance for registration of cardiac cine MRI and the registration of chest CT with comparable results to conventional image registration while being orders of magnitude faster.¹²⁰

ML/DL algorithms can be used to develop prognostic and predictive tools in cancer therapeutics beyond the RECIST criterion with its known shortcomings. Hosny et al. demonstrated the feasibility of prognostication using CT in non-squamous cell lung cancer assessment achieving area under the curves (AUCs) of 0.7–0.71 in surgical and radiation treated cohorts with a CNN.²¹ While Hylton et al.⁶⁴ demonstrated the benefit of a semi-manual functional tumor volume from MRI to predict recurrence-free survival-results from the ACRIN 6657/CALGB 150007 I-SPY 1 TRIAL on neoadjuvant chemotherapy for breast cancer. Drukker et al.⁶⁵ showed that an automated computer-extracted most-enhancing tumor volume radiomic feature could perform as well. Recent applications have utilized weak supervised learning for integration of electronic health records with imaging data to improve detection and diagnosis. For instance, Wang et al. demonstrated this approach for classification and localization of common thoracic diseases from chest X-ray images¹¹⁴ and Zhu et al. for detection of pulmonary nodules in lung CT images.¹¹⁵

Maximizing image quality is the goal of manufacturers of imaging equipment with the ultimate aim being to improve diagnostic accuracy. These improvements can come with added costs, radiation exposure, and often increases in time on the scanner. Post-acquisition processing of images or use of ML/DL approaches has shown promise in improving image quality and potentially improving patient throughput. Improving spatial resolution of images of the knee on MRI⁷⁹ and reduction of the dose of intravenous Gd-based contrast agents for brain MRI are examples of this potential.⁸⁰ Similar efforts have been demonstrated in thoracic CT using DL to assess acceptable image quality prior to interpretation.⁸¹

In prostate cancer, there is a major shift in the paradigm of diagnosis, which involves the use of multiparametric MRI (mpMRI) for cancer detection prior to biopsy. It is anticipated that there will be a marked increase in demand for prostate mpMRI. CNNs have shown promise in detection of prostate cancer on MRI showing near human performance and they can potentially be used to aid the radiologist in reporting and improve efficiency and consistency of reporting across the radiology community, thus reducing costs.⁷⁴ Further work to show robustness of these algorithms across institutions and MRI equipments is still needed (Figure 4).

Figure 4.

Permission has been obtained to reproduce this image.Explainable AI: A Class activation map from Ensemble Learning Architecture with DeepConvolutional Neural Network. An ensemble deepconvolutional neural network gave the probability of cancer in this patient of 83%. The patient has a clinically significant Gleason grade Group II cancer. (A) An apparent diffusion coefficient map from MRI annotated by a radiologist showing the cancer outlined in blue. (B) A gradient-weighted class activation map (Grad-Cam) shows that the area of highest activation matches the site of tumor giving confidence that the DCNN is deriving the high probability of cancer based on the features of the cancer itself (reprinted from Cao et al.⁷⁴). AI, artificial intelligence; DCNN, deep convolutional neural network.

Many of the original advancements in AI in radiology have occurred in breast imaging,^{58,59,73,75,118,137,145} where CNNs have contributed to breast cancer risk assessment, detection, and diagnosis. Much research has focused on the formats of the input and the proper usage of outputs, as “garbage in, garbage out” can stymie success. For example, in Figure 5, Antropova et al⁵⁹ demonstrated how CNN performance varies depending on the input format of breast MRIs. In addition, recent investigations have shown that the fusion of human-engineered imaging features (radiomics) with DL extraction methods can yield statistically significant improvements as compared to either one separately.⁶⁰

Figure 5.

Full breast MRI images and ROIs for (a) a MIP image of the second postcontrast subtraction MRI, (b) a center slice of the second post-contrast MRI, (c) a central slice of the second postcontrast subtraction MRI, and (d) ROC curves showing the performance of three classifiers. The classifiers were trained on CNN features extracted from ROIs selected on the three input formats. (reprinted from Antropova et al.⁵⁹). CNN, convolutional neural network; MIP, maximum intensity projection; ROC, receiver operating characteristic.

ML advances are also contributing to cancer discovery where, through the benefit of multidisciplinary, multi-institution collaborations, the computer-extracted tumor information (i.e. image-based phenotypes) is mapped to other “-omics” such as histopathology and genomics.^{61,69–71,109} Figure 6 demonstrates significant associations that were found between breast MRI radiomic phenotypes with genomic features as part of the NCI TCGA/TCIA collaboration. In cancer discovery, once relationships between image features and genomics (radiogenomics) are found to yield an image-based “cancer signature,” these computer-extracted radiomic features can be used as “virtual” biopsies for when an actual biopsy is not practical, such as in cancer screening, monitoring response to therapy, or assessing risk of recurrence.^62,137

Figure 6.

Overview of identified statistically significant associations between radiomic phenotypes and genomic features. Each node is a genomic feature or a radiomic phenotype. Each line is an identified statistically significant association. Associations were deemed as statistically significant if the (multiple comparison) adjusted p-values ≤ 0.05. (reprinted from Zhu et al.⁷¹).

Applications in radiation oncology

The treatment of cancer using high energy ionizing radiation, known as radiotherapy, involves a large set of complex processes spanning consultation to treatment planning to delivery with a large amount of heterogeneous information (e.g. clinical, dosimetric, imaging, and biologics), human–machine interaction, optimization and decision making. Hence, ML/DL can help automate many of these processes while allowing for personalization of treatment via application of its predictive analytics^146–148 as depicted in the schematics of Figure 7.

Figure 7.

The schematics highlights AI role in radiation oncology. Adapted from Meyer et al.¹⁴⁶ AI, artificial intelligence; PET, positron emission tomography; QA, quality assurance; CBCT, cone beam CT.

Automation applications of ML/DL in radiotherapy has been an area of tremendous progress and it includes but is not limited to: (1) auto-contouring (segmentation) of organs, tissues and the targeted cancer,^22,26–46 which is a major bottleneck in terms of treatment planning efficiency and the availability of such ML/DL tools can reduce this burden significantly; (2) dose prediction,^90–92 which is an area where AI can improve efficiency in radiotherapy and mitigate prior authorization delays; (3) knowledge-based treatment planning,^{93–97,123,149} which is important for mitigating common errors, particularly in under served or disadvantaged settings with limited resources; (4) radiation physics (machine or patient) quality assurance (QA) to improve safe radiation administration^82–86,119 ; and (4) motion (respiratory or cardiac) management during radiation delivery^98,99 to enhance tumor targeting and limit uninvolved tissue exposure.

Predictive analytics of ML/DL in radiotherapy includes: (1) image-guidance^100,150 to ensure that the planning objectives of tumor targeting and normal tissue avoidance are maintained; (2) response modeling of treatment outcomes such as tumor control or toxicities^{101,102,106,107,110–113} ; and treatment adaptation to account for geometrical changes¹⁵¹ or adapting prescription to improve outcomes.^121,122 Details about these and other applications are reviewed in the literature.¹⁵² An example for adaptive radiotherapy to improve outcomes is shown in Figure 8, which combines changes in imaging (radiomics) with dosimetric (dosiomics) and biological factors (genomics) to personalize response to radiotherapy. In this case, for a population of non-small cell lung cancer patients, PET-FDG as detected by mid-treatment was used for escalating the radiation dose and was shown to improve local tumor control at 2 year follow-up.¹⁵³ However, the dose adaptations in such studies were based on clinicians’ subjective assessment. Alternatively, to objectively assess the adaptive dose per fraction, a 3-component deep reinforcement learning (DRL) approach with neural network architectures was developed. The architecture was composed of (1) a generative adversarial network (GAN) to learn patient population characteristics necessary for DRL training from a relatively limited sample size; (2) a radiotherapy artificial environment was reconstructed by a deep neural network utilizing both original and synthetic data (by generative adversarial network) to estimate the transition probabilities for adaptation of personalized radiotherapy patients’ treatment courses; and (3) a deep Q‐network (DQN) applied to the radiotherapy artificial environment for choosing the optimal dose in a response‐adapted treatment setting. The DRL was trained on large-scale patient characteristics including clinical, genetic, and imaging radiomics features in addition to tumor and lung dosimetric variables. In comparison with the clinical protocol,¹⁵³ although on average slightly less aggressive, the DRL achieved a root mean-squared error = 0.5 Gy for dose escalation recommendation. Interestingly, on occasion, the DRL also seemed to suggest better decisions than the clinical ones in terms of mitigating toxicity risks and improving local control as seen in Figure 8.¹²¹

Figure 8.

A deep reinforcement learning for automated radiation adaptation in lung cancer. (a) Three different neural networks are used for synthetic data generation (GAN), the radiotherapy environment (DNN), and Deep Q-network (DQN). (b) Performance results showing DQN (black solid line) vs clinical decision (blue dashed line) with RMSE error = 0.5 Gy. Evaluation against eventual outcomes of good (green dots), bad (red dots), and potentially good decisions (orange dots) are shown suggesting not only comparable but also at instances better overall performance by the DQN (reprinted from Tseng et al.¹²¹). DNN, deep neural network; DQN, Deep Q-network; GAN, generative adversarial network; RMSE, root mean-squared error.

AI necessary infrastructure and resources

Access to large well-annotated data sets remains the number one requirement for advancement of AI in radiological sciences. The majority of currently implemented AI approaches rely upon supervised learning techniques. Thus, radiologists face the daunting task of annotating these large data sets. The process of curation and annotation can be helped through mining of radiology reports using natural language processing (NLP), semi- or weak-supervised learning to help reduce the number of cases that must be manually annotated. There have been enormous efforts to collect, curate, and share large data sets through the NCI’s TCIA,¹⁵⁴ which intakes clinical trial studies as well as investigators’ research data sets.

Storage of large data sets scrubbed of patient information is not part of the normal archival process for patient records. Data lakes specifically designed for research need to be created that can handle large imaging and therapy data sets. This requires investment and hiring of data science teams to manage these data lakes. Furthermore, data provenance and governance policies that address concerns of institutional review boards and the broader ethical issues around the use of large patient data sets for research and sharing of large amounts of data with for profit companies also need to be developed.

In addition to curation of data, there is a need to curate or QA the data analytic pipelines developed to train, test and validate an ML/DL algorithm. Live data scenarios or even changes in versions of software, hardware platforms and open source libraries can all affect the predicted output. Being able to reproduce and share analytic pipelines in the community is vital to develop confidence in the results of AI studies.

For the radiologist/oncologist having an intuitive sense of how the classifier reached its conclusion is of utmost importance particularly as we are in the early days of AI testing and adoption. This is especially true when the AI-based analysis result in a conclusion discrepant with the physician own’s opinion. Unfortunately, many DL algorithms suffer from the black box stigma.¹⁵⁵ Several approaches to derive proxy models, develop attention maps, provide disentangled representation or learn with known operators have been emerging to create a more interpretable/explainable AI paradigm.^156–160 Eventually, if AI performs consistently and at a better than human performance level then the need for fully explainable AI may diminish. However, AI researchers and industry also have recognized this need by developing new tools to quantify uncertainties,¹⁶¹ to understand feature attributions,¹⁶² or to investigate model’s behavior such as the What-If tool in TensorFlow by Google, for instance.

Current academic and commercial tools

Tools for annotation of images are available, however, they are not necessarily optimized for use in image analysis in the AI era. There is a growing need for common annotation platforms that are in the open source community. These tools should provide access to segmentations using standard formats, so that these can be fed into analytic pipelines and shared across the community. 3D slicer¹⁶³ is an example of an open source viewing and annotation tool. CLARA is a software collection of developer toolkits built on NVIDIA’s computing platform aimed at accelerating computation, artificial intelligence, and advanced visualization being offered to the research community.¹⁶⁴

Future of AI in radiological sciences

Significant progress in computational hardware, with parallel processing capabilities (e.g. Graphics Processing Unit, Tensor Processing Unit, Intelligent Processing Unit etc.), has accelerated the calculation of large and sophisticated algorithms from weeks to days and few hours only. This is coupled with the availability of open-library software platforms (e.g. Tensorflow,¹⁶⁵ Pytorch,¹⁶⁶ etc.) with notable success rivaling that of humans when applied to large databases such as ImageNet,¹⁶⁷ a large-scale database with more than one million annotated natural images (animals, plants, persons, devices, instruments etc) with about 20,000 text descriptors. However, an ImageNet-like medical imaging database is yet to be realized with caveats related to data sharing, provenance, patient privacy and the nature of medical imaging acquisition that not only varies in technologies and parameters but also shifts over time with new developments. Moreover, issues related to learning bias^168,169 and adversarial examples^170,171 need to be accounted for too. For instance, an ML algorithm developed for predicting the risk of pneumonia counter intuitively suggested that patients with pneumonia and asthma to be at a lower risk of death than patients with pneumonia but without asthma.¹⁶⁹ Similar controversial examples were noted in the case of skin cancer risk prediction, where the presence of a ruler in the image may be a cue for the ML algorithm of high risk¹⁷² or the appearance of a tube in a chest X-ray being indicative of severe lung disease.¹⁷³ These examples and others stress the importance of data quality and context when training and applying ML/DL algorithms.

The development of trustworthy AI is currently a major goal in the AI community for medical applications with efforts led by the European Union and other international bodies that advocate for lawful, ethical and robust application from technological and societal perspectives. Towards this goal, there has been shifts toward developing more explainable/interpretable AI algorithms,¹⁷⁴ which would allow for better transparency, oversight, and accountability. The radiological clinic of the future can be thought of to be composed of explainable AI platforms with streamlines access to data lakes, assisting clinicians to improve efficiency as well as achieve better diagnostic or therapeutic decisions.

Conclusion

The past few years have witnessed a tremendous rise in AI applications to a wide range of areas in radiological sciences (diagnostic and therapy) including automation of segmentation, improving image quality, and developing decision-support systems for personalization of detection and treatment. AI is expected to alter the way patients are being managed and how doctors reach their clinical decisions. Appropriately developed and used, such diagnostics are expected to be faster, cheaper, and more accurate than ever. However, one has to be mindful of the characteristics and the limitations of this disruptive technology as well. AI methods require large amounts of data for their training and validation, which also beg the questions of computerized trust, data sharing, and privacy concerns. With the pre-existing domain knowledge, the merits of human-engineered analytics or features standing for accumulative knowledge based on numerous observations should be incorporated, hybridized, and inherently infused with modern deep learning architectures. Aside from commercial hype, the cautious deployment and assistance of the state-of-the-art AI technologies, imaging informatics hold tremendous potentials to provide better precision and personalized healthcare for patients while reducing the cost burden on society.