Machine Learning and Deep Neural Networks in Thoracic and Cardiovascular Imaging

Summary

Advances in technology have always had the potential and opportunity to shape the practice of medicine, and in no medical specialty has technology been more rapidly embraced and adopted than radiology. Machine learning and deep neural networks promise to transform the practice of medicine, and in particular the practice of diagnostic radiology. These technologies are evolving at a rapid pace due to innovations in computational hardware and novel neural network architectures. Several cutting-edge post-processing analysis applications are actively being developed in the fields of thoracic and cardiovascular imaging, including applications for lesion detection and characterization, lung parenchymal characterization, coronary artery assessment, cardiac volumetry and function, and anatomic localization.

Cardiothoracic and cardiovascular imaging lies at the technological forefront of radiology due to a confluence of technical advances. Enhanced equipment has enabled CT and MRI scanners that can safely capture images that freeze the motion of the heart to exquisitely delineate fine anatomical structures. Computing hardware developments have enabled an explosion in computational capabilities and in data storage. Progress in software and fluid mechanical models are enabling complex 3D and 4D reconstructions to not only visualize and assess the dynamic motion of the heart, but also quantify its blood flow and hemodynamics. And now, innovations in machine learning, particularly in the form of deep neural networks, are enabling us to leverage the increasingly massive data repositories that are prevalent in the field.

Here, we discuss developments in machine learning techniques and deep neural networks to highlight their likely role in future radiological practice, both in and outside of image interpretation and analysis. We discuss the concepts of validation, generalizability and clinical utility as they pertain to this and other new technologies, and we reflect upon the opportunities and challenges of bringing these into daily use.

Cardiothoracic Imaging at the Forefront of Technological Advances

From its inception, radiology has been a discipline filled with innovators and pioneers, who have ambitiously sought to solve challenging clinical problems with new technology. From the earliest application of x-rays for to diagnosis and intervention, to the advent of computed tomography (CT) and magnetic resonance imaging (MRI), to 3D post-processing and computer-aided detection (CAD), the specialty has often been the first to embrace innovations that allow efficient diagnosis of disease to improve the care of patients. Increasingly advanced CT and MRI scanners are capable of capturing images that freeze heart motion and exquisitely delineate fine anatomical structures at a level of detail never before seen outside of the operating room. In addition, advances in software and computational fluid mechanical models are enabling complex 3D and 4D reconstructions to visualize and assess the dynamic motion of the heart and quantify blood flow and hemodynamics (Figure 1). Due to its intrinsic dynamic and anatomical complexities, cardiothoracic imaging is poised at the forefront of transformative technologies as machine learning and deep neural networks begin to revolutionize medicine and image analysis.

Figure 1.

Static images from a cardiac-gated 4D MRA (left) and 4D Flow (right), showing congenital branch pulmonary artery stenosis and enlarged intercostal collateral arteries supplying the left lower lobe. New technologies such as this provide increasingly detailed understanding of pathophysiology, for greater patient-specific insights.

Machine Learning and the Advancements of Deep Neural Networks

Since the development of the earliest computers there has always been an interest in utilizing these systems to automate tasks. Starting with simple numerical calculation, computer programs quickly progressed in complexity. Modern high-level languages such as Python abstract away computational intricacies and have made it simpler to develop algorithms to ingest data and subsequently use them to make predictions. Commonly used supervised (using input and output pairs) and unsupervised (using only inputs) machine learning methods were developed and quickly expanded to include linear and logistic regression, tree-based methods such as random forests and gradient boosted trees, kernel-based methods such as support vector machines (SVM), and clustering methods like k-means and E-M clustering¹ (see Figure 2). Traditional machine learning methods share a common limitation – they require features to be specified in advance. The need for computational features meant that highly specialized computer scientists were required to conceptualize relevant features and manually translate them into code to be used as input for the machine learning algorithms, a process sometimes referred to as “feature engineering.” These earlier methods were therefore somewhat inaccessible as they required a considerable amount of time and expertise to train and fine-tune. In contrast, neural networks can automatically learn input features from the data, alleviating much of the time and expertise necessary for training and fine-tuning.

Figure 2.

Traditional machine learning algorithms are categorized into “unsupervised” learning, requiring input data only, and “supervised” learning, requiring input and output data. Deep neural networks, a form of supervised learning, comprise the most recent advances in machine learning.

The field of artificial neural networks developed in parallel to traditional machine learning methods, albeit with a rockier history. The first precursor to current neural networks was the perceptron, a “brain model” for supervised learning of binary classifiers, developed in 1958². Despite the initial excitement this generated, the perceptron was not capable of solving some basic mathematical problems such as the exclusive-or (XOR), thus leading users to realize that it was not a universal solution. This led to several decades of an “AI winter” in which neural network development effectively came to a halt³. Neural network research remained relatively dormant until the work of Rumelhart et al. in 1986, which demonstrated that networks of neuron-like units could effectively be trained using gradient descent and back-propagation techniques⁴. This discovery, along with a demonstrated ability for these networks to learn any function, revived the field of neural network research⁵.

Through the 1990s and early 2000s, several breakthroughs were made. Neural networks were used to recognize handwritten digits, and pre-training with unspecified features enabled deep neural networks with multiple layers^6,7. Progress was, however, intermittent and hindered by the large amount of data and computing resources needed to train these networks. A considerable breakthrough came in 2012 when Krizhevsky et al. trained a large, deep convolutional neural network to classify objects in the ImageNet LSVRC contest and won by a significant margin^8,9. The victory was attributed to a number of network architectural components, but also the use of GPUs to accelerate network training with access to the large ImageNet dataset (1.2 million images). Since then, there has been an explosion of neural network research (examples of simple architectures are shown in Figure 3). Much attention has been directed to building larger network architectures up to hundreds of layers deep and novel ways to train them successfully^10-12. In parallel, convolutional network architectures have also been created for tasks including object detection, localization, and segmentation^13,14.

Figure 3.

Layout of an example (A) multilayer neural network and (B) deep convolutional neural network, for which many architectures are possible. Convolutional neural networks are capable of learning and extracting combinations of image features to render a result, such as the localization of the left ventricular apex (bottom right) without being explicitly programmed to look for specific features.

As neural network research has grown, high level software frameworks like Keras were developed, allowing increased accessibility of image analysis and enabling its use in multiple other fields, including medicine¹⁵. Medical imaging has presented its own challenges, and network architectures originally built to operate on natural scene images needed to be optimized for medical image analysis¹⁶. These obstacles have sparked inventive solutions, including novel architectures such as UNet, which was designed for semantic segmentation in the context of challenges that are commonly encountered in biomedical images (eg. small sample sizes and high image resolution)¹⁷. Numerous networks have since been developed to solve a diverse range of problems in medicine, such as organ volumetry and lesion identification (Figure 4 and 5).

Figure 4.

Architecture of a cascaded system of multiple neural networks, each building upon the outputs of a preceding network. In this example, the (a) proposal network to identifies candidate pulmonary nodules, the (b) classification network identifies “true” pulmonary nodules from false-positives, and the (c) segmentation network delineates the boundaries of each nodule. Together, the proposal and classification networks may be used as a method for computer-aided detection (CAD).

Figure 5.

Results of a cascaded convolutional neural network designed for pulmonary nodule detection and segmentation. The neural networks appear to overcome limitations of earlier algorithms in identifying boundaries of nodules near pleural surfaces.

Realizing clinical usage of this technology in medicine presents a range of unique complexities that may be new to the machine learning community. Challenges that must be considered include integration into the healthcare system while maintaining patient security and privacy, wide-ranging normal and abnormal human morphology, and liability and accountability concerns. Artificial intelligence heralds a sea change in medical imaging, offering an opportunity for radiologists to be leaders in, and shape the field of computer-aided diagnostics. Early adopters of technologies have the opportunity to set precedents and standards for their use and validation. As such, radiologists may play a critical role to shape the role of machine learning-based diagnostics in clinical practice.

Applications in Cardiovascular and Thoracic Imaging

One of the earliest introductions of machine learning into cardiovascular medicine involved reading and interpreting ECG tracings, a technology that has now been incorporated into everyday clinical practice¹⁸ and has benefited from the development of convolutional neural networks that allow detection and classification of arrythmias^19-21. Machine learning has since been applied to various tasks in cardiovascular and thoracic imaging including segmentation, characterization, quantification, lung nodule detection and measurement, and lung cancer prognosis and treatment. Many of these applications have been consolidated in prior reviews^22-24 and we provide a survey of those relevant to cardiovascular imaging in Table 1 and those relevant to thoracic imaging in Table 2.

Table 1.

Applications of Machine Learning in Cardiovascular Imaging

Modality	Ref	Author	Description	ML Technique
MRI	25	Avendi 2017	RV segmentation	CNN
	27	Winther 2017	RV, LV endocardium and epicardium	CNN
	28	Tan 2018	LV segmentation	ANN
	33	Baessler 2018	Myocardial scar detection	Random forests
	34	Dawes 2017	Pulmonary hypertension prognosis	PCA
ECHO	35	Ortiz 1995	HF prognosis	ANN
	36	Narula 2016	HCM vs athlete's heart	SVM, Random forests, ANN
	37	Sengupta 2016	constrictive pericarditis vs restrictive cardiomyopathy	AMC, random forest, k-NN, SVM
	38	Sengur 2012	valvular disease	SVM
	39	Moghaddasi 2016	MR severity	SVM
	41	Sudarshan 2015	MI detection	SVM
CT	43	Wolterink 2016	CAC scoring	CNN
	45	Isgum 2012	CAC scoring	k-NN, SVM
	52	Itu 2016	FFR estimation	deep neural network
	53	Motwani 2017	Prognosis	Logistic regression
	54	Mannil 2018	MI detection	Decision tree, k-NN, random forest, ANN

Table 2.

Applications of Machine Learning in Thoracic Imaging

Topic	Ref	Author	Description	ML Technique
Lung Nodules	61	Lo 2018	Pulmonary nodule detection	CADe
Radiomics	63	Li 2018	NSCLC prognosis	Unsupervised 2-way clustering
	65	Song 2016	NSCLC prognosis	SVM
Radiogenomics	71	Yamamoto 2014	Genetic classification	Random Forest
COPD	73	Ying 2016	COPD classification	Deep neural network
	74	Gonzalez 2012	COPD staging	CNN
Abdominal aortic thrombus	75	Lopez-Linares 2018	Thrombus detection	CNN

Applications to Cardiovascular Disease

One major emphasis of machine learning applications in cardiac MRI is ventricular segmentation for quantification of volumetry and function. This is a particularly attractive problem for machine learning as it is typically a time-consuming aspect of these exams, often requiring manual outlining by a skilled operator. Deep learning algorithms have been generated that approximate the cardiac measurements of expert readers^25-29. One such algorithm proposed by Avendi et al. was well correlated with ground-truth measurements (0.99 for end-systolic and 0.98 for end diastolic)²⁵. Most recently, several commercial vendors have begun to take an interest in this technology and to integrate these algorithms into their software^30-32. Outside of cardiac segmentation on MRI, algorithms have also been developed to detect subacute or chronic myocardial scar³³ and to predict patient survival and mechanisms of right heart failure in pulmonary hypertension³⁴.

Machine learning techniques have also been applied to characterizing cardiac disease on echocardiography (ECHO). Ortiz et al. used neural networks to analyze cardiac contractility to predict one-year mortality in patients with heart failure³⁵. Since this early work, supervised machine learning techniques have used ECHO to differentiate hypertrophic cardiomyopathy from athlete’s heart³⁶, classify and differentiate constrictive pericarditis from restrictive cardiomyopathy³⁷, diagnose valvular heart disease³⁸, grade severity of mitral valve regurgitation³⁹, automate ejection fraction measurement⁴⁰, and detect the presence of myocardial infarction^41,42.

Several machine learning applications have also been developed to assist in the interpretation of CT. For example, algorithms have been developed for automation of coronary artery calcium scoring^43-46 and assessment of functional significance of coronary lesions. More recently, ML techniques have been leveraged to better assess the hemodynamic significance of coronary stenosis. It has recently become possible to simulate the results obtained from an intracoronary pressure wire at the time of coronary catheterization with computational fluid simulations based on coronary CTA, the so-called fractional flow reserve (FFR)^47-49. However, the time and expertise needed for coronary segmentation and flow simulation may delay availability of results and limit its use. With this concept in mind, several groups have attempted to use machine learning approaches to accomplish this similar task^50-52. Other applications of machine learning on CT include prediction of 5-year all-cause mortality⁵³, and detecting the presence of myocardial infarct using texture analysis methods⁵⁴.

Applications to Thoracic Imaging

Outside of the heart, another area of recent rapid development has been detection and measurement of lung nodules. Lung cancer has long been the leading cause of cancer-related mortality in the United States, and significant attention has been focused on establishing screening guidelines for the purpose of earlier detection. A landmark study published in 2011 compared low dose CT with CXR in screening high-risk patients for pulmonary nodules⁵⁵. Despite low incidence of detected cancer in the patient group and high false positive rates, the study was terminated early, as screening with low dose CT showed reduced mortality from lung cancer. In response, the United States Preventative Services Task Force recommended routine screening for this high risk group, which is predicted to increase the number of pulmonary CTs performed annually⁵⁶. The volume of studies combined with the time-consuming nature of detection, measurement, and comparison of pulmonary nodules, has prompted a boom in thoracic machine learning research. To address the need to improve accuracy and efficiency in utilizing machine learning approaches multiple research groups and challenges have focused on developing computer-aided detection (CAD) algorithms for pulmonary nodules^57,58.

In the late 2000s, there was considerable activity and interest in developing CAD algorithms to address this utilizing traditional machine learning techniques⁵⁹. These programs were shown to slightly improve radiologist detection of pulmonary nodules when used concurrently in interpretation. However, sensitivity of these algorithms by themselves were relatively low compared to experienced radiologists. They were also often associated with a high false-positive rate, leading to increased time spent examining false nodules detected by the software. In addition, detection of nodules adjacent to structures such as vessels or pleura tended to be missed by the software⁶⁰. This is an issue for both CAD programs and radiologists, as difficulty with confidently identifying the borders of a nodule can decrease the ability to assess growth and changes over time, a characteristic imperative for lung cancer treatment planning and follow up. More recently, a CAD nodule detection system evaluated by Lo et al. incorporated a pulmonary vessel image suppression function⁶¹. This improved detection of nodules that were initially missed due to their close relationship to vessels, and decreased interpretation time. Lung nodule detection increased from 64% to 80%, though at the cost of a slight reduction in specificity⁶¹. While traditional machine learning approaches like this still yield modest nodule sensitivity in this range of performance⁵⁹, early data from public challenges seems to show that deep learning approaches may improve upon this considerably⁶².

The explosion of available data in conjunction with advances in machine learning has opened the field of image analysis to an exciting direction with potential to predict patient prognosis and even response to treatment. Components of a software package developed at UPenn were used to stratify patients with similar treatments for early Non-small cell lung cancer (NSCLC) into two distinct survival groups. This was achieved using an unsupervised clustering analysis method based on distinctive radiographic imaging features⁶³. This field of image analysis, termed “radiomics”, is geared at characterizing image features and correlating them with tumor phenotype with the intent of classifying and staging tumors noninvasively. Extracted features convert radiographic images into mineable data and can be employed to build predictive and prognostic models²⁴. Subsequently, several groups have used machine learning to associate phenotypic descriptors on CT with overall survival^64,65 or disease-free survival⁶⁶ in NSCLC. A review by Kolossvary et al. also describes how radiomic techniques may be implemented to assist with coronary artery calcium scoring⁶⁷.

The related concept of “radiogenomics” encompasses the effort to combine imaging phenotypes and tumor genetic data to perform better prognostication and target therapeutic decisions. Vardhanabhuti et al. comprehensively reviews the development of this concept in lung cancer⁶⁸. Of note, a recent publication by Zhou et al. found certain nodule imaging characteristics correlated with specific metagene groups, speculating that noninvasive imaging may help direct targeted therapeutic treatment through inference of the genetic or cell surface markers⁶⁹. Significant interest has been placed on attempting to define a radiomic signature for individual gene mutations (eg. EGFR, ALK, K-ras), and to correlate this with treatment response to targeted inhibitory agents^70-72. While machine learning was not utilized in most of these studies to connect gene expression data with individual imaging characteristics of a patient’s tumor, it is possible that deep learning may influence and expedite the next generation of radiomic studies. Specifically, the use of deep learning and its ability to automate the process of “feature engineering” across all scales of imaging phenotypes may provide an opportunity to bridge the gap between genetic, histologic, and imaging data.

The opportunities for machine learning to assist in diagnosis, prognostication and treatment certainly are not limited to cardiac disease and oncologic applications. Other promising areas of thoracic research include the use of machine learning for aortic segmentation, thrombus detection, and fibrotic and COPD disease classification^73-76.

Developing Communities, Competitions, and the Challenges of Public Data Sets

The rapid growth of machine learning technologies and their potential applications has prompted the formation and growth of multiple research conferences. This includes the Conference on Machine Intelligence in Medical Imaging (C-MIMI), and has encouraged the formation of machine learning interest groups within existing radiological societies such as the International Society for Magnetic Resonance in Medicine Machine Learning (ISMRM ML). A number of these conferences and groups have created open challenges aimed at promoting the development of algorithms for specific tasks or to solve clinical problems. The International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), is an annual conference which hosts multiple challenges across many fields, and is notable for highlighting cardiac segmentation, coronary artery reconstruction, and left ventricle shape modeling⁷⁷. Several challenges have also been devoted to thoracic imaging, specifically lung nodule/lesion detection. The International Symposium on Biomedical Imaging (ISBI) offered a challenge called LUNA (LUng Nodule Analysis) in 2016 which called for algorithms that could automatically detect nodules on chest CT. Similar challenges were announced for the SPIE MI 2015 conference, the Data Science Bowl in 2017, and this year’s RSNA conference.

With these challenges, the organizers have offered datasets for the algorithms to be trained and tested on. The availability of large imaging datasets is currently a bottleneck for many areas of research and can be difficult to obtain outside of a major hospital-affiliated research group. Multiple factors contribute to this, including IRB approvals for data gathering, the need to assure patient privacy and anonymity, the complexities of working with DICOM files and their size, and the cost of assembling and maintaining a database. Several public repositories of thoracic CT images are available, however, and have fueled machine learning growth and algorithm development (reviewed in Morris et al.⁷⁸). The National Lung Screening Trial (NLST) enabled the collection of a large volume of patient data for former smokers in an effort to determine the utility of low dose CT screening, a recommendation later made by the US Preventative Service Task Force^56,79. This imaging databank, in addition to blood, sputum, and urine data, has been made available to researchers. Similarly, the UK Biobank study collected genetic data and imaging, and has made this available for research⁸⁰, and the publicly available Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) contains over seven thousand annotated nodules for the purposes of CAD development⁸¹.

While machine learning scientists continue to innovate with the available data and the community works to address the issue of data availability, researchers have been able to use these challenge datasets to achieve excellent performance^25,29,82. For example, the highly utilized UNet architecture was developed for, and won, a segmentation challenge for electron microscopy at ISBI 2012¹⁷. Though primarily developed to segment neuronal structures in electron microscopy stacks, it has since been successfully applied to a variety of biomedical image segmentation tasks^26,83. While these early studies show great promise, the most common training and testing sets from challenges are still limited in scope compared to the range of diseases that would be seen in clinical practice. Thus, these algorithms must yet prove themselves in the real world as larger and more routine clinical datasets become available.

With the rapidly expanding availability of imaging data and desire to develop clinically useful technology, there has been increasing effort to establish guidelines and offer guidance by larger radiological societies. The American College of Radiology (ACR) recently established a Data Science Institute (DSI) to steer the introduction of AI into the practice of radiology. The ACR has begun defining standards for training, testing, validating, integrating, and monitoring AI algorithms in clinical practice. To do this the Data Science Institute has defined “use cases” pinpointing precise scenarios within radiology workflows where potential automation could enhance patient care. These use cases are organized by organ system and include twenty five cardiac and one thoracic application offered in an open source directory labeled TOUCH-AI⁸⁴. The ACR will also evaluate algorithm performance by providing ongoing post-market assessment⁸⁵. A goal of developing standardized pathways for algorithm validation will be to expedite the FDA regulatory review process to allow these modalities to have a clinical impact as soon as possible while maintaining a high level of quality.

FDA Approval Process for New Technologies and Software

The Food and Drug Administration (FDA) currently has two approval processes for new medical devices relevant to imaging analysis software: the 510k clearance and pre-market approval (PMA) clearance. PMA clearance is a more stringent application path, and typically involves clinical trials to demonstrate safety and efficacy. 510k clearance does not require the same rigorous clinical trials necessary for a new class of drugs or devices, but demands that equivalence to existing products or technologies be shown⁸⁶. In December of 2016 the FDA expedited the regulatory process for certain medical devices by adding its Breakthrough Devices provision to the 21^st Century Cures Act. The goal was to “help patients have more timely access to devices and breakthrough technologies”, with submissions ranging from dental implants to deep learning-based computer aided diagnostic programs⁸⁷. Through this, artificial intelligence in some forms can be classified as a medical device, and gain clearance through a separate “de novo” pathway without a predicate device for comparison⁸⁷.

Many applications may fall under the umbrella of computer aided detection (CAD). The FDA notes two major CAD categories: CADe, which includes tools aimed at automating detection and focusing the attention of a clinician onto an area of an image, and CADx which includes tools aimed at assessment or likelihood of a disease and automation of diagnosis⁸⁸. While CADe products fall under the 510k application path, CADx products have typically fallen into the more stringent PMA application path. As CADx products have potential for more significant patient impact, the FDA has historically been more cautious. Recently, however, the requirements for CADx tools have been reduced⁸⁹. Certain cancer detection tools will now fall under a new category of device and can be approved under the less stringent 510K pathway. The growing number of applications integrating neural networks has likely contributed to this change. Even with more lenient approval processes, CAD products will continue to require regulatory attention going forward.

Validation, Utility and Generalizability

As the population ages and becomes more medically complex, the need for imaging will continue to increase. Deep learning-based diagnostic tools offer many potential benefits, and may increase physician efficiency, improve accuracy of diagnosis, and enhance consistency between different sites and institutions. As early adopters, radiologists have the opportunity to set the standards for this technology and direct evaluation of programs to ensure their accuracy and safety for patients. Bringing a technology to the clinic will involve several key elements: peer review, analytical validity, clinical validity, clinical utility, and generalizability. Within these, the overall performance compared to a ground truth, performance on individual components of the task, and mechanisms of failure, should be examined.

While the medical literature has historically undergone a process of peer-review, it is important to recognize that pre-print websites are increasingly used as a form of communication, especially in the machine learning community. In this rapidly changing and evolving field, a repository for the pre-release of manuscripts such as arXiv is valuable for sharing information and distributing cutting edge findings. However, it is important to recognize the limitations and pitfalls of this method of communication. Manuscripts on pre-release repositories are moderated upon submission, but are not required to undergo the peer review process, and new versions may be uploaded or updated at any time. Therefore, authors must use caution when making clinical decisions based on manuscripts from preprint websites until peer-review scrutiny is applied.

Analytical validity is typically assessed during the testing and development of new technologies, and tests the performance relative to pre-existing technologies. For example, an algorithm might measure the caliber of the aorta, and comparison of the measurements between the algorithm and a human observer may be sufficient. It is harder to assess analytical validity when the outcome metric is subjective. In such situations, a consensus among experts may be necessary. For example, when contouring cardiac chambers the apical and basal heart boundaries are a source of disagreement between readers and protocols vary between institutions. However, a consensus average of segmentations from a panel of experts can be helpful for training algorithms⁹⁰. Analytical validity can also be established by analyzing algorithm performance on individual components of a task. Understanding where an algorithm fails can provide insight into areas that require human supervision. If an algorithm has particular difficulty segmenting the cardiac apex, this might not be readily apparent if only the overall similarity scores are considered (Figure 6). Further, these insights may direct research to improve the performance of the algorithms.

Figure 6.

Consideration of the components of the overall task (e.g., dice per slice when segmenting the ventricles of the heart) can help identify sources of error and understanding of areas where oversight may be necessary. Without considering the individual components (e.g., considering only overall dice per volume), systematic errors may be overlooked.

Clinical validity is a concept distinct from analytical validity, and touches upon whether a technology works within the range of performance needed for clinical decision-making. In the example of an algorithm that measures the caliber of the ascending aorta, it is important to assess whether any differences in measurement between the algorithm and the current standard would impact the clinical management. In other words, it may be possible to prove differences between an algorithm and human observer, but these may not be clinically relevant. In the example of ventricular segmentation, the apex of the left ventricle may have considerably different contours between an algorithm and human observers, but may contribute very little to the total volume, and therefore have little impact on the overall volumes or ejection fraction.

Clinical utility is a related concept for determining the potential value of a new tool. This is where radiologists may play a key role in the development of technology. We may routinely only provide linear or bilinear measurements of pulmonary nodules and lymph nodes in daily clinical practice due to efficiency and time-constraints, but it may also be clinically useful to provide volumetric measurements. We may choose not to provide such measurements if the measurement tools are too cumbersome, inaccurate, or work too inconsistently to be useful. Deep convolutional neural networks have potential to address this need if well implemented into the radiologists’ workflow, and clinical domain knowledge is critical for directing the development of new applications. With such collaboration, new technologies will be more likely adopted into practice.

One major consideration for algorithms that are to be used in clinical practice is the extent to which they will generalize to clinical populations. The recent tragic accidents from Uber and Tesla have shown that overestimation of the robustness and generalizability of this technology can yield fatal results^91,92. Many algorithms are trained on a limited patient population, or from public data sets of predominantly normal patients, and may not necessarily work on the typical population seen in clinical practice. Another pitfall is that algorithms may be developed on a single scanner type or imaging technique that is institution-specific. Such algorithms may not generalize to other scanners or protocols. For example, an algorithm designed to detect pulmonary nodules on 1.25-mm hard kernel reconstructions may not work as well on thick slices. As such, the developers of these tools must work to ensure that the patient populations and imaging protocols on which models are developed are representative of clinical patient populations, and users must appreciate limitations of models as they deploy them in clinical practice. One of the major advantages of modern deep learning models is their ability to learn and improve by being “fine-tuned” on new training data. This allows refinement of models, and retraining over time on broader populations.

Conclusion

The applications of deep neural networks and other machine learning technologies to long-standing problems in radiology are rapidly advancing and promise to shape the future of the specialty. Both supervised and unsupervised techniques have been applied in current technologies (see Table 1 and Table 2). They will likely become fundamental to practice, and soon, as pervasive and unnoticeable as other technologies we have integrated into daily use. In many ways these technologies are already a large part of our lives. Phones recognize us, speak to us, manage our schedules, and reroute our wrong turns. Entertainment systems and online shopping trackers recommend things we would enjoy based on previous selections and others with similar behaviors. These machine learning processes are so well used that they have become transparent in our daily activities. Outside of digital transcription, machine learning algorithms are only beginning to emerge in the daily practice of radiology. The breadth of problems that machine learning can help address is immense, and will likely mature rapidly in areas of detection, characterization and prognostication of disease, and individualized treatment decisions. As early adopters of this new technology, radiologists should be cautious consumers, and think critically about the new advancements to ensure that they are safe and effective tools in clinical practice. Integration of machine learning into the daily workflow has the potential to augment our capabilities and make radiologists more efficient, more focused on diagnosis and higher order tasks, and better able to address the needs of referring physicians and patients.