Engineering and clinical use of artificial intelligence (AI) with machine learning and data science advancements: radiology leading the way for future

Abstract

Over the years, many clinical and engineering methods have been adapted for testing and screening for the presence of diseases. The most commonly used methods for diagnosis and analysis are computed tomography (CT) and X-ray imaging. Manual interpretation of these images is the current gold standard but can be subject to human error, is tedious, and is time-consuming. To improve efficiency and productivity, incorporating machine learning (ML) and deep learning (DL) algorithms could expedite the process. This article aims to review the role of artificial intelligence (AI) and its contribution to data science as well as various learning algorithms in radiology. We will analyze and explore the potential applications in image interpretation and radiological advances for AI. Furthermore, we will discuss the usage, methodology implemented, future of these concepts in radiology, and their limitations and challenges.

Introduction

In a world of fast-advancing technology, a vast amount of data are generated and constantly increasing.¹ Given appropriate tools, these data can be captured and could provide a very insightful resource. Data capture for this purpose has already been implemented in various fields such as defense, healthcare, media and communication, cybersecurity, and banking, to name a few.

Evolution in hardware and software application has led to an escalating number of tasks performed by machines that were initially unimaginable.² The most noteworthy tool has been the introduction of learning algorithms. Tasks can now be performed, which were previously limited to humans, thus indicating that these algorithms have significantly improved recently.³ Among the myriad methods available, it is comparatively less challenging to train deep learning (DL) algorithms; however, the downsides are vast data requirements and high supervision during the interpretation of results.⁴ For example, in DL systems, the activation function rectified linear unit (ReLU) outshined the sigmoid function in terms of performance. To understand this better, the ReLU function calculates the sum of input values, and if it falls below 0, it gives an output of 0; else, the output will be the same as the input.^5–7

In addition, DL algorithms have been able to outdo the performance of other AI approaches and medical experts in specific tasks such as recognizing pneumonia on imaging scans.^8–10 In 2012, a DL algorithm first entered this field, which dramatically reduced the error rate from 0.258 to 0.153 in 1 year. After 3 years, in 2015, using DL methods, the quality of error decreased to the extent that it was the lowest value obtained by human observers.¹¹

While radiology is an expanding field, these novel learning algorithms could improve its standards and clinical efficiency by achieving swift clinical and economic benefits. Healthcare today is an evolving technological landscape in which data-generating devices are increasingly ubiquitous, creating what is commonly referred to as ‘Big Data’. Researchers are investigating how these quantifications benefit radiological improvements, and quite enthralling pioneers have been identified. In their current application, a methodology where a vast number of quantitative features are extracted using mathematical equations from radiographic medical images is known as radiomics.¹² The radiomics features can be categorized into texture features (wavelet texture, laws’ texture, run-length encoding, gray-level co-occurrence matrix, etc.), first- and second-order statistics features (homogeneity, entropy, standard deviation, skewness, etc.), and shape-size features (volume, longest diameter, etc.). Radiomics can offer a comprehensive understanding of radiographic qualities of fundamental tissues. The acquired information can be used throughout the clinical care path to improve diagnosis and treatment planning, as well as assess the potential and subsequent response to treatment.¹² Figure 1 shows the patient journey in conventional versus artificial intelligence (AI) setup for diagnosis and prediction of the disease.

Figure 1.

Patient journey: conventional versus artificial intelligence.

In this article, previously used learning algorithms will be explained and those most useful to the field of radiology will be described, examining current and upcoming applications. We will endeavor to evaluate how this noteworthy new-fangled tool will be able to benefit radiologists in the future.¹³ The concept of data science and various learning algorithms including the application of these radiology techniques will be reviewed. Furthermore, we will discuss various roles these techniques play, their pitfalls, advantages, and scope in the field of radiology.

Methods

Search strategy and article selection

A non-systematic review of all relevant English-language literature in Radiology that were published in the last decade (2010–2020) was conducted in June 2020 using MEDLINE, Elsevier, Springer, RSNA, Scopus, and Google Scholar. Our search strategy involved creating a search string based on a combination of keywords. They were as follows: ‘Data Science’, ‘radiology’, ‘Artificial intelligence’, ‘AI’, ‘Machine learning’, ‘ML’, ‘ANN’, ‘Neural networks’, ‘convolutional networks’, ‘Deep Learning’, ‘DL’, ‘MRI’, ‘CT scans’, ‘X-Ray’, ‘healthcare’, ‘diagnosis’, ‘medicine’, ‘algorithms’, ‘image processing’, ‘image recognition’, and ‘image reconstruction’. This review includes only the original research work published in English.

Inclusion criteria

1. Articles on Radiology that address data science, DL, and machine learning (ML).

2. Full-text original articles on all aspects of analysis, algorithms, and outcomes of radiological diagnosis.

Exclusion criteria

1. Commentaries, articles with no full-text context, and book chapters.

2. Articles without image processing and articles that do not address radiology directly but are generic healthcare articles.

The literature review was performed, conforming to the inclusion and exclusion criteria. The evaluation of titles and abstracts, screening, and full article text was conducted for the chosen articles that satisfied the inclusion criteria. Furthermore, the authors manually reviewed the selected articles’ reference list to screen for any additional work of interest. The authors resolved the disagreements about eligibility for a consensus decision after discussion.

Results

Contemporary history of data science and learning algorithms

AI is expressed as ‘the capacity of computers or other machines to exhibit or simulate intelligent behavior’, and it is now a flourishing area and the subject of extensive research in several fields.^14,15 The advent of ML, which is the potential of an AI system to extract information from un-processed data (raw data) and learn from experience, was also growing. Thus, the need for ‘human operators to formally define all the information needed by computers’ was avoided to an extent.¹⁶

ML algorithms were developed and used to evaluate healthcare data sets. ML provides a number of vital tools for performing intelligent data analysis. The digital revolution has made data collection and storage extremely inexpensive and accessible. Modern hospitals have state-of-the-art monitoring and data collection systems. In contrast to rule-based algorithms, ML takes advantage of increased exposure to large and new data sets and has the ability to improve and learn with experience.¹⁴

DL is often referred to as being one of the most complex collections of algorithms. Although DL has made considerable strides in recent times, some may consider DL algorithms a hindrance to healthcare and radiology.¹⁶ DL is nevertheless a useful tool, although it depends on the requirement and usage, and is much easier to train. Figure 2 shows the pathway of artificial intelligence (AI) in various medical imaging modalities.

Figure 2.

Pathway of artificial intelligence in medical imaging.

To understand the above terms, one must be able to differentiate DL algorithms from ML algorithms, especially the data presentation technique in the system. ML is a branch of AI that enables machines to learn the shape of data as well as adapt and improve their performance without the need for human intervention. DL is a class of ML algorithms that gradually extracts higher level features from the raw data using several layers. Almost always, ML algorithms require structured data, whereas DL networks rely on artificial neural network (ANN) layers. The biological neural network of the human brain is the inspiration for ANNs. An ANN is made up of numerous artificial neurons, each with its own weight, that form a layer (or level). A multilevel representation or network, which is a fully connected network, is created by connecting neurons on one level to neurons on the next level in a multitude of ways. An ANN is built up of input (shape of input data), hidden (one or multiple level), and output layers (number of class labels). Because each successive layer is interconnected in an ANN, the number of weights will rapidly increase as the number of layers increases, affecting the scaling and learning of ANN. For image analysis, this problem can be solved by applying a series of small filters (convolution filters) to the input image and subsampling the filter activation space until there are enough high-level features to use ANN, which leads to Convolutional Neural Networks (CNNs). CNNs are currently a very successful solution for image classification and recognition applications. A CNN can be built with a few convolutional layers with some activation function, which are frequently accompanied by a max-pooling layer for subsampling. Following the acquisition of high-level features via subsampling and convolution, the high-level features are fed through the fully connected layers (hidden layer and output layer) to achieve the final decision. Lately, DL has gained significant recognition in the consumer world and the healthcare community, while ML algorithms have been a research subject for several years.¹⁷ The difference between DL and ‘traditional’ ML is extensively discussed over its functionality and capability. The distinction is critical, especially in the context of medical imaging. In traditional machine learning, the first step typically is feature extraction. This indicates that in order to classify an item, one must first establish which qualities are significant and then implement algorithms that can classify the object,¹⁸ whereas DL utilizes several layers of algorithms to find patterns and imitate human cognition. DL algorithms have recently gained a resurgence of interest after observing a 10% decrease in the top-5 error on the ImageNet data set.¹⁷ DL outperforms traditional ML as data volume increases and does not perform well when the data are small. A large volume of data is required for DL algorithms to fully understand the data. High-end devices are required for the majority of DL algorithms, while traditional ML algorithms can run on low-end machines. Mathematical processes such as matrix multiplication are inherent to DL algorithms.

It usually takes a long time to train a DL algorithm. An algorithm’s training time is longer than typical because there are so many parameters involved. Training the state-of-the-art DL system ‘ResNet’ takes approximately 2 weeks. ML, on the contrary, can be trained in a matter of seconds to a few hours.

The era of modern medicine

Statistics is a critical feature of ML, and statistical interference is the basis of ML algorithms. Different ML models are developed using different statistical models, and these are crucial steps in making the right assumptions while training the data. Statistics alone may not be able to bring about a change in radiology, but its application in promising fields will give us better insights when used in radiology. The development of AI-driven applications in healthcare has significantly augmented the demand for image identification tools. ML and DL algorithms are widely used in various domains of healthcare and are applied to the images for extraction of features which help to provide valuable insights into diagnosis and patient treatment (Figure 3).

Figure 3.

Global market for machine leaning (ML)-based image analysis.

Radiology has dealt with accelerated technological advancements almost as much as in any other clinical field. Since radiology is fundamentally a data-driven discipline, it is particularly suited to the use of data processing methods. The world market for algorithms in image analysis is growing rapidly, and progress in ML and its applications, including medical imaging, seems to be quite promising.¹⁹ ML enables computers to understand like humans and extract or identify patterns. Therefore, machines can process more data sets and derive data characteristics than we can do manually.²⁰ One of the most challenging tasks during clinical image interpretation is identifying the disease and swiftly separating anomalies from normal anatomy.

Google’s Peter Norvig has demonstrated that vast volumes of information and data will address weak points in ML algorithms.²¹ Narrow-scope ML algorithms do not need vast data from testing but might need data from high-quality ground truth training. In medical imaging research, as with some other ML forms, the volume of data required differs significantly according to the task at hand. For instance, segmentation tasks may need only limited data; however, performance classification tasks such as classifying a liver lesion as malignant or benign may require many more labeled data. This ultimately depends on the number of classification methods used to differentiate and classify a lesion as malignant or benign.²²

Learning algorithms in radiology: a breakthrough

DL approaches and ML algorithms, in general, have a tremendous opportunity to shape the radiology practice. Disease identification is one of the most challenging and time-consuming tasks, especially when interpreting images and identifying them as normal or abnormal findings. The type of data used in radiology is mostly ‘labeled data’. Since we are dealing with labeled data, the supervised learning model is preferred in most cases of radiology. The algorithm is trained with enough labeled data to build an optimal model. A substrate for developing ‘learning algorithms’ is provided by a substantial quantity of image and report data, now available in digital form (‘Big Data’). The inability to compile a sufficiently large training set is a possible setback, resulting in less reliable or generalizable tests. Figure 4 shows the learning algorithms and their practical utility that are implemented in radiology applications.²³

Figure 4.

Clinical applications of machine learning in radiology.

Image acquisition

ML can make imaging applications smart and time-efficient. ML-based models of data analysis can minimize imaging time. Thus, intelligent imaging systems may eliminate needless imaging, improve orientation, and improve detecting characterization. For example, smart magnetic resonance imaging (MRI) will identify a lesion and propose changes in the subsequent sequence to obtain favorable lesion characterization.²⁴ This will, therefore, benefit patients from faster scanning and enhanced and improved scan reports.

Automated detection and interpretation of findings

ML can detect significant unintended findings. Certain fields that may benefit from potential ML include cancer detection through mammograms and diagnosis of pulmonary nodules. Examining the bone age and automated assessment of anatomical age based on medical imaging has great utility for pediatric radiology and endocrinology. Additional ML studies that were carried out identified important findings such as pneumothoraces, fractures, organ lacerations, and stroke.^25–31

Interpretation of scans is a frequent discussion area and the topic of contention as to whether such techniques could/would eventually fully replace radiologists. The interpretation of detected findings ‘requires a high level of expert knowledge, experience, clinical judgment and correlation based on each clinical scenario’. Most opinionated leaders in this field have indicated that radiologists should utilize these techniques to increase their reporting accuracy rather than replacing them. With ML, extraction of the features from breast MRI might enhance the analysis of breast cancer diagnostic results.²⁶ An ML approach based on radiological attributes (semantic characteristics such as contour, texture, and margin) of incidental pulmonary nodules was shown to enhance cancer prediction accuracy and diagnostic understanding of the same.²⁷

Post-processing: image segmentation, registration, quantification

With the accessibility of large image data set and using ML, medical imaging has made substantial advances in post-processing activities such as image recognition, segmentation, and quantification. Studies have demonstrated how to use ML to segment images and aid radiologists with image synthesis, image analysis, and image quantification. For example, an algorithm based on a CNN to segment the volume of adipose tissue on computed tomography (CT) scan images or even an estimate of CT images from the accompanying MR images can be rendered using an adversarial generative network.³² ML can also be applied for the quantitative assessment of three-dimensional (3D) structures in cross-sectional imaging.^33,34

Image quality analytics

The image quality and accuracy are crucial in radiology (Figure 5). However, the large number of images, the greater the time required for analysis. To solve this issue, numerical observers in ML (also known as model observers) have been developed as a proxy for image quality measurement for human observers. With ML and DL advancements, it is now possible to detect or even anticipate bad image quality immediately, enabling radiologists to resolve these irregularities before reviewing the images.³⁵

Figure 5.

Graphical representation of detection versus error in prediction.

Although significant technical advancements in ML have been made in recent years, including improvements in DL algorithms, further progress in the performance of graphics processing units (GPUs) and memory will enhance its capabilities.

Thinking beyond image analysis

The learning algorithms aid radiologists in image analysis and interpretation. However, they also have a use beyond imaging. For instance, the introduction of speech recognition and translation of texts in the final radiology reports was advantageous to patients as they could be translated into various languages and could reduce the potential for errors.^1,36–38 These algorithms are continuously learning and can handle various tasks apart from image analysis. (Table 1).

Table 1.

Beyond image analysis: detection of medical conditions using learning algorithms.

Authors	Objective	Model and algorithm	Outcome	Results
Wu et al.39	Diagnosis of breast cancer	• A three-layer ANN is used • The interpretation is based on features retrieved from mammograms by radiologists	The performance of ANN was better than that of radiologists alone, to distinguish between benign and malignant lesions	Yielding a value of 0.95 for the area under the curve (AUC)
McBee et al.40	Detect metastatic disease	A five-layer CNN is developed to detect metastatic disease	This result was complementary to the low false-positive rates obtained by researchers using a DL algorithm for ophthalmologists	The model achieved an image-level AUC score above 97%
	Diabetic retinopathy	A five-layer CNN with two different validation sets and different set points		It achieved 90.3% and 87.0% sensitivity and 98.1% and 98.5% specificity
Wang and Summers20	Neurological disease diagnosis	• Kernel-based learning methods in radiology are used in computer- aided design (CAD) • SVMs for detection of micro calcifications on mammography	ML algorithm allows radiologists to make decision on data like traditional X-rays, CT, MRI, PET, and radiology records	It is likely to be applied in reinforcement learning. The CAD system reported 80% sensitivity.
	Text analysis of radiology reports	• Converting text into a structured format using NLP. • This enabled computers to derive meaning from human language.		Tests on a set of 230 radiology reports showed high accuracy and recall in text recovery.
Hussain et al.41	Prostate cancer detection	• The Gaussian kernel method is used • It is used to solve specific applications involving many, heterogeneous types of data with the help of support vector machine (SVM)	80 cancerous tissues were perceived efficiently with a high specificity and sensitivity	Accuracy of 98.34% with AUC of 0.999, specificity of 90–95%, and sensitivity of 92–96%
Jnawali et al.42	Intracranial hemorrhage (ICH) detection	• A DL algorithm consists of 1D CNN, long short-term memory (LSTM) units • Trained and tested on a data set CT radiological report	LSTMs are able to extract key ICH features, already been proved using a weakly labeled data set in the computer vision problems	The AUC is 0.94, which is very promising
Al-antari et al.43	Breast cancer detection	• First model: It randomly extracts four regions of interest with a size of 32 × 32 pixels from a detected mass • Second model: The whole detected breast mass is utilized • Using deep belief network (DBN).	The outcomes prove that the proposed DBN outperforms the traditional classifiers	The overall accuracies of DBN are 92.86% and 90.84% for the two ROI techniques, respectively.
Dikaios et al.44	Prostate cancer detection	• A layer-wise unsupervised pre-training model, followed by supervised fine-tuning • Each layer is pre-trained using a restricted Boltzmann machine (RBM)	DBN outperformed the other models; when extended to an individual patient cohort, their performance was not significantly reduced	DBN performed better than the other models, demonstrating value of ROC-AUC = 0.811
Pavithra and Pattar45	Pneumonia or lung cancer detection	• Two stages of feature extraction and classification; feature extraction is done using Gabor filter • Classification is using neural network. • Feed-forward network (FFN) and radial bias function (RBF)	Three types of ANN classifiers were studied, out of which, FFN and RBF network have shown best performance	Accuracy been obtained is about 94.8% for FFN. Accuracy is 94.82% for RBF network

ANN, artificial neural network; AUC, area under curve; CAD, computer-aided design; CNN, convolutional neural network; CT, computed tomography; DBN, deep belief network; DL, deep learning; FFN, feed-forward network; ICH, intracranial hemorrhage; LSTM, long short-term memory; ML, machine learning; MRI, magnetic resonance imaging; NLP, natural language processing; PET, positron emission tomography; RBF, radial bias function; RBM, restricted Boltzmann machine; ROC, receiver operating characteristic; ROI, region of interest; SVM, support vector machine.

Optimize MR scanner utilization

MRI scanning is particularly time-consuming due to the variety and number of sequences often required, improvements in efficiency, and the number of scans/sequences obtained per time could reduce cost. An ongoing fundamental study utilized neural networks to help decide the ideal time allotment per scan dependent on different information boundaries (scan protocol, patient age, contrast utilization, and convention mean of impromptu sequence recurrences). This can optimize scanner usage and diminish costs.⁴⁶

Image reconstruction

Recently, a deep image prior (DIP) framework was proposed, which showed that CNNs have the intrinsic ability to regularize various problems without pre-training. Since no prior training pairs are needed, random noise can be employed as the network input to generate denoised images.^47,48 In addition to using training pairs to perform supervised learning, many methods focus on exploiting images acquired from the same patient from a previous scan to improve the quality of the image. To test the effectiveness of the conditional DIP framework, an experiment was performed by using either the uniform random noise or the patient’s MR image from before as the network input. ML reconstruction of the real brain data at the 60th iteration was treated as the label image and 300 epochs were run.⁴⁹ When the input was random noise, the image was smooth, but some cortex structures could be recovered. When the input was MR image, more cortex structures showed up.⁴⁹ As per the studies, compared with the ML-plus-filter method and the CNN Penalty method, the kernel method and the DIP Recon method can recover more cortex details, and the image noise in the white matter is much reduced.⁴⁹

In another method, DL was used to rapidly perform phase recovery and reconstruct complex-valued images of the specimen using a single intensity–only hologram. This process was less time-consuming and necessitated approximately 3.11 s on a GPU-based laptop or computer to recover the phase and amplitude images of a specimen.⁵⁰

Radiology reporting and analytics

By applying ML methods to natural language processing (NLP), researchers can extricate information from free-text radiology reports, additional data and estimations from descriptive radiology reports, and even track commendations made by radiologists to alluding doctors. There have been efforts to extricate clinically significant information from such content with progress in NLP, including important findings, which could help in the detection of breast cancer with the help of Breast Imaging-Reporting and Data System (BI-RADS) classes and discoveries. Fleischner Society commends frequently utilizing heuristic methods.^51–55 ML and NLP algorithm calculations could help track radiologists’ proposals and help in the follow-up.⁵⁶

Creation of study schedules

Creating study protocols is a part of a radiologist’s role.⁵⁷ It includes referring to health records, information on the clinical details and imaging request, previous images, and radiology reports. It is a crucial step and has radiation dose implications for patients. By applying data interpretation to the available information, these learning algorithms can precisely ascertain imaging protocols, saving time and money.^58,59

Scheduling, screening, and optimization

One significant way radiomics is changing healthcare is by ‘well man or woman’ screening, thereby improving general health and using AI to recognize patients at risk of harm. These advances likewise can help improve on-going screening. AI applications are can help in understanding well-being via patient screening⁶⁰ or add an addendum to existing reports which can potentially change radiology practice.⁶¹

Automated radiation dose estimation

ML algorithms could support radiologists and technologists by making dose estimations before tests and helping to reduce the dosage. Many studies to improve image quality and decrease radiation dose in CT scans are being carried out. They want to reduce CT radiation, even though these outcomes compromise with expanded image noise and lower quality of images because there are confinements of the usually utilized filtered back-projection reconstruction strategies. A portion of the more current iterative recreation advances has diminished noise in images produced with lower dosages.⁶² DL can decrease radiation dose significantly. The idea is to train a classifier for mapping out ‘noisy’ images produced from ultra-dose CT protocols to excellent quality images from standard protocols, utilizing DL strategies.^63,64

Complications and future of learning algorithms in radiology

While a significant number of training models are accessible for issues identified with images, the data sets for clinical images are usually very small, with an average number of patients on the scale of hundreds. The huge number of parameters in a deep neural network that requires streamlining brings about a great danger of over-training and ensuing low performances on data sets that were not utilized in the training procedure. Given the probability of over-training and over-fitting of images, a high probability of reporting performance does not show the actual capacity of a model to group, predict, or segment when the validation is not handled appropriately. Every new development contributes to computational speed, although this does not mean fewer flaws. The inability to gather an adequately sized training set is a potential limitation, which could impact the outcomes, making them less precise or generalized.⁴ This complication of over-fitting was already noted.⁵⁰

Concerning the future scope of learning algorithms, there is an overall understanding that DL will have a significant role in the practice of radiology, especially for MRI. There will be no lack of data as they are continuously generated; learning algorithms can extract information and make useful predictions and outcomes. Some anticipate that DL algorithms will handle enormous tasks, leaving radiologists with more opportunity to concentrate on more clinical duties such as multidisciplinary meetings (MDTs) and intellectually taxing tasks. In contrast, others agree that radiologists and DL algorithms will work together to produce a performance that is better than when working as individuals. Finally, some foresee that DL algorithms will supplant radiologists (at least in their image interpretation analysis) altogether. It seems that the future of AI is intricately linked to radiology, and specialist scan reporting will increasingly use AI to work synergistically.

Conclusion

This article discusses the current practices using ML algorithms and advancements in this field. Although the extent of their influence on radiology is unclear, these algorithms give a promising new arrangement of methods for cross-examining image data that ought to be analyzed with robustness. Radiology has needed to cope with technological change as much as any other branch in healthcare, with radiologists having profited enormously from working with advanced digital frameworks. Yet, concerns exist about machines taking these jobs away from humans, showing a potential, even likely, cultural boundary to accept DL in radiology.

Although some envision that machines may replace radiologists in the near future, most healthcare leaders do not accept that idea. It is believed that machines may help to improve the accuracy, but will not eradicate radiologists’ need. The job of a radiologist is not only to analyze images and identify the pathology, but it also extends much beyond just interpreting images into clinical knowledge and correlation, and so although machines can perform many sophisticated tasks, they may or may not be able to replace medical practitioners in their application beyond image diagnosis.

In addition, if the quantity of the available labeled image is constrained for a given application, it will be exigent to train DL algorithm frameworks or systems, and there is a danger of ‘over-fitting’ the information with an increase in generalization and lack of specific detail. Data science and learning algorithms contribute extensively to the healthcare field, and there is a positive growth in this field. More healthcare practitioners are becoming aware of these techniques, and the true potential of these tools will be harnessed in the years to come.