Deep learning and its role in COVID-19 medical imaging

Abstract

COVID-19 is one of the greatest global public health challenges in history. COVID-19 is caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and is estimated to have an cumulative global case-fatality rate as high as 7.2% (Onder et al., 2020) [1]. As the SARS-CoV-2 spread across the globe it catalyzed new urgency in building systems to allow rapid sharing and dissemination of data between international healthcare infrastructures and governments in a worldwide effort focused on case tracking/tracing, identifying effective therapeutic protocols, securing healthcare resources, and in drug and vaccine research.

In addition to the worldwide efforts to share clinical and routine population health data, there are many large-scale efforts to collect and disseminate medical imaging data, owing to the critical role that imaging has played in diagnosis and management around the world. Given reported false negative rates of the reverse transcriptase polymerase chain reaction (RT-PCR) of up to 61% (Centers for Disease Control and Prevention, Division of Viral Diseases, 2020; Kucirka et al., 2020) [2,3], imaging can be used as an important adjunct or alternative. Furthermore, there has been a shortage of test-kits worldwide and laboratories in many testing sites have struggled to process the available tests within a reasonable time frame.

Given these issues surrounding COVID-19, many groups began to explore the benefits of ‘big data’ processing and algorithms to assist with the diagnosis and therapeutic development of COVID-19.

Introduction

While COVID-19 provides the motivation for large scale clinical and imaging data collection and dissemination efforts, opportunities for leveraging artificial Intelligence (AI) are rapidly increasing. The academic field of Artificial Intelligence attempts to both understand and build intelligent entities, and encompasses both Machine Learning and its subfield Deep learning. Deep Learning is realized through algorithms which are structurally composed of artificial neurons and multiple data processing layers in a deep architecture referred to as a Deep Neural Network [1,2]. A subtype, namely Convolutional Neural Networks (CNNs) as shown in Fig. 1 have proven themselves particularly successful due to astonishing breakthroughs in image and video processing [1], and have been of great interest to the radiology community due to its potential applications [3]. The data of interest is input to the network along with its ground truth label; for example the pixel data of a chest x-ray along with the diagnosis ‘pneumonia’. Large amounts of input data are required, and through the so-called ‘training phase’, the processing layers in the neural network are able to learn representations of the input data with higher and higher level of abstraction [1,2]. In a classification task, the output layer of the CNN can finally learn to associate one or several classes with the input image, meaning that in the chest x-ray example the CNN learns to detect and classify pneumonia on chest x-rays [1,4] (see Fig. 2).

Fig. 1.

The architecture of a simple Convolutional Neural Network which are used for image-processing tasks [5].

Fig. 2.

Heat Map generated for a patient with COVID-19 using Chester, a publicly accessible deep learning model for chest x-ray diagnostics [8].

In this review, we discuss some of the ways that AI has played a role assisting with this pandemic, particularly from the point of view of thoracic imaging. Chest X-ray and Chest CT are the two most common imaging studies for diagnosis and management of COVID-19 patients. In their favor, chest radiography and CT scans are readily available at most medical centers, are routinely obtained and usually interpreted with faster turnaround than the SARS-CoV-2 laboratory examination.

To note, while out of the depth of this manuscript, an excellent comparative assessment of 17 CNNs was performed by Elgendi et al. [6] with regards to various architectures to use in the radiographic diagnosis of COVID-19, and they identified an optimal pretrained deep neural network for the task of detecting COVID-19 pneumonia.

Initiatives

The most common imaging study obtained for patients is the chest radiograph. Unfortunately, while carrying a high specificity in this population, the sensitivity on chest x-ray with regards to detection of SARS-CoV-2 is much lower. This has not stopped groups from trying to utilize AI algorithms to help with early detection. In a non-peer-reviewed study [7], a trained model demonstrated an overall sensitivity and positive predictive value both at 92.9%, with results for each label showing sensitivity and positive predictive value at 94.8% and 98.9% for COVID-19 pneumonia, 89% and 91.8% for nonCOVID-19 pneumonia and 95% and 88.8% for normal lung. The group then validated the program using CXRs of patients from their institution with confirmed COVID-19 diagnoses along with non-COVID-19 pneumonia and normal CXRs. They determined that their model performed with 100% sensitivity, 95% specificity, 97% accuracy, 91% positive predictive value, and 100% negative predictive value. These are impressive findings if they can be confirmed via peer review and replicated on a large scale. That said, there remain obvious limitations to the use of chest x-ray for the detection of COVID-19. The radiographic manifestations of COVID are often subtle, although more visible on CT scan.

Several deep learning algorithms for detection of COVID-19 pneumonia on CT scans have been proposed. Bai et al. [9]. Collected CT-scans from 1186 patients with either non-COVID pneumonia or RT-PCR confirmed COVID pneumonia from 11 different hospitals in USA and China. They trained a deep learning model to discriminate between COVID and non-COVID pneumonia reaching an AUROC of 0.90 on an independent test set. Even more exciting, Bai et al. subsequently provided the model output to radiologists, and demonstrated that AI-assistance significantly improved radiologist diagnostic accuracy from 85% to 90% in distinguishing COVID pneumonia from non-COVID pneumonia [9]. As such, incorporating deep learning models for COVID-19 imaging as clinical decision support tools could be a promising approach for leveraging AI capabilities in routine clinical practice in the efforts against SARS-CoV-2.

To address the issue of data availability, particularly in the earlier stages of the pandemic, groups applied transfer learning strategies to their algorithms [10]. Horry et al. were able to apply this strategy to multimodal imaging data (CT, Ultrasound, X-ray) with commendable results [11].

While the majority of published work rely on imaging data only, Mei, et al. [12] used AI algorithms to integrate chest CT findings with clinical symptoms, exposure history and laboratory testing to rapidly diagnose patients who were positive for COVID-19. Among a total of 905 patients tested by real-time RT–PCR assay and next-generation sequencing RT–PCR, 419 (46.3%) tested positive for SARS-CoV-2. In a test set of 279 patients, the AI system achieved an area under the receiver operating characteristic curve (AUROC) of 0.92 and had equal sensitivity as compared to a senior thoracic radiologist. The AI system also improved the detection of patients who were positive for COVID-19 via RT–PCR, but who presented with normal CT scans, correctly identifying 17 of 25 (68%) patients, whereas radiologists classified all of these patients as COVID-19 negative.

Bernheim et al. [13] investigated the most common CT findings in relationship to the time between symptom onset and the initial CT scan i.e. early, 0–2 days (36 patients), intermediate 3–5 days (33 patients), late 6–12 days (25 patients). The hallmarks of COVID-19 infection on imaging were bilateral and peripheral ground-glass and consolidative pulmonary opacities. Notably, 20/36 (56%) of early patients had a normal CT. With a longer time after the onset of symptoms, CT findings were more frequent, including consolidation, bilateral and peripheral disease, greater total lung involvement, linear opacities, “crazy-paving” pattern and the “reverse halo” sign. Bilateral lung involvement was observed in 10/36 early patients (28%), 25/33 intermediate patients (76%), and 22/25 late patients (88%).

Zhang et al. [14]. described their AI system for the diagnosis of COVID-19 pneumonia based on chest CT images. The performance of the system was comparable to that of practicing radiologists with significant clinical experience and could assist and improve the performance of junior radiologists. Augmenting this was the development of a clinical prognostic model based on their AI system utilizing CT parameters and clinical data. This enhances the argument that AI can be used to assist clinical management. Based on their database, they were able to identify a composite score of ≥0.5 as the high-risk group in terms of the eventual progression into severe or critical illness resulting in ICU admission, mechanical ventilation, or death. Importantly, an estimated time to this progression could also be provided.

It is evident that deep learning can play an important role in the detection and classification of COVID-19 on radiological imaging. In addition, using deep learning for segmentation of COVID-19 lung lesions can also be clinically useful. For example, segmentation masks can guide physician attention towards COVID-19 pneumonia lesions, and they can be used to quantify and accurately monitor the extent of lung parenchymal disease. Wang et al. [15] developed a deep learning model for segmentation of COVID-19 pneumonia lesions and achieved a Dice coefficient of 0.80 on a test set of 130 CT scans. Similarly Fan et al. [16] trained a deep learning model to segment consolidation and ground glass opacity separately and achieved a Dice coefficient of 0.62 and 0.46 on each task respectively.

Although chest x-rays and CT scans are most commonly employed for imaging of patients with COVID-19 pneumonia, lung ultrasound has also proven to be effective in visualizing and monitoring lung parenchymal disease caused by SARS-CoV-2 [17]. Lung ultrasound further carries the advantage of not imposing any radiation and eliminating the need for transferring the patient, since the examination can be performed bedside [18]. Roy et al. [19] developed and tested several deep learning models for detection of COVID-19 associated patterns on lung ultrasound scans. Their models concurrently output the severity of lung disease on a 4-point scale and segmented the pathological area on each lung ultrasound scan. Roy et al. argue that their model could be beneficial in a point-of-care assessment of disease severity and also be used for triaging patients with COVID-19 [19].

Common to the efforts in deep learning for COVID-19 imaging is the usage and need for large imaging datasets with solid ground truth diagnosis such as RT-PCR confirmed COVID-19 and final outcomes such as death or time to discharge. Training datasets can preferably be augmented further with the addition of clinical data, possibly even in the form of time-series data containing repeated imaging, vitals and blood tests. Initiatives to publicly disseminate COVID-19 imaging datasets have been made by a few research groups [14,20,21], and the Radiological Society of North America launched the International COVID-19 Open Radiology Database with annotated images [22]. Industry is also paying close attention to the commercial potential of coronavirus detection. In example, Siemens Healthineers’ expert teams and their collaboration partners are working together on the development of dedicated algorithms that help face the challenges caused by the COVID-19 pandemic.

One such example is their algorithm CT pneumonia analysis, which is designed to automatically identify and quantify abnormal patterns in the lungs, enabling simple-to-use analysis of non-contrast chest CT scans for research purposes. The system identifies lungs, lobes and abnormalities associated with pneumonia. It also computes high-opacity abnormalities, as they are shown to correlate with severe symptoms. The results could be used to analyze the severity and progression of abnormalities in patients exhibiting COVID-19 symptoms. The AI-Rad Companion Research CT Pneumonia Analysis prototype is offering a cloud-based option that runs on the teamplay digital health platform.

Alongside academic and industry efforts, the National Institutes of Health has launched and is leading the Medical Imaging and Data Resource Center (MIDRC) [23]. This multi-institutional collaboration seeks to harness the power of artificial intelligence and medical imaging to fight COVID-19 by creating new tools that physicians can use for early detection and personalized therapies for COVID-19 patients.

The MIDRC recognized that features of infected lungs and hearts seen on medical images can help assess disease severity, predict response to treatment, and improve patient outcomes. However, a major challenge is to rapidly and accurately identify these signatures and evaluate this information in combination with many other clinical symptoms and tests. They surmise that some of the newly developed diagnostics will include machine learning algorithms.

It goes without saying that the deployment of these evaluative algorithms must be done in a manner that makes sense at the user level – the Radiologists’ reading station. There are efforts to ensure that the appropriate user interface and integration for these models take place [24]. Given the global nature of this pandemic and to pay attention to future collaborative endeavors, developing interfaces that can be used in low-computing power environments (e.g. environments that have less hardware capacity or internet connectivity) is paramount to successful clinical integration and utility.

Conclusions

While the pandemic continues to change the shape of the current practice of medicine and data sharing around the globe, one point that becomes clear is that Artificial Intelligence will continue to be front and center in these efforts. The use of chest imaging techniques to serve as primary, secondary, or adjunctive diagnostics is ever-increasing and will only continue to grow. AI algorithms can also help in a predictive fashion, forecasting time to clinical deterioration or improvement. As collaboratives and consortiums develop, the availability of imaging data has also been less controversial, further catalyzing advancements as large diverse datasets are leveraged for machine learning applications. Overall, the sharing of data and innovation within the global machine learning research community that this pandemic has facilitated ultimately bodes well for the future of data sharing efforts using AI in healthcare during this pandemic and beyond.

Declaration of competing interest

No conflicts of interest.