A deep transfer learning model with classical data augmentation and CGAN to detect COVID-19 from chest CT radiography digital images

Abstract

The Coronavirus disease 2019 (COVID-19) is the fastest transmittable virus caused by severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2). The detection of COVID-19 using artificial intelligence techniques and especially deep learning will help to detect this virus in early stages which will reflect in increasing the opportunities of fast recovery of patients worldwide. This will lead to release the pressure off the healthcare system around the world. In this research, classical data augmentation techniques along with Conditional Generative Adversarial Nets (CGAN) based on a deep transfer learning model for COVID-19 detection in chest CT scan images will be presented. The limited benchmark datasets for COVID-19 especially in chest CT images are the main motivation of this research. The main idea is to collect all the possible images for COVID-19 that exists until the very writing of this research and use the classical data augmentations along with CGAN to generate more images to help in the detection of the COVID-19. In this study, five different deep convolutional neural network-based models (AlexNet, VGGNet16, VGGNet19, GoogleNet, and ResNet50) have been selected for the investigation to detect the Coronavirus-infected patient using chest CT radiographs digital images. The classical data augmentations along with CGAN improve the performance of classification in all selected deep transfer models. The outcomes show that ResNet50 is the most appropriate deep learning model to detect the COVID-19 from limited chest CT dataset using the classical data augmentation with testing accuracy of 82.91%, sensitivity 77.66%, and specificity of 87.62%.

Introduction

At the end of February 2003, the Chinese population was infected with a severe acute respiratory syndrome (SARS) virus causing in Guangdong province in China. SARS was named SARS-CoV and confirmed as a member of the beta Coronavirus subgroup [1]. In 2019, Wuhan in China was infected by a 2019 novel Coronavirus that killed hundreds and infected thousands of humans within few days of the 2019 novel Coronavirus epidemic. The World Health Organization (WHO) named the novel virus as 2019 Coronavirus (2019-nCov) which can cause respiratory disease and severe pneumonia [2]. In 2020, the International Committee on Taxonomy of Viruses (ICTV) announced the 2019 Coronavirus as severe acute respiratory syndrome Coronavirus-2 (SARS-CoV-2) and the disease as Coronavirus disease 2019 (COVID-19) [3–5]. The family of Coronaviruses is alpha (α), beta (β), gamma (γ), and delta (δ) Coronavirus. 2019-nCov was reported to be a member of the β group of Coronaviruses. The SARS Coronavirus epidemic affected 26 countries and outcomes in more than 8000 cases in 2003. The SARS-CoV-2 epidemic infected more than 1.5 million individuals with death rate of 4%, across 150 countries, till the date of this writing. The transmission rate of SARS-CoV-2 is higher than SARS Coronavirus because S protein in the RBD region of SARS-CoV-2 may have enhanced its transmission [6].

In 2012, Middle East Respiratory Syndrome (MERS) was reported in Saudi Arabia as an illness caused by a Coronavirus. SARS and MERS are BetaCoronaviruses (β-CoVs or Beta-CoVs) that are transmitted to people from some cats and Arabian camels, respectively [7, 8]. The sale of wild animals may be the source of Coronavirus infection. The discovery of multiple offspring of pangolin Coronavirus and their similarity to SARS-CoV-2 suggests that pangolins should be considered as possible hosts of novel Coronaviruses. WHO recommendations were to reduce the risk of transmission of Coronavirus from animals to humans in wild animal markets [9]. Person to person Coronavirus transmissions have been reported in different cases outside China, namely in Italy [10], USA [11], Germany [12], and Vietnam [13], Nepal [14]. On 11 April 2020, SARS-CoV-2 claimed more than 1.7 million cases, 400,000 recovered cases, and 100,000 death cases. Figure 1 shows some statistics about recovered and death cases of COVID-19 [15].

Fig. 1.

Statistics of COVID-19 in some countries

Deep transfer learning (DTL) is a deep learning technique that reuses a trained deep learning model that is inspired by neurons of the brain [16, 17]. DTL is quickly becoming a critical technique in image/video classification and detection. DTL improves medical systems to realize higher outcomes, widen illness scope, and implement applicable real-time medical image [18–20] disease detection systems. In 2012, Krizhevsky and et al. and Ciregan et al. [21, 22] showed how convolutional neural networks (CNN/ConvNet) based on graphics processing unit (GPU) can enhance many image benchmark classification such as MNIST [23], Chinese characters [24], NORB (jittered, cluttered) [25], traffic signs [26], large-scale ImageNet [27], Arabic digits recognition [28], and Arabic handwritten characters recognition [29]. In the following years, various advances in CNN further decreased the error rate on the image/video classification competition. Many DTL models were introduced as AlexNet [22], VGGNet [30], GoogleNet [31], ResNet [32], Xception [33], DenseNet [34], Inception-V3 [35].

This section surveys the recent scientific papers on applying deep learning in the field of medical chest computerized tomography (CT) and X-ray images [36] classification. Christe et al. [37] introduced a computer-aided detection method based on deep learning that was able to detect idiopathic pulmonary fibrosis with similar accuracy to a human reader. The proposed system used for the automatic classification of CT images into 4 radiological diagnostic categories. The model achieved an F-score (harmonic mean for precision and recall) of 80%. In [38], the authors introduced a novel system for the automated classification of interstitial lung abnormality patterns in computed tomography images. The proposed system was an ensemble of deep convolutional neural networks (DCNNs) that detects more features by incorporating dimensional architectures. The outcome of the ensemble is a sensitivity of 91,41% and specificity of 98,18%.

In this paper, we introduce DTL models to classify limited COVID-19 chest CT scan digital images. To input adopting CT images of the chest to the DCNN, we enriched the medical chest CT images using classical data augmentation and CGAN to generate more CT images. After that, a classifier is used to ensemble the class (COVID/NonCOVID) outputs of the classification outcomes. The proposed DTL models were evaluated on the COVID-19 CT scan images dataset. The novelty of this research is conducted as follows: (1) The introduced DTL models have end-to-end structure without classical feature extraction and selection methods. (2) We show that data augmentation and conditional generative adversarial network (CGAN) is an effective technique to generate CT images. (3) Chest CT images are one of the best tools for the classification of COVID-19. (4) The DTL models have been shown to yield very high accuracy in the limited COVID-19 dataset. The rest of the paper is organized as follows. Section 2 discusses the dataset used in our research. Section 3 introduces the proposed models, while Sect. 4 illustrates the achieved outcomes and its discussion. Finally, Sect. 5 provides conclusions and directions for further research.

Dataset

The COVID-19 CT scan digital images dataset [39] utilized in this research was created by Zhao et al. (https://github.com/UCSD-AI4H/COVID-CT). The authors collected 760 preprints about COVID-19 from bioRxiv1 (https://www.biorxiv.org) and medRxiv2 (https://www.medrxiv.org) posted from January 19 to March 25 that report patient cases of COVID-19 CT. The dataset is organized into 3 folders (train, validation, and test) and contains subfolders for each image category (COVID/NonCOVID). There are 742 CT images and 2 categories (COVID/NonCOVID). The number of images for each class is presented in Table 1. Table 1 also illustrates the proposed method to increase the number of COVID-19 CT images using augmentation and CGAN. Figure 2 illustrates samples of CT images used for this research.

Table 1.

Number of images for each class in COVID-19 CT dataset

Dataset	Train set		Validation set		Test set
	COVID	NonCOVID	COVID	NonCOVID	COVID	NonCOVID
COVID-19	191	234	60	58	94	105
COVID-19 + Aug	2292	2808	720	696	94	105
COVID-19 + CGAN	2191	2234	210	208	94	105
COVID-19 + Aug + CGAN	4292	4808	870	846	94	105

Fig. 2.

Samples of the used COVID/NonCOVID CT images used in this research

Proposed model

The proposed architecture consists of two main components; the first component is the data augmentation using classical data augmentation techniques along with CGAN, while the second component is the DTL model as shown in Fig. 3. Mainly, the classical data augmentation and CGAN are used in the preprocessing phase while the DTL is used in the performance measurement phase.

Fig. 3.

The proposed architecture of the classical data augmentation along with CGAN and DTL models

Algorithm 1

It introduces the proposed model in detail below. Let $M=$ {AlexNet, VGGNet16, VGGNet19, GoogleNet, and ResNet50} be the set of DTL models. Each DTL is fine-tuned with the COVID-19 CT Images dataset $\left(X_{\mathit{input}},Z_{\mathit{output}}\right)$, where $X_{\mathit{input}}$ is input of the set of $N$ images, each of size, 256 × 256, 227 × 227, and 224 × 224, the $Z_{\mathit{output}}$ contain the corresponding labels, $Z_{\mathit{output}}=\{z/z\in$ {COVID; NonCOVID}}. $\mu$ is a hyper-parameter that updates the weights of our DTL. The dataset is divided into train, validate, and test, training set $\left({\text{X}}_{\text{train}};{\text{Z}}_{\text{train}}\right)$, validate set $\left({\text{X}}_{\text{val}};{\text{Z}}_{\text{val}}\right)$, test set $\left({\text{X}}_{\text{test}};{\text{Z}}_{\text{test}}\right)$. The training data is then divided into mini-batches, each of size $n=32$, such that $\left(X_i;Z_i\right)$ $\in$ $\left({\text{X}}_{\text{train}};{\text{Z}}_{\text{train}}\right)$; $i=1,2,\dots ,\frac{N}{n}$ and iteratively optimizes (fine-tuning) the DCNN model $d\in D$ to reduce the empirical loss as illustrated in Eq. (1).

$$L\left(w,X_i\right)=\frac{1}{n}\sum _{x\in X_i,z\in Z_i}l\left(m\left(x,w\right),z\right)$$ 1

where $l\left(.\right)\text{is}\text{the}\text{binary}\text{entropy}\text{loss}\text{penalty}\text{function},\text{and}m\left(x,w\right)$ is the DCNN model that predicts class $z$ for input $x$ given $w$ is a weight.

Deep transfer learning

DTL is the most successful reuse type of deep convolutional neural network model for image/video classification. A single DTL model contains many different layers of convolution and pooling layer that work on feature extraction from image/video and more complex deep features in deeper layers.

Let layer $l$ be a convolutional layer. Suppose that we have some $N\times N$ square neuron nodes which are followed by a convolutional layer. If we use an $M\times M$ filter (mask) $W$, then convolutional layer output will be of size $\left(N-M+1\right)\times \left(N-M+1\right)$ which produces $k$-feature maps that are illustrated in Fig. 4. The convolutional layer acts as a feature extractor that grabs features of the inputs. The convolution layer extracts features from the image like edges, lines, and corners, to compute the pre-nonlinearity input to some unit. Then, the input of layer $l-1$ comprises is computed in Eq. (2):

$$Z_i^l=B_i^l+\sum _{a=1}^N\sum _{b=1}^N{W}_i{X}_{\left(i+a\right)\left(j+b\right)}^{l-1}$$ 2

where $B_i^l$ is a bias matrix and $W_i$ is the mask of size $M\times M$. Then, the convolutional layer applies its activation function in Eq. (3):

$$Net=\sigma \left(Z_i^l\right)$$ 3

where $\sigma \left(.\right)$ is called nonlinearity, function applied to achieve nonlinearity in DTL, which contains many types such as tanh, sigmoid, rectified linear units (ReLU). In our method, we utilize ReLU in Eq. (4) as the activation function for faster training process:

$$\sigma \left(u\right)=\text{max}\left(0,u\right)$$ 4

Fig. 4.

Illustration of the convolutional and pooling layer which produce feature maps

The loss function is the criterion for the training process. Our loss function in Eq. (5) is defined as the sum of the binary-entropy loss and the box regression loss:

$$\text{Loss}\left(s,t\right)={\text{Loss}}_{\text{cls}}\left(s_{c^{\ast }}\right)+\lambda \left[p^{\ast }>0\right]L_{\text{reg}}\left(v,v^{\ast }\right)$$ 5

where $s_{c^{\ast }}$ denotes the predicted score class $c^{\ast }$ while $v$ and $v^{\ast }$ denote $\left[v_x,v_y,v_w,v_h\right]$ of bounding boxes. $\lambda \left[p^{\ast }>0\right]$ indicates that we only consider the boxes of non-background (the box is background if $p^{\ast }=0$). This loss function contains two parts for bounding box regression loss ${\text{Loss}}_{\text{reg}}\text{and}$ classification loss ${\text{Loss}}_{\text{cls}}$ and in Eqs. (6–8):

$${\text{Loss}}_{\text{cls}}\left(s_{c^{\ast }}\right)=-log\left(s_{c^{\ast }}\right)$$ 6

and

$${\text{Loss}}_{\text{reg}}\left(v,v^{\ast }\right)=\sum _{i\in \left(x,y,w,h\right)}{R}_{L1}\left(v_i-v_i^{\ast }\right)$$ 7

where

$$R_{L1}\left(u\right)=\left\{\begin{array}{cc}0.5u^2,\hfill & \text{if}\left|u\right|<0\\ \left|u\right|-0.5,\hfill & \text{otherwise}\end{array}\right)$$ 8

Data augmentation

The main idea behind this research is to perform a transfer learning with augmented COVID-19 CT images. To increase the performance of the proposed transfer learning models, training data amount and validate data is a very important factor. The most popular classical data augmentation method is to perform a combination of affine image transformations [40]. Different methods for classical data augmentation such as rotation, shifting, flipping, zooming, transformation, add noise were selected to be applied to the original dataset. Figure 5 shows examples of COVID-19 CT augmented images. The achieved performance measurement will be discussed in the experimental results section.

Fig. 5.

Perform augmentation methods to increase limited COVID-19 CT scan images

Conditional generative adversarial network

CGANs consist of two different types of networks (generator network and discriminator network) with the conditional label as shown in Fig. 6. A CGAN is a type of GAN that takes labels in the training process. The generator network in this paper consists of 4 transposed convolutional layers, 3 ReLU layers, 3 batch normalization layers, and tanh layer at the end of the model, while the discriminator network consists of 4 convolutional layers, 3 leaky ReLU, and 2 batch normalization layers. All the convolutional and transposed convolutional layers use the same filter size of 5 × 5 pixels with 20, 10, 5 filters for each layer for the generated network but 5, 10, 20, 40 for each layer in the discriminator network. Figure 7 presents the structure and the sequence of layers of the CGAN network proposed in this research. We trained our CGAN model as shown in the right Fig. 8, and on the left, some generated CT images.

Fig. 6.

Conditional generative adversarial network model

Fig. 7.

The structure of the proposed CGAN network

Fig. 8.

CGAN training and samples of the generated image

The CGAN network helped in overcoming the overfitting problem caused by the limited number of CT images in the COVID-19 dataset. Figure 9 presents samples of the output of the CGAN network for the COVID-19 class. Moreover, it increased the number of the dataset images to be 10 times more than the original one. The number of images in the augmented dataset reached 4425 images in the train set and 418 in the validation set after using the CGAN network for 2 classes. This will help in achieving better testing accuracy and performance matrices. The achieved performance measurement will be discussed in the experimental results section.

Fig. 9.

Samples of COVID-19 CT images generated by the CGAN model

Experimental results

The proposed model is trained on a high-end graphics processing unit (GPU). The GPU used (NVIDIA RTX 2070) contains 2304 CUDA core and comes with the CUDA deep neural network library (CuDNN) for GPU learning, the deep learning package TensorFlow machine learning and MATLAB as back end library. The proposed model has been tested under four different scenarios, the first scenario is to test the DTL models with the original COVID-19 CT dataset, the second scenario with data augmentation, the third one with CGAN, and the last one combines all three scenarios. All the test experiment scenarios included the two classes (COVID/NonCOVID). Every scenario consists of the validation phase and the testing phase as shown in Table 2.

Table 2.

Configuration of DTL models

Model	Layers	Batch size	Momentum	Epoch	Learning rate	Optimizer
AlexNet	8	32	0.9	50	0.001	Adam
VGGNet16	16	32	0.9	50	0.001	Adam
VGGNet19	19	32	0.9	50	0.001	Adam
GoogleNet	22	32	0.9	50	0.001	Adam
ResNet50	50	32	0.9	50	0.001	Adam

Table 2 shows the five DTL models with initial learning rate ($\mu$) equal to 0.001 and the number of epochs equal to 50. Also, the mini-batch size is set to 32 and early stopping to be 5 epochs if the accuracy didn’t improve. In terms of optimizer technique, Adam [41] is chosen as our optimizer technique, which updates weights parameters. This optimizer technique is a combination of root mean square propagation (RMSprop) and stochastic gradient descent (SGD) with momentum. To avoid deep learning network overfitting problems, we utilize the dropout method [42] as well as the early stopping technique [43] to select the most appropriate training iteration.

Verification and testing accuracy measurement

Testing accuracy is one of the estimations which demonstrates the performance measurement of any DTL models. The confusion matrix also is one of the performance measurements which give more insights into the achieved testing accuracy. DTL confusion matrices for two classes with different scenarios are illustrated in Fig. 10. The first DTL model investigated is AlexNet along with four scenarios as shown in Fig. 10. Table 3 shows that the highest testing accuracy using AlexNet is 76.4% when the COVID-19 CT dataset is augmented with data augmentation along with CGAN. The second DTL model is investigated with VGGNet16. Table 3 shows that the highest testing accuracy using VGGNet16 is 78.9% when the COVID-19 CT dataset is augmented with classical data augmentation along with CGAN. The third DTL model is investigated with VGGNet19. Table 3 shows that the highest testing accuracy using VGGNet19 is 76.9% when the COVID-19 CT dataset is not augmented. The fourth DTL model is investigated with GoogleNet. Table 3 shows that the highest testing accuracy using GoogleNet is 77.4% when the COVID-19 CT dataset is augmented with the classical data augmentation along with CGAN.

Fig. 10.

DTL Confusion matrices for two classes with different scenarios

Table 3.

DTL testing accuracy for the different four scenarios

Dataset	AlexNet (%)	VGGNet16 (%)	VGGNet19 (%)	GoogleNet (%)	ResNet50 (%)
COVID-19	67.34	72.36	76.88	75.38	76.38
COVID-19 with augmentation	75.38	77.89	69.35	76.88	82.91
COVID-19 with CGAN	68.34	70.85	73.37	75.88	77.39
COVID-19 with aug and CGAN	76.38	78.89	73.87	77.39	81.41

The final DTL model will be investigated with ResNet50. The highest testing accuracy is 82.9% when the COVID-19 CT dataset is augmented with classical data augmentation as shown in Fig. 11. Table 3 summarizes the testing accuracy for the different deep transfer learning models for 2 classes with the four scenarios. Table 3 illustrates testing accuracy for the different four scenarios, the Resnet50 achieved the highest accuracy with 82.9%, this is due to the large number of parameters in the Resnet50 architecture which contains millions of parameters which are not larger than VGGNet and GoogleNet, but the VGGNet and GoogleNet only include 16, and 22 layers while the Resnet50 includes 50 layers.

Fig. 11.

Confusion matrix of highest accuracy for ResNet50 in COVID-19 with classical data augmentation

Performance evaluation and discussion

To quantitatively evaluate the performance of the proposed models, other performance matrices are investigated. The most common performance measures in the field of deep learning are sensitivity, specificity, precision, accuracy and F1 score [44] and they are presented from Eq. (9) to Eq. (13).

$$\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\left(\text{TP}+\text{FP}\right)+\left(\text{TN}+\text{FN}\right)}$$ 9

where TP (true positives) is the count of correctly labeled instances of the class under observation, FP (false positives) is the count of miss-classified labeled of rest of the classes, TN (true negatives) is the count of correctly labeled instances of rest of the classes, and FN (false negatives) is the count of miss-classified labeled of the class under observation.

$$\text{Sensitivity}=\frac{\text{TP}}{\left(\text{TP}+\text{FN}\right)}$$ 10

$$\text{Specificity}=\frac{\text{TP}}{\left(\text{FP}+\text{TN}\right)}$$ 11

$$\text{Precision}=\frac{\text{TP}}{\left(\text{TP}+\text{FP}\right)}$$ 12

$$\text{F}1\text{Score}=\frac{2\text{TP}}{\left(2\text{TP}+\text{FP}+\text{FN}\right)}$$ 13

Figure 12 presents the performance metrics for different scenarios with DTL models for the COVID-19 CT dataset. The highest sensitivity of 88.3% (Table 4) is achieved by scenario-1 and scenario-2 (COVID-19 only and with augmentation) based on AlexNet and VGGNet19 that refers to the test’s ability to correctly classify COVID-19 CT patients who do have the condition. In the example of a CT scan medical test used to classify and detect a COVID-19 disease, the detection rate (sensitivity) of the test is the proportion of people who test positive for the COVID-19 malady among those who have the COVID-19 malady. A negative result in a test with a high detection rate is useful for getting rid of the COVID-19 CT malady.

Fig. 12.

Performance measurements for COVID-19 CT in four scenarios

Table 4.

Testing Sensitivity for the different 4 scenarios

Dataset	AlexNet (%)	VGGNet16 (%)	VGGNet19 (%)	GoogleNet (%)	ResNet50 (%)
COVID-19	88.30	61.70	70.21	69.15	65.96
COVID-19 with augmentation	73.40	75.53	88.30	74.47	77.66
COVID-19 with CGAN	78.72	61.70	53.19	75.53	61.70
COVID-19 with aug and CGAN	63.83	62.77	71.28	71.28	80.85

A test with high specificity determines that a person does not have COVID-19 as shown in Table 5. Sensitivity and specificity can be summarized by a single quantity called the balanced accuracy as shown in Table 6, which is defined as the mean of both measures in Eq. (14):

$$\text{Balanced,accuracy}=\frac{\text{Sensitivity}+\text{Specificity}}{2}$$ 14

Table 5.

Testing Specificity for the different 4 scenarios

Dataset	AlexNet (%)	VGGNet16 (%)	VGGNet19 (%)	GoogleNet (%)	ResNet50 (%)
COVID-19	48.57	81.90	82.86	80.95	85.71
COVID-19 with augmentation	77.14	80.00	52.38	79.05	87.62
COVID-19 with CGAN	59.05	79.05	91.43	76.19	91.43
COVID-19 with aug and CGAN	87.62	93.33	76.19	82.86	81.90

Bold value indicates high Specificity

Table 6.

Testing Balanced accuracy for the different 4 scenarios

Dataset	AlexNet (%)	VGGNet16 (%)	VGGNet19 (%)	GoogleNet (%)	ResNet50 (%)
COVID-19	68.44	71.80	76.54	75.05	75.84
COVID-19 with augmentation	75.27	77.77	70.34	76.76	82.64
COVID-19 with CGAN	68.89	70.38	72.31	75.86	76.57
COVID-19 with aug and CGAN	75.73	78.05	73.74	77.07	81.38

Bold values indicate 3 high Balanced accuracy

The balanced accuracy is in the range [0,1] where a value of 0 and 1 indicates the worst and the best classifier, respectively.

As shown in Table 6, ResNet50 is the best classifier to detect the COVID-19 in CT dataset with classical data augmentation along with CGAN. The classical data augmentation along with CGAN improves the performance of classification in all deep transfer models (AlexNet, VGGNet16, GoogleNet, ResNet50). The other bottleneck is the limited size of the COVID-19 CT database. Predictably the performance of deep transfer models can be further improved if more data are collected in the future. Although we have achieved promising accuracy rates, the proposed model in this study needs to be tested on larger-scale datasets that include different COVID-19 CT images to increase the testing accuracy and extend it in other medical applications. As future work, we plan to classify COVID-19 using a neutrosophic approach [45] and deep learning.

Conclusion and future works

In 2019, the world was infected by the 2019 novel Coronavirus that killed thousands and infected over a million human within few months of the 2019 novel Coronavirus epidemic. In this paper, classical data augmentations along with CGAN with deep transfer learning for COVID-19 detection in limited chest CT scan images are presented. The number of COVID-19 CT images of the collected dataset was 742 images for two types of labels. The classical data augmentation and CGAN helped to increase the CT dataset and overcoming the overfitting problem. Moreover, five deep transfer learning models (AlexNet, VGGNet16, VGGNet19, GoogleNet, and ResNet50) were selected in this paper for investigation. Using a combination of classical data augmentation and CGAN with deep transfer learning improved testing accuracy, and performance measurements such as sensitivity, specificity, precision, accuracy, and F1 score. The results show that ResNet50 is the best deep learning model to detect the COVID-19 from chest CT dataset. The major drawback is not trying most of deep learning models to get highest performance measurement. As future work, we plan to approach the COVID-19 study from machine learning and deep learning.

Funding

This research received no external funding.

Compliance with ethical standards

Conflict of interest

The author declares no conflict of interest.