Automatic diagnosis of COVID-19 from CT images using CycleGAN and transfer learning

Abstract

The outbreak of the corona virus disease (COVID-19) has changed the lives of most people on Earth. Given the high prevalence of this disease, its correct diagnosis in order to quarantine patients is of the utmost importance in the steps of fighting this pandemic. Among the various modalities used for diagnosis, medical imaging, especially computed tomography (CT) imaging, has been the focus of many previous studies due to its accuracy and availability. In addition, automation of diagnostic methods can be of great help to physicians. In this paper, a method based on pre-trained deep neural networks is presented, which, by taking advantage of a cyclic generative adversarial net (CycleGAN) model for data augmentation, has reached state-of-the-art performance for the task at hand, i.e., 99.60% accuracy. Also, in order to evaluate the method, a dataset containing 3163 images from 189 patients has been collected and labeled by physicians. Unlike prior datasets, normal data have been collected from people suspected of having COVID-19 disease and not from data from other diseases, and this database is made available publicly. Moreover, the method’s reliability is further evaluated by calibration metrics, and its decision is interpreted by Grad-CAM also to find suspicious regions as another output of the method and make its decisions trustworthy and explainable.

1. Introduction

The COVID-19 disease was initially spotted in December of 2019 in Wuhan, China, and was detected worldwide shortly after that [1]. In January 2020, the World Health Organization (WHO) stated its outbreak as a public health emergency and global concern, and later, a pandemic in March of 2020 [2]. SARS-CoV-2 is a novel strain of coronavirus that causes COVID-19; also, it has not been seen previously in humans [3]. Coronaviruses are common among animals, and some can infect humans [3], [4]. Bats are the natural hosts of these viruses, and several other species of animals have also been identified as sources [5], [6], [7]. For example, MERS-CoV-3 is transmitted from camels to humans, while SARS-CoV-14 is transmitted from intermediate hosts such as civet cats that were involved in the development of SARS-CoV-1 [8], [9]. The new coronavirus is genetically closely related to the SARS-CoV-1 virus [10].

The SARS-CoV-2 virus is transmitted mainly through respiratory droplets and aerosols from an infected person while sneezing, coughing, talking, or breathing in the presence of others [11], [12]. The virus can stay active at varying surfaces from a few hours to several days, and prior research has estimated the incubation period of this disease to be within 1 and 14 days [13]. However, the amount of live virus decreases over time and may not always be present in sufficient quantities to cause infection [14], [15].

The most frequent symptoms of COVID-19 include fever, dry cough, and fatigue. Pain, diarrhea, headache, sore throat, conjunctivitis, loss of taste or smell are other variable symptoms of the virus [16], [17]. Nevertheless, the most severe symptoms seen in COVID-19 patients include difficulty for breathing or shortness of breath, chest pain, and loss of movement or ability to speak [16], [17], [18].

Early diagnosis of this disease in the preliminary stages is vital. So far, various screening methods have been introduced for the diagnosis of COVID-19. At present, nucleic acid–based molecular diagnosis, by reverse transcriptase-polymerase chain reaction (RT-PCR) test, is considered the gold standard for early detection of COVID-19 [19], [20]. According to a WHO report, all diagnoses of COVID-19 must be verified by RT-PCR [21]. However, performing the RT-PCR test needs specialized equipment and equipped laboratories that are not available in most countries and takes at least 24 h to determine the test outcome. Also, the test result may not be accurate and may require to redo RT-PCR or other tests. Therefore, X-ray and CT-Scan imaging can be used as a primary diagnostic method for screening people suspected of having COVID-19 [22], [23].

X-ray imaging is one of the medical imaging techniques used to diagnose COVID-19 [24], [25]. X-ray imaging benefits include low cost and low risks of radiation that are dangerous to human health [26], [27]. In this imaging technique, the detection of COVID-19 is a relatively complicated task. An X-ray physician may also misdiagnose diseases such as pulmonary tuberculosis [28], [29].

CT-Scan imaging is used to reduce COVID-19 detection error. CT-scans have very high contrast and resolution, and are very successful in diagnosing lung diseases such as COVID-19 [30], [31], [32]. CT-Scan can also be used as a clinical feature of COVID-19 disease patients. CT scans of subjects with COVID-19 had shown marked destruction of the pulmonary parenchyma 6, such as interstitial inflammation and extensive consolidation [33]. During CT-Scan imaging of patients, multiple slices are recorded to diagnose COVID-19. This high number of CT-Scan images requires a high accuracy from specialists for accurate diagnosis of COVID-19. Factors such as eye exhaustion or a massive number of patients to interpret CT-Scan may lead to misdiagnosis of COVID-19 by specialists [34].

Due to the stated challenges, the use of artificial intelligence (AI) methods for accurate diagnosis of COVID-19 on CT-Scan or X-ray imaging modalities is of utmost importance. The design of computer aided diagnosis systems (CADS) based on AI using CT-Scan or X-ray images for precise diagnosis of COVID-19 has been highly regarded by researchers [35], [36], [37]. Deep learning (DL) is one of the fields of AI, and many research papers have been published on their application for diagnosing COVID-19 [38], [39].

In this paper, a new method of diagnosing COVID-19 from CT-Scan images using DL is presented. First, CT-Scan images of people with COVID-19 and normal people were recorded in Gonabad Hospital (Iran). Next, three expert radiologists have labeled the patients’ images. They have also selected informative slices from each scan. Then, after preprocessing data with a Gaussian filter, various deep learning networks were trained in order to separate COVID-19 from healthy patients. In this step, a CycleGAN [40], [41] architecture was first used for data augmentation of CT-Scan data; after that, a number of pre-trained deep networks [42] such as DenseNet [43], ResNet [44], ResNest [45], and ViT [46] have been used to classify CT-Scan images. Fig. 1 shows the block diagram of method. The results show that the proposed method of this study has promising and reliable results in detecting COVID-19 from CT-scan images of the lung.

Fig. 1.

Overall diagram of proposed method.

The rest of the paper is organized as follows. In the next section, we present a review of previous research on the diagnosis of COVID-19 from CT-Scan images using DL techniques. In Section 3, the proposed method of this research is presented. In Section 4, the evaluation process and the results of the proposed method are presented. Section 5 includes the discussion of paper and finally, the paper ends with the conclusion and future directions.

2. Related works

Prior research papers on the diagnosis of the COVID-19 disease using machine learning can be divided according to the algorithms used or the underlying modalities. Fig. 2 shows various types of methods that can be used for diagnosis of COVID-19. As can be seen in this figure, the methods based on medical imaging can be divided into two groups: CT scan and X-ray. The focus of this article is on CT scan modality. Also, machine learning algorithms can be divided into two categories: DL [47], [48], [49], [50] and conventional machine learning methods [51], [52], [53]. Due to the large number of machine learning papers for diagnosing COVID-19 disease from CT modality, we have only reviewed papers that have used deep learning methods for this imaging modality. Table 1 provides an overview of these papers, the datasets used by them, the components of methods, and finally, their performance. Moreover, some of those works are reviewed in the following:

Fig. 2.

Various criteria used for COVID-19 detection and their categories.

In [54], researchers introduced a method for detecting COVID-19 from CT images using pre-trained models. First, patches extraction was performed on CT images in the pre-processing step. Next, they used GoogleNet, VGG-16, and ResNet-50 models in the feature extraction step and then fused the feature together. Next, feature ranking and finally classification was done using SVM. In this work, they were able to achieve adequate results.

In another research, Yu et al. proposed a method for detecting COVID-19 from CT images using ResGNet-C architecture [55]. In this article, they used a clinical dataset to conduct experiments. The pre-processing of CT images for the diagnosis of COVID-19 is based on the visual inception method. Finally, the proposed ResGNet-C model has been selected for feature extraction and classification. In this work, in order to evaluate the results, K Fold with K=5 has also been used.

A new detection method for COVID-19 based on U-Net and dual-branch combination network architectures was introduced by Gao et al. [56]. In this work, all the experiments have been done on a clinical CT dataset containing the infected patients with COVID-19 and HC. They used a DA method to generate synthetic CT images in the pre-processing step. Next, U-Net and dual-branch combination network architectures were used to detect COVID-19. They also used attention maps as a post-processing step in their proposed method. The research results show the high efficiency of their proposed method in diagnosing COVID-19.

Next, Ouyang et al. presented a proposed 3D ResNet34 architecture for the detection of COVID-19 [57]. In this article, the tests are done on a clinical dataset of CT images. In the pre-processing step, segmentation and generating lung masks were performed on CT images. Next, the 3D architecture of ResNet34 along with attention layers was used to detect COVID-19. Also, the gradient-weighted class activation mapping (Grad-CAM) method was tested as a pre-processing step to detect suspected areas of COVID-19 in CT images accurately.

In Loey et al.’s research [58], a method of detecting COVID-19 based on transfer learning was proposed along with the traditional DA technique and conditional GAN. In this work, pre-processing of CT images includes resizing and conventional DA steps. Next, the conditional GAN model was used along with ResNet50 to detect COVID-19. The results of the proposed method show that they have achieved outstanding results in diagnosing COVID-19 from CT images.

In another research, a method for detecting COVID-19 from CT images using GAN architecture was introduced by Goel et al. [59]. In this work, experiments have been performed on the SARS-CoV-2 CT-Scan dataset. In this dataset, all CT images are pre-processed. Therefore, the researchers applied the CT images of the dataset to the input of a proposed DL architecture. The proposed DL architecture of this paper includes a GAN model along with the Whale optimization method. In the classification section, researchers also used K Fold with K=10 to evaluate the results.

Next, In [60], researchers first used different datasets to demonstrate the effectiveness of the proposed method for detecting COVID-19. This paper’s feature extraction from CT images is based on ResNet-50 architecture. Next, a 3D CNN architecture was used to classify features. In addition, K Fold with K=10 has been used to evaluate the proposed model in this work. The research results show the proper performance of their proposed method in diagnosing COVID-19.

Moreover, In another study, Amyar et al. used a new multi-task learning architecture to detect COVID-19 [61]. In the first step, different datasets of CT images were selected for experiments. Next, resizing and intensity normalization steps were used to pre-process CT data. Finally, a new DL architecture based on multi-task learning was proposed for CT data segmentation and classification. The research results show that the method proposed in this article has reached a high accuracy in diagnosing COVID-19 from CT images.

Lastly, In [62], the authors presented a proposed DL architecture for the detection of COVID-19 called Covid CT-net. In this research, researchers used the Kaggle dataset to conduct experiments. The pre-processing step of this work includes resizing, normalization, and DA steps. Next, the proposed architecture of the Covid CT-net for accurate diagnosis of COVID-19 from CT images is presented. Also, they used a post-processing step in this article and displayed the suspected areas of COVID-19 in the CT images as a heat map. In this work, they were able to achieve promising results.

3. Materials and methods

This section of the paper is devoted to discussing the applied method and its components. In this paper, we have firstly collected a new CT scan dataset from COVID-19 patients; then, from each scan, the informative slices were selected by physicians. After that, several convolutional neural networks pre-trained on the ImageNet dataset [63] were fine-tuned to the task at hand. Here, we trained a Resnet-50 architecture [44], an EfficientNet B3 architecture [64], a Densenet-121 architecture [43], a ResNest-50 architecture [45], and a ViT architecture [46]. These networks were selected to include both attention-based models (ViT and ResNest) and conventional CNN networks (Resnet, Densenet, and EfficientNet). Also, in each category, we have selected networks that have reached more promising results than their predecessors. Several data augmentation techniques alongside a CycleGAN model were applied to improve the performance of each network further.

Here, the details of each step are presented; firstly, an explanation is given on the applied dataset. Specifications of each applied deep neural network (DNN) are discussed afterward. Finally, CycleGAN is explained in the last part alongside the overall proposed method.

Table 1.

Review of related works.

Ref	Dataset	Modality	Number of cases	Pre-Processing	DNN	Post-Processing	Performance criteria
[54]	Clinical	CT	3000 COVID-19 Images, 3000 Non-COVID-19 Images	Patches Extraction	VGG-16, GoogleNet, ResNet-50	Feature Fusion, Ranking Technique, SVM	Acc $=$ 98.27 Sen $=$ 98.93 Spec $=$ 97.60 Prec $=$ 97.63
[65]	Datasets from [66] & [67]	CT	460 COVID-19 Images, 397 Healthy Control (HC) Images	Data Augmentation (DA)	CNN Based on SqueezeNet	Class Activation Mapping (CAM)	Acc $=$ 85.03 Sen $=$ 87.55 Spec $=$ 81.95 Prec $=$ 85.01
[68]	Various Datasets	CT	2373 COVID-19 Images, 2890 Pneumonia Images, 3193 Tuberculosis Images, 3038 Healthy Images	–	Ensemble DCCNs	–	Acc $=$ 98.83 Sen $=$ 98.83 Spec $=$ 98.82 F1-Score $=$ 98.30
[69]	Clinical	CT	98 COVID-19 Patients, 103 Non-COVID-19 Patients	Visual Inspection	BigBiGAN	–	Sen $=$ 80 Spec $=$ 75
[55]	Clinical	CT	148 Images from 66 COVID-19 Patients, 148 Images from 66 HC Subjects	Visual Inspection	ResGNet-C	–	Acc $=$ 96.62 Sen $=$ 97.33 Spec $=$ 95.91 Prec $=$ 96.21
[70]	COVID-CT Dataset	CT	349 COVID-19 Images, 397 Non-COVID-19 Images	Scaling Process, DA	Multiple Kernels-ELM -based DNN	–	Acc $=$ 98.36 Sen $=$ 98.28 Spec $=$ 98.44 Prec $=$ 98.22
[56]	Clinical	CT	210,395 Images From 704 COVID-19 Patients and 498 Non-COVID-19 Subjects	DA	U-net Dual-Branch Combination Network	Attention Maps	Acc $=$ 92.87 Sen $=$ 92.86 Spec $=$ 92.91
[67]	Various Dataset	CT	2933 COVID-19 Images	Deleting Outliers, Normalization, Resizing	Ensemble DNN	–	Acc $=$ 99.054 Sen $=$ 99.05 Spec $=$ 99.6 F1-Score $=$ 98.59
[71]	Clinical	CT	320 COVID-19 Images, 320 Healthy Control Images	Histogram Stretching, Margin Crop, Resizing, Down Sampling	FGCNet	Gradient- Weighted CAM (Grad-CAM)	Acc $=$ 97.14 Sen $=$ 97.71 Spec $=$ 96.56 Prec $=$ 96.61
[72]	Clinical	CT	180 Viral Pneumonia, 94 COVID-19 Cases	ROIs Extraction	Modified Inception	–	Acc $=$ 89.5 Sen $=$ 88 Spec $=$ 87 F1-Score $=$ 77
[57]	Clinical	CT	3389 COVID-19 Images, 1593 Non-COVID-19 Images	Segmentation, Generating Lung Masks	3D ResNet34 with Online Attention	Grad-CAM	Acc $=$ 87.5 Sen $=$ 86.9 Spec $=$ 90.1 F1-Score $=$ 82.0
[73]	COVIDx-CT Dataset	CT	104,009 Images From 1489 Patient Cases	Automatic Cropping Algorithm, DA	COVIDNet-CT	–	Acc $=$ 99.1 Sen $=$ 97.3 PPV $=$ 99.7
[74]	Various Datasets	CT	349 COVID-19 Images, 397 Non-COVID-19 Images	Resizing, Normalization, Wavelet-Based DA	ResNet18	Localization of Abnormality	Acc $=$ 99.4 Sen $=$ 100 Spec $=$ 98.6
[58]	COVID-CT	CT	345 COVID-19 Images, 397 Non-COVID-19 Images	Resizing, DA	Conditional GAN ResNet50	–	Acc $=$ 82.91 Sen $=$ 77.66 Spec $=$ 87.62
[75]	Clinical	CT	151 COVID-19 Patient, 498 Non-COVID-19 Patient	Resizing, Padding, DA	3D-CNN	Interpretation by Two Radiologists	AUC $=$ 70
[59]	SARS-CoV-2 CT-Scan Dataset	CT	1252 CT COVID-19 Images, 1230 CT non-COVID-19 Images	–	GAN with Whale Optimization Algorithm	–	Acc $=$ 99.22 Sen $=$ 99.78 Spec $=$ 97.78 F1-score $=$ 98.79
[66]	Various Datasets	CT	1684 COVID-19 Patient, 1055 Pneumonia, 914 Normal Patients	Resizing	Inception V1	Interpretation by 6 Radiologists, t-SNE Method	Acc $=$ 95.78 AUC $=$ 99.4
[76]	Clinical	CT	2267 COVID-19 CT Images, 1235 HC CT Images	Compressing, Normalization, Cropping, Resizing	ResNet50	–	Acc $=$ 93 Sen $=$ 93 Spec $=$ 92 F1-Score $=$ 92
[77]	Clinical	CT	108 COVID-19 Patients, 86 Non-COVID-19 Patients	Visual Inspection, Grey-Scaling, Resizing	Various Networks	–	Acc $=$ 99.51 Sen $=$ 100 Spec $=$ 99.02
[60]	Various Datasets	CT	413 COVID-19 Images, 439 Non-COVID-19 Images	Feature Extraction with ResNet-50	3D-CNN	–	Acc $=$ 93.01 Sen $=$ 91.45 Spec $=$ 94.77 Prec $=$ 94.77
[78]	Clinical	CT	150 3D COVID-19 Chest CT, CAP and NP Patients (450 Patient Scans)	Sliding Window, DA	Multi-View U-Net 3D-CNN	Weakly Supervised Lesions Localization, CAM	Acc $=$ 90.6 Sen $=$ 83.3 Spec $=$ 95.6 Prec $=$ 74.1
[61]	Various Datasets	CT	449 COVID-19 Patients, 425 Normal, 98 Lung Cancer, 397 Different Kinds of Pathology	Resizing, Intensity Normalization	Autoencoder Based DNN	–	Dice $=$ 88 Acc $=$ 94.67 Sen $=$ 96 Spec $=$ 92
[79]	COVID-19 CT from [66]	CT	746 Images	–	GAN	–	Acc $=$ 84.9 Sen $=$ 85.33 Prec $=$ 85.33
[80]	COVID-19 CT Datasets, Cohen	CT	345 COVID-19 CT Images, 375 Non-COVID-19 CT Image	2D Redundant Discrete WT (RDWT) Method, Resizing	ResNet50	Grad-CAM, Occlusion Sensitivity Technique	Acc $=$ 92.2 Sen $=$ 90.4 Spec $=$ 93.3 F1-Score $=$ 91.5
[81]	SARS-CoV-2 CT Scan Dataset	CT	1262 COVID-19 Images, 1230 HC Images	–	Convolutional Support Vector Machine (CSVM)	–	Acc $=$ 94.03 Sen $=$ 96.09 Spec $=$ 92.01 Pre $=$ 92.19
[82]	Chest CT and X-ray	X-ray, CT	5857 Chest X-rays, 767 Chest CTs	–	Various Networks	Heat Map	Acc $=$ 75 (CT)
[83]	medseg DlinRadiology	CT	10 Axial Volumetric CTS (Each Containing 100 Slices of COVID-19 Images)	Resizing	VGG16, Resnet-50 U-net	–	Acc $=$ 99.4 Spec $=$ 99.5 Sen $=$ 80.83 Dice $=$ 72.4 IOU $=$ 61.59
[84]	BasrahDataset	CT	50 Cases, 1425 Images	Gray-Scaling, Resizing	VGG 16	–	Acc $=$ 99 F1-Score $=$ 99
[62]	Kaggle	CT	1252 COVID-19 CT Images, 1240 non-COVID-19 CT Images	Resizing, Normalization, DA	Covid CT-net	heat map	Acc $=$ 95.78 Sen $=$ 96 Spec $=$ 95.56
[85]	COVID-CT	CT	708 CTs, 312 with COVID-19, 396 Non-COVID-19	Normalization	LeNet-5	–	Acc $=$ 95.07 Sen $=$ 95.09 Prec $=$ 94.99

3.1. Dataset

In this paper, a new CT scans dataset of COVID-19 patients was collected from Gonabad Hospital in Iran; all data were recorded by radiologists between June 2020 and December 2020. The number of subjects with COVID-19 is 90, and 99 of the subjects are normal. In this study, three volumetric recordings were gathered from each patient in a 4-day duration (each recording two days apart from the last one); moreover, every recording contains at least 20 slices. It is noteworthy to mention that the normal subjects are patients with suspicious symptoms and not merely a control group; this makes this dataset unique compared to its prior ones, as they usually have used scans of other diseases for the control group. Patients with COVID-19 or normal subjects range in age from 2 to 88 years; 69 of which are female and 120 are male (both COVID-19 and normal classes). A total of 1766 slices of these scans were finally selected by specialist physicians from the abnormal class and 1397 from the normal class; the labeling of each CT image was performed by three experienced radiologists along with two infectious disease physicians. In addition, RT-PCR was taken from each subject to confirm labelings. All ethical approvals have been obtained from the hospital to use CT scans of COVID-19 patients and normal individuals for research purposes. Fig. 3 illustrates a few CT scans of healthy individuals and patients with COVID-19.

Fig. 3.

Examples of dataset, first two rows contain images from healthy subjects, whereas the last two rows contain images from COVID-19 patients.

3.2. Deep neural networks

3.2.1. Resnet

ResNet [44] architecture was introduced in 2015 with a depth of 152 layers; it is known as the deepest architecture up to that year and still is considered as one of the deepest. There are various versions of the architecture with different depths that are used depending on the need. This network’s main idea was to use a block called residual block, which tried to solve the problem of vanishing gradients, allowing the network to go deeper without reducing performance. Proving its capabilities by winning the Image Net Challenge in 2015; the ideas of this network have been applied in many others ever since. In this paper, a version of this network with a depth of 50 has been used, which is a wise choice given the considerably smaller amount of data compared to the ImageNet database.

3.2.2. EfficientNet

Three different criteria must be tested to design a convolutional neural network: the depth, width, and resolution of the input images. Choosing the proper values of the three criteria in such a way that they form a suitable network together is a challenging task. Increasing the depth can lead to finding complex patterns, but it can also cause problems such as vanishing gradients. More width can increase the quality of the features learned, but accuracy for such network tends to quickly saturate. Also, high image quality can have a detrimental effect on accuracy. The network was introduced in [64] with a study on how to scale the network in all three criteria properly. Using a step-by-step scheme, the network first finds the best structure for a small dataset and then scales that structure according to the activity. The network has been used for many tasks, including diagnosing autism [86] and schizophrenia [87].

3.2.3. Densenet

Introduced by Huang et al. [43], DenseNet, densely connected convolutional networks, has improved the baseline performance on benchmark computer vision task and shown its efficiency. Utilizing residuals in a better approach has allowed this network to exploit fewer parameters and go deeper. Also, by feature reuse, the number of parameters is reduced dramatically. Its building blocks are dense blocks and transition layers. Compared to ResNet, DenseNet uses concatenation in residuals rather than summing them up. To make this possible, each feature vector of each layer is chosen to have the same size for each dense block; also, training these networks has been shown to be easier than prior ones [43]. This is arguably due to the implicit deep supervision where the gradient is flowing back more quickly. The capability to have thin layers is another remarkable difference in DenseNet compared to other state-of-the-art techniques. The parameter K, the growth rate, determines the number of features for each layer’s dense block. These feature vectors are then concatenated with the preceding ones and given as input to the subsequent layer. Eliminating optimization difficulties for scaling up to hundreds of layers is another DenseNet superiority.

3.2.4. ViT

Arguably, the main problem with convolutional neural networks is their failure in encoding relative spatial information. In order to overcome this issue, researchers in [46] have adopted the self-attention mechanism from natural language processing (NLP) models. Basically, attention can be defined as trainable weights that model each part of an input sentence’s importance. Changing networks from NLP to computer vision, pixels are picked as parts of the image to train the attention model on them. Nevertheless, pixels are very small parts of an image; thus, one can pick a bigger segment of an image as one of its parts, i.e., a 16 by 16 block of images. ViT uses a similar idea; by dividing the image into smaller patches to train the attention model on them. Also, ViT-Large has 24 layers with a hidden size of 1,024 and 16 attention heads. Examination shows not only superior results but also significantly reduced training time and also less demand for hardware resources [46].

3.2.5. ResNest

Developed by researchers from Amazon and UC Davis, ResNest [45] is also another attention-based neural network that has also adopted the ideas behind ResNet structure. In its first appearance, this network has shown significant performance improvement without a large increase in the number of parameters, surpassing prior adaptations of ResNet such as ResNeXt and SEnet. In their paper, they have proposed a modular Split-Attention block that can distribute attention to several feature-map groups. The split-attention block is made of the feature-map group and split-attention operations; then, by stacking those split-attention blocks similar to ResNet, researchers were able to produce this new variant. The novelties of their paper are not merely introducing a new structure, but they also introduced a number of training strategies.

3.3. Data augmentation and training process

Generative adversarial networks were first introduced in 2014 [88] and found their way into various fields shortly after [47]. They have also been used as a method for data augmentation, and network pretraining [89] previously as well. A particular type of these networks is CycleGAN [40], a network created mainly for unpaired image-to-image translation. In this particular form of image-to-image translation, there is no need for a dataset containing paired images, which is itself a challenging task. The CycleGAN comprises of training of two generator discriminators simultaneously. One generator uses the first group of images as input and creates data for the second group, and the other generator does the opposite. Discriminator models are then utilized to distinguish the generated data from real ones and feed the gradients to generators subsequently.

The CycleGAN used in this paper has a similar structure to the one presented in the main paper [40]. Compared to other GAN paradigms, CycleGAN uses image-to-image translation, which simplifies the training process, especially where training data is limited, which also helps to create data of the desired class easily. However, using other GAN paradigms, such as conditional GAN [90], one can also create data of a specific class, yet training those methods is more complicated. A diagram of the CycleGAN is presented in Fig. 4, and also a few samples of generated data are illustrated in Fig. 5.

Fig. 4.

Overall diagram of applied CycleGAN.

Fig. 5.

Examples of generated data, the first column shows normal data from the main dataset. The second column shows the generated abnormal data from those images. The third column shows abnormal data from the main dataset. Lastly, the fourth column shows the generated normal data from those images.

In this paper, to train the networks properly, the following process is used. First, we preprocessed images by applying a Gaussian filter to remove any possible noise in images [91]. Then, we applied several data augmentation techniques [92], namely, by using random flips, rotations, zooms, warps, lighting transforms, and also presizing [93] to increase the size of training data and help the model to generalize better. Then, each network is trained on the augmented training dataset. After training, networks are evaluated on a test set. Also, to select hyperparameters of networks, the test dataset itself is split into test and validation sets, and the validation set is used to evaluate models with different hyperparameters and find the best setting for each network.

We also studied our models’ performance by training them using an augmented dataset generated by means of a CycleGAN model implemented using the UPIT library [94].

4. Results

4.1. Environment setup and hyper parameter selection

All models were trained using the FastAI library [93] and applying fine-tuning to the pre-trained models available at the timm repository [95] using a GPU Nvidia RTX 2080 Ti with 11 GB of RAM. Aside from CycleGAN data augmentation, other data augmentation methods were also implemented using FastAI, and their parameters are shown in Table 2. As for the CycleGAN implementation, the UPIT library [94] was used. To find the best hyperparameters, such as the learning rate for the task at hand, and to evaluate our models properly, we divided the data into three parts: the first one for training, the second one for validation, and the last one for testing. We used 70 percent of the data for training, 15 for validation, and 15 for testing. While this division is done on data points (images) and not subjects, we made sure that no two slices of any patient are presented in two different parts simultaneously to make the results trustworthy. In [96], it has been shown that a 70/30 split is the best train-to-test ratio. However, given that we needed two different sets for validation and test, we further split the data into 70/15/15. To set the learning rate for the architectures, we employed the two-stage procedure similar to the one presented in [64]; lastly, we applied early stopping in all the architectures to avoid overfitting. The final selected values for batch size and hyperparameters are all available in Table 3.

Table 2.

Data augmentation parameters.

Technique parameter	Value
Resize	640
Size Crop	512
Minimum Scale Crop	0.75
Horizontal Flip	True
Maximum Degree of Rotation	10
Minimum Zoom	1
Maximum Zoom	1.1
Maximum Scale of Changing Brightness	0.2
Maximum Wrap	0.2
Probability of Affine Transform	0.75
Probability of Changing Brightness and Contrast	0.75

Table 3.

Selected hyperparameters for each network.

Network	Batch size	Learning rate
Densenet-121	16	1.00E−03
EfficientNet-B3	16	1.00E−03
Resnet-50	16	1.00E−03
ResNeSt-50	16	1.00E−04
ViT	16	1.00E−05

4.2. Evaluation metrics

The evaluation of each network’s performance is measured by several different statistical metrics, considering that merely relying on one measure of accuracy, it is not possible to measure all the different aspects of the performance of a network. The metrics used in this article are accuracy, precision, recall, F1-score, and area under receiver operating characteristic (ROC) curve (AUC) [97]. How to calculate these metrics is also shown in Table 4. In this table, TP shows the number of positive cases that have been correctly classified, TN has shown the number of negative cases that have been correctly classified, and FP and FN are the numbers of positive and negative cases that have been misclassified, respectively. In addition, for each network, a learning curve is plotted that shows the speed of learning and how to converge.

Table 4.

Statistical metrics for performance evaluation.

Performance evaluation parameter	Mathematical equation
Accuracy	$\frac{TP+TN}{FP+FN+TP+TN}$
Precision	$\frac{TP}{FP+TP}$
Recall	$\frac{TP}{FN+TP}$
F1-Score	$2\frac{Prec\times Sens}{Prec+Sens}$
AUC	Area Under ROC Curve

Nearly all the metrics mentioned before show discriminative performances, i.e., how each network has performed if it merely outputs a label for each input. This approach is quite common and considered de facto in evaluating results for ML methods; however, one can argue that by evaluating models this way, probabilities assigned to each class are totally ignored. Intuitively, probabilities given by models can be viewed as certainty models have assigned to each class, so it might be reasonable to work up a metric that considers these values as well. Nonetheless, these probabilities are considered to measure the reliability of the model.

Here, we have used Expected Calibration Error (ECE) [98] to evaluate the reliability of models; for calculation of this metric, firstly, confidence scores (probabilities assigned by models) are binned uniformly, the number of bins has been selected to be five here. So the first bin would be $[0.0,0.2)$. $B_m$ is all the data points in Bin $m$. For each bin, accuracy is calculated by this equation:

$$Acc\left(B_m\right)=\frac{1}{\left|B_m\right|}\sum _{i\in B_m}1({\hat{y}}_i=y_i)$$ (1)

And confidence as:

$$Conf\left(B_m\right)=\frac{1}{\left|B_m\right|}\sum _{i\in B_m}{\hat{p}}_i$$ (2)

Where $y_i$ is the true label of $i$th sample, ${\hat{y}}_i$ is the assigned label to $i$th sample, and ${\hat{p}}_i$ is the corresponding assigned probability.

Finally, the ECE is calculated by this equation:

$$ECE=\sum _{m=1}^M\frac{\left|B_m\right|}{n}|Acc\left(B_m\right)-Conf\left(B_m\right)|$$ (3)

To give an intuition over this metric, first, we must study what confidence scores show. A confidence of 0.9 indicates that we expect 90 percent of the outcome of the model to be correct if the model is calibrated. Now the distance of actual accuracy of the model with this confidence shows how much the model is calibrated. ECE is a weighted average of these calibration errors.

4.3. Performaces

This part of the paper is dedicated to showing the results of networks. Each network is first trained without using the CycleGAN, and then the effect of adding CycleGAN is measured. Table 5, Table 6 demonstrate the network results without and with CycleGAN, and Fig. 6, Fig. 7 also show the networks’ learning curves. To make the results reliable, each network is evaluated ten times, and then the mean of performances, with confidence intervals, are presented. As observable in these tables, CycleGAN has improved the performance of EfficientNet, Resnet, and ResNeSt dramatically. Nevertheless, the ViT results show no sign of improvement in the presence of CycleGAN; this is arguably due to its robustness or indistinguishability of wrongly classified samples from the other class.

Table 5.

Results without CycleGAN.

Network	Accuracy (%)	Precision (%)	Recall (%)	F1-score (%)	AUC (%)	ECE
Densenet-121	88.05 ± 3.81	80.94 ± 5.55	93.18 ± 3.70	87.36 ± 3.63	96.71 ± 2.80	0.118 ± 0.130
EfficientNet-B3	94.69 ± 2.04	92.55 ± 3.40	96.77 ± 1.57	94.09 ± 2.19	99.03 ± 0.66	0.067 ± 0.012
Resnet-50	94.69 ± 1.15	90.31 ± 2.21	98.74 ± 0.48	94.25 ± 1.15	99.43 ± 0.20	0.060 ± 0.025
ResNeSt-50	96.30 ± 2.31	93.96 ± 3.00	98.02 ± 2.43	95.97 ± 2.53	99.60 ± 1.18	0.020 ± 0.007
ViT	99.60 ± 0.79	99.46 ± 1.39	99.64 ± 0.38	99.55 ± 0.88	99.99 ± 0.10	0.014 ± 0.007

Table 6.

Results with CycleGAN.

Network	Accuracy (%)	Precision (%)	Recall (%)	F1-score (%)	AUC (%)	ECE
Densenet-121	89.24 ± 3.78	81.67 ± 5.84	96.23 ± 2.45	88.64 ± 3.55	97.22 ± 1.89	0.060 ± 0.026
EfficientNet-B3	98.25 ± 2.57	97.03 ± 3.51	99.28 ± 2.20	98.05 ± 2.80	99.79 ± 0.90	0.058 ± 0.185
Resnet-50	96.20 ± 0.79	94.09 ± 1.33	97.49 ± 1.11	95.78 ± 0.87	99.43 ± 0.42	0.033 ± 0.028
ResNeSt-50	98.89 ± 1.09	98.58 ± 1.34	99.10 ± 1.70	98.75 ± 1.24	99.95 ± 0.22	0.013 ± 0.006
ViT	99.20 ± 2.91	98.92 ± 3.97	98.92 ± 2.41	99.10 ± 3.19	99.95 ± 0.92	0.068 ± 0.037

Fig. 6.

Learning curve of networks without CycleGAN; $x$ axis shows the number of epoch and $y$ axis shows reached value for each metric (without any scaling).

Fig. 7.

Learning curve of networks with CycleGAN; $x$ axis shows the number of epoch and $y$ axis shows reached value for each metric (without any scaling).

As observable through results, ECEs are also consistent with other findings; they have improved by applying GAN, and they are mainly better for models with better performances. By this limited testing, applying CycleGAN can be viewed as a means to improve the calibration and reliability of models, yet this is arguably a loose deduction. A substantial literature has investigated methods for improvement of reliability, but as this is not the focus of our study, we have not investigated them here. An interested reader can study about different methods on [99], such as confidence penalty [100] and temperature scaling [101].

Lastly, ROC curve for one run of the networks is also plotted in Fig. 8.

Fig. 8.

ROC curve of networks without and with CycleGAN.

4.4. Qualitative analysis

In previous sections, each network was evaluated thoroughly by quantitative measures, both on performance and reliability. Nevertheless, aside from quantitative results, one expects any network not only to perform well but also to learn to extract meaningful information from data. These results are mainly considered qualitative outputs; one way to assess these performance types is to extract localized importance maps that show how much each part of the image has contributed more toward making the final decision [102].

Here, we have used Grad-CAM [103] to obtain them. This method uses the gradient of target outputs to visualize a heat map of activation from the last convolutional layer; in other words, it shows how much each part of the image is weighted for that specific class. Fig. 9 shows a sample of these types of results. Since showing results for each network would take immoderate space from the manuscript, we have visualized only for ViT and Resnet; this way, we have picked one network from attention-based types (ViT) and one from the conventional CNN types (Resnet). Notably, samples used for this figure are all from abnormal class.

Fig. 9.

Grad-CAM output for Resnet and ViT, with and without CycleGAN.

Fig. 9 demonstrates a few extraordinary points about how each network has learned. Firstly, we can observe that networks have put more weight on the more salient parts of images for the detection of abnormalities in CT images. Secondly, we can observe that attention-based methods have learned a better localization of important parts. It is worth mentioning that values closer to zero contribute the least, whilst the contribution is increased as we get further from zero. Lastly, the effect of applying GAN is also observable from this figure; after applying GAN, networks activations have concentrated more toward informative parts.

5. Discussion

In recent years, convolutional neural networks have revolutionized the field of image processing. Medical diagnoses are no exception, and today in numerous research papers in this field, the use of these networks to achieve the best accuracy is seen. Diagnosis of COVID-19 disease from CT images is also one of the applications of these networks. In this article, the performance of different networks in this task was examined, and also by applying a new method, an attempt was made to improve the performance of these networks. The networks used in this paper were Resnet, EfficientNet, Densenet, ViT, and ResNest, and the data augmentation method was based on CycleGAN. Table 7 summarizes the proposed method of previous papers. By comparing this table with our current work, the advantages of our work can be listed as using ViT, a transformer-based architecture that has achieved state-of-the-art performances; collecting a new dataset; and finally using CycleGAN for data augmentation.

Table 7.

Summary of related works.

Ref	Dataset	Number of Cases (Images)	Pre-Processing	DNN	Performance (%)
[54]	SIRM	3000 COVID-19, 3000 HC	Patches Extraction	PreTrain Networks	Acc $=$ 98.27
[65]	Zhao et al	460 COVID-19, 397 HC	DA	SqueezeNet	Acc $=$ 85.03
[68]	Indian	2373 COVID-19, 6321 HC	–	Ensemble DCCNs	Acc $=$ 98.83
[69]	Different Datasets	–	Visual Inspection	BigBiGAN	–
[55]	Clinical	148 COVID-19, 148 HC	Visual Inspection	ResGNet-C	Acc $=$ 96.62
[70]	Clinical	349 COVID-19, 397 HC	Scaling Process, DA	MKs-ELM-DNN	Acc $=$ 98.36
[56]	COVID-CT	–	DA	U-net + DCN	Acc $=$ 92.87
[67]	Public Dataset	2933 COVID-19	Normalization, Resizing	EDL_COVID	Acc $=$ 99.054
[71]	Clinical	320 COVID-19, 320 HC	HS, Margin Crop, Resizing	FGCNet	Acc $=$ 97.14
[72]	Clinical	–	ROIs Extraction	Modified Inception	Acc $=$ 89.5
[57]	Clinical	3389 COVID-19, 1593 HC	Standard Preprocessing	3D ResNet34	Acc $=$ 87.5
[73]	COVIDx-CT	104,009	DA	COVIDNet-CT	Acc $=$ 99.1
[74]	Different Datasets	349 COVID-19, 397 HC	Resizing, Normalization, DA	ResNet18	Acc $=$ 99.4
[58]	COVID-CT	345 COVID-19, 397 HC	Resizing, DA	CGAN + ResNet50	Acc $=$ 82.91
[75]	Clinical	–	Resizing, Padding, DA	3D-CNN	AUC $=$ 70
[59]	SARS-CoV-2	1252 COVID-19, 1230 HC	–	GAN with WOA + InceptionV3	Acc $=$ 99.22
[66]	Different Datasets	–	Resizing	Inception V1	Acc $=$ 95.78
[76]	Clinical	2267 COVID-19, 1235 HC	Normalization, Cropping, Resizing	ResNet50	Acc $=$ 93
[77]	Clinical	–	Visual Inspection, ROI, Cropping and Resizing	ResNet 101	Acc $=$ 99.51
[60]	Different Datasets	413 COVID-19, 439 HC	–	ResNet-50 + 3D-CNN	Acc $=$ 93.01
[78]	Clinical	–	DA	Multi-View U-Net + 3D-CNN	Acc $=$ 90.6
[61]	Different Datasets	–	Resized, Intensity Normalized	FCN	Acc $=$ 94.67
[79]	Zhao et al	746	–	GAN + ShuffleNet	Acc $=$ 84.9
[80]	COVID-CT	345 COVID-19, 375 HC	2D RDWT, Resizing	ResNet50	Acc $=$ 92.2
[81]	SARS-CoV-2	1262 COVID-19, 1230 HC	–	CSVM	Acc $=$ 94.03
[82]	Different Datasets	767	–	Different PreTrain Methods	Acc $=$ 75
[83]	MedSeg DII	–	Resizing	U-Net + VGG16 and Resnet-50	Acc $=$ 99.4
[84]	Basrah	1425	Resizing	VGG 16	Acc $=$ 99
[62]	Kaggle	1252 COVID, 1240 HC	Resizing, Normalization, DA	COVID CT-net	Acc $=$ 95.78
[85]	COVID-CT	312 COVID-19, 396 HC	Normalization	LeNet-5	Acc $=$ 95.07
Ours	Clinical	1766 COVID-19, 1397 HC	Filtering, DA using CycleGAN	Different PreTrain Methods	Acc $=$ 99.60

Eventually, the ViT network reached an accuracy of 99.60%, which shows its state-of-the-art performance and proves that it can be used as the heart of a CADS. By comparing the performances of our method compared to previous works in Table 1, our methods’ superiority is quite observable. The advantages of adding CycleGAN were also clearly displayed, and it was shown that this method could be used for this task by data augmentation to improve the performance of most deep neural networks. Nonetheless, the reliability of models was also asserted by calibration error, and the effect of applying CycleGAN on the improvement of reliability was demonstrated as well. Lastly, Grad-CAM was used to visualize the activation heatmap of a few samples to show how networks have learned to extract useful information.

Finally, this article’s achievements can be summarized in: first, introducing a new database and its public release; second, examining the performance and reliability of various neural networks on this database by standard evaluation metrics and calibration ones; thirdly, interpreting the decision of networks to find suspicious regions and finally, evaluating the use of CycleGAN for data augmentation and its impact on networks performances and reliability. Additionally, the performance of ViT was never previously studied for this task, which was investigated in this paper as well. To evaluate the method, a CT scan dataset was collected by physicians, which we also made available to researchers in public. Also, due to the fact that this dataset was collected from people suspected of having COVID-19, normal class data, unlike many previous datasets in this field, were collected from patients with suspicious symptoms and not from other diseases.

6. Conclusion, and future works

The new virus COVID-19 or nCOV-2019 is a hazardous infection that enters the human body through the respiratory system. This disease is contagious and is spreading among people all over the world. In recent years, several methods have been tested for rapid diagnosis of COVID-19, including medical tests and medical imaging methods. The most important medical imaging methods in diagnosing COVID-19 include CT, X-ray, and ultrasound. According to the WHO report, people are diagnosed with COVID-19 who have a positive PCR test and a medical imaging method at the same time. Diagnosing COVID-19 from medical images poses many challenges for specialist physicians. To resolve these challenges, in recent years, extensive research has been done in the field of diagnosing COVID-19 from medical images using DL techniques. The main goal of all these research papers is to achieve a DL-based method to help quickly diagnose COVID-19 from medical images.

In this paper, a DL-based CADS is proposed for the diagnosis of COVID-19 from CT images. The proposed CADS consists of dataset creation, pre-processing, feature extraction, and post-processing stages. In this work, the COVID-19 dataset has been gathered clinically by some authors of the paper and made available publicly, and this is the first outcome of the paper. As stated in Section 3 of the manuscript, this dataset contains 1397 images of HC subjects and 1766 images of subjects with COVID-19. CT data pre-processing, including image denoising and data augmentation, is performed in the second step of the proposed CADS. At this stage, a new GAN architecture called CycleGAN was used for data augmentation. In [104], researchers used the CycleGAN method to detect COVID-19. In their research, the CycleGAN model is used in the pre-processing step only for X-ray images [104]. Until now, CycleGAN has not been used to generate artificial CT images in the diagnosis of COVID-19. In our research, for the first time, the CycleGAN model has been utilized to generate artificial CT images in the diagnosis of COVID-19, and therefore this is the second novelty part of this paper. In the third step, the latest pre-trained and transformer models were tested to detect COVID-19 from CT images. In this section, the ViT architecture (vision transformer architecture) is used for the first time for the diagnosis of COVID-19 along with the CycleGAN model, and therefore it is the third and final novelty of the manuscript. The results show that the combination of the CycleGAN can help deep learning models in improving their performances by data augmentation and also that the ViT architecture is well suited for the task at hand. In order to show the performance of the proposed CADS, various statistical parameters, including accuracy, precision, recall, F1-score, and AUC, have been calculated for various DL models. In addition, for the first time, the ECE parameter, which is the most important calibration criterion, has also been calculated for each of the DL models. Finally, a post-processing step based on explainable AI has been done to show the suspected areas of COVID-19 in CT images. At this stage, the researchers used the Grad-CAM method as a post-processing step in the proposed CADS. This step will help the specialist physicians to see the suspected areas of COVID-19 in the CT images for the diagnosis of COVID-19.

For future work, several different paths can be considered; first, more complicated methods in deep neural networks can be used, such as deep metric, few-shot learning, or feature fusion solutions. Also, the combination of different datasets to improve the accuracy and evaluate its impact on the training of different networks can be examined. Finally, combining different modalities to increase accuracy can also be a direction for future research.

CRediT authorship contribution statement

Navid Ghassemi: Conceptualization, Methodology, Software, Validation, Formal analysis, Resources, Writing – original draft, Writing – review & editing, Visualization. Afshin Shoeibi: Conceptualization, Methodology, Software, Validation, Formal analysis, Resources, Writing – original draft, Writing – review & editing, Visualization. Marjane Khodatars: Validation, Formal analysis, Resources, Writing – original draft, Writing – review & editing. Jonathan Heras: Software, Visualization. Alireza Rahimi: Software, Formal analysis. Assef Zare: Writing – review & editing. Yu-Dong Zhang: Methodology, Visualization. Ram Bilas Pachori: Methodology, Visualization. J. Manuel Gorriz: Conceptualization, Methodology, Software, Validation, Formal analysis, Resources, Writing – original draft, Writing – review & editing, Visualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.