Automatic detection of Covid-19 from chest X-ray and lung computed tomography images using deep neural networks and transfer learning

Abstract

The world has been undergoing the most ever unprecedented circumstances caused by the coronavirus pandemic, which is having a devastating global effect in different aspects of life. Since there are not effective antiviral treatments for Covid-19 yet, it is crucial to early detect and monitor the progression of the disease, thereby helping to reduce mortality. While different measures are being used to combat the virus, medical imaging techniques have been examined to support doctors in diagnosing the disease. In this paper, we present a practical solution for the detection of Covid-19 from chest X-ray (CXR) and lung computed tomography (LCT) images, exploiting cutting-edge Machine Learning techniques. As the main classification engine, we make use of EfficientNet and MixNet, two recently developed families of deep neural networks. Furthermore, to make the training more effective and efficient, we apply three transfer learning algorithms. The ultimate aim is to build a reliable expert system to detect Covid-19 from different sources of images, making it be a multi-purpose AI diagnosing system. We validated our proposed approach using four real-world datasets. The first two are CXR datasets consist of 15,000 and 17,905 images, respectively. The other two are LCT datasets with 2,482 and 411,528 images, respectively. The five-fold cross-validation methodology was used to evaluate the approach, where the dataset is split into five parts, and accordingly the evaluation is conducted in five rounds. By each evaluation, four parts are combined to form the training data, and the remaining one is used for testing. We obtained an encouraging prediction performance for all the considered datasets. In all the configurations, the obtained accuracy is always larger than 95.0%. Compared to various existing studies, our approach yields a substantial performance gain. Moreover, such an improvement is statistically significant.

1. Introduction

Covid-19 is a coronavirus-induced infection that can be associated with a coagulopathy and infection-induced inflammatory changes [1]. The disease poses a serious threat to public health, and thus in March 2020, the World Health Organization (WHO) declared Covid-19 a pandemic. At the time of writing, the virus has infected more than four hundreds millions of people, and has claimed over six million peoples’ lives worldwide.1 The clinical spectrum of the disease is very wide, ranging from fever, dry cough and diarrhea, but can be combined with mild pneumonia and mild dyspnoea. In some cases, the infection can evolve to severe pneumonia, causing approximately 5% of the infected patients to severe lung dysfunction. Given the circumstances, patients need ventilation as they are highly exposed to multiple extra pulmonary organ failure.

Since so far there have been no effective antiviral vaccines for Covid-19, it is crucial to reduce mortality by early detecting and monitoring the progression of the disease [2], so as to effectively personalize patient’s treatment. Radiology is part of a fundamental process to detect whether or not the radiological outcomes are consistent with the infection and radiologists should expedite as much as possible the exploration, and provide accurate reports of their findings. Chest X-ray (CXR) and lung computed tomography (LCT) images of Covid-19 patients usually show multifocal, bilateral and peripheral lesions, but in the early phase of the disease they may present a unifocal lesion, most commonly located in the inferior lobe of the right lung. Considering the fact that the number of false positives obtained by swab results is considerably large, an expert system able to provide doctors with a preliminary diagnosis by automated recognition of Covid-19 from CXR or LCT images would be of crucial importance.

The proliferation of disruptive Machine Learning (ML) and especially Deep Learning (DL) algorithms in recent years has enabled a plethora of applications across several domains [3], [4]. Such techniques work on the basis of complex artificial neural networks, which are capable of effectively learning from data by means of a large number of hyper-parameters distributed in different network layers. In this way, they are able to simulate humans’ cognitive functions, aiming to acquire real-world knowledge autonomously [5]. In a nutshell, ML/DL techniques are an advanced paradigm that brings in substantial improvement in performance compared to conventional learning algorithms. In the Health care sector, the potential of ML/DL to allow for rapid diagnosis of diseases has also been proven by various research work [6].

Aiming to assist the clinical care, this paper presents a practical solution for the detection of Covid-19 from CXR and LCT images exploiting two cutting-edge deep neural network families, i.e., EfficientNet [7] and MixNet [8]. Moreover, we empower the learning process by means of three different transfer learning strategies, namely ImageNet [9], AdvProp [10] and Noisy Student [11]. The evaluation on four large CXR and LCT images datasets demonstrate that our proposed models obtain a superior performance compared to the existing studies that we are aware of. In this respect, our work makes the following contributions:

• •A system for detection of Covid-19 from CXR and LCT images exploiting cutting-edge deep learning algorithms;
• •A successful empirical evaluation on four large datasets consisting of CXR and LCT images;
• •A software prototype in the form of a mobile app ready to be downloaded.

The paper is structured as follows. Section 2 briefly introduces EfficientNet and MixNet as well as the transfer learning methods. In Section 3, we present the dataset and metrics used for our evaluation, while in Section 4 we analyze the experimental results. The related work is reviewed in Section 5. Finally, Section 6 discusses future work and concludes the paper.

2. Background

As a base for our presentation, Section 2.1 provides a background on two families of deep neural networks, i.e., EfficientNet and MixNet, which are used as the classification engine in our work. Afterwards, a brief introduction to transfer learning is given in Section 2.2.

2.1. EfficientNet and MixNet

Based on the observation that a better accuracy and efficiency can be obtained by imposing a balance between all network dimensions, EfficientNet [7] has been proposed by scaling in three dimensions, i.e., width, depth, and resolution, using a set of fixed scaling coefficients that meet some specific constraints. By the most compact configuration, i.e., EfficientNet-B0, there are 18 convolution layers in total, i.e., D $=18$, and each layer is equipped with a kernel k(3,3) or k(5,5). The input image contains three color channels R, G, B, each of size 224 × 224. The next layers are scaled down in resolution to reduce the feature map size, but scaled up in width to increase accuracy. For instance, the second convolution layer consists of W $=$ 16 filters, and the number of filters in the next convolution layer is W $=$ 24. The maximum number of filters is D $=$ 1280 in correspondence of the last layer, which is fed to the final fully connected layer. The other configurations of the EfficientNet family are generated from EfficientNet-B0 by means of different scaling values [7]. EfficientNet-B7 outperforms a CNN by achieving a better accuracy, while considerably reducing the number of parameters.

Generally, kernels of size k(3,3) [12], [13], k(5,5) [14], or k(7,7) [15] are used as filters for deep neural networks. However, larger kernels can potentially improve a model’s accuracy and efficiency. Furthermore, large kernels help to capture high-resolution patterns, while small kernels allow us to better extract low-resolution ones. To maintain a balance between accuracy and efficiency, the MixNet [8] family has been built based on the MobileNets architectures [13], [16]. This network family also aims to reduce the number of parameters as well as FLOPs, i.e., the metric used to measure the computational complexity counted as the number of float-point operations (in billions). The most simple architecture of the MixNet family is MixNet-Small, which consists of a large number of layers and channels. Furthermore, the size of the filters varies depending on the layers. Similar to the EfficientNet family, other configurations of the MixNet family, such as MixNet-Medium or MixNet-Large, are derived from MixNet-S with different scaling values.

2.2. Transfer learning

To fine tune the hyper-parameters, i.e., the internal weights and biases, deep neural networks need a huge number of labeled data. Moreover, the deeper/wider a network is, the more parameters it possesses. As a result, deeper/wider networks require more data to guide the learning, with the aim of avoiding overfitting and being effective. Thus, it is crucial to train them with enough data, in order to facilitate the learning. Nevertheless, such a requirement is hard to come by in real-world settings, since the labeling process usually is done manually, thus being time consuming and subject to error [17]. To this end, transfer learning has been conceptualized as an effective way to extract and transfer the knowledge from a well-defined source domain to a novice target domain [18], [19]. In other words, transfer learning facilitates the export of existing convolution weights from a model trained using large datasets to create new accurate models exploiting a relatively lower number of labeled images. As it has been shown in various studies [20], [21], transfer learning remains helpful even when the target domain is quite different from the one in which the original weights have been obtained. In this work, we consider the following learning methods:

• •ImageNet [9]: The ImageNet dataset has been widely exploited to apply transfer learning by several studies, since it contains more than 14 million images, covering miscellaneous categories;
• •AdvProp [10]: adversarial propagation has been proposed as an improved training scheme, with the ultimate aim of avoiding overfitting. The method treats adversarial examples as additional examples, and uses a separate auxiliary batch norm for adversarial examples;
• •NS [11]: the Noisy Student learning method attempts to improve ImageNet classification Noisy Student Training by: (i) enlarging the trainee/student equal to or larger than the trainer/teacher, aiming to make the trainee learn better on a large dataset, and (ii) adding noise to the student, thus forcing him to learn more.

To assist doctors in early detecting Covid-19 from CXR and LCT images, we develop an expert system that makes use of EfficientNet and MixNet as the classification engine. Moreover, in order to accelerate the learning process and to achieve a higher accuracy, we propose the three different learning strategies for obtaining network weights mentioned above, i.e., ImageNet, AdvProp, and NS. The succeeding section introduces the evaluation settings used to study the performance of our approach.

3. Evaluation

We explain in detail the datasets and methods used to study the performance of our proposed solution. Four real-world datasets have been used in the evaluation. Moreover, we make use of recent implementations2 of EfficientNet and MixNet, which have been built atop the PyTorch framework.3 We imported pre-trained weights from various sources to speed up the training. We publish the tool developed through this paper in GitHub to facilitate future research.4

3.1. Research objectives

Given an automated classifier, it is of crucial importance to get accurate predictions with respect to various quality metrics. In the evaluation, we focus on the following performance characteristics: (i) We identify which neural network family obtains the best prediction performance for the CXR datasets; (ii) Afterwards, we determine which deep neural network is the most suitable one for classifying LCT images; (iii) We ascertain which transfer learning method among ImageNet, AdvProp, and NS is beneficial to EfficientNet or MixNet; (iv) We measure the average recognition speed to see if the proposed model is feasible in practice in terms of timing efficiency; Finally, (v) A comparison with well-established baselines is performed to validate if our approach brings advancements in the field.

3.2. Datasets

To investigate the performance of our approach, with CXR we exploited two datasets used in some previous studies [22], [23]. Their characteristics are summarized in Table 1, Table 2. In all of them, except D ${}_3$, there are three categories, i.e., Covid-19 , Normal, and Pneumonia. Only the category Normal exhibits no symptoms, while the other two categories, Covid-19 and Pneumonia, correspond to different levels of infection-induced inflammatory changes.

Table 1.

Chest X-ray datasets.

Dataset	Type	Categories			Total
		Covid-19	Normal	Pneumonia
D${}_1$	Train	98	7,966	5,447	13,511
	Test	10	885	594	1,489
	Total	108	8,851	6,041	15,000
D${}_2$	Train	261	8,154	5,909	14,324
	Test	66	2,038	1,477	3,581
	Total	327	10,192	7,386	17,905

Table 2.

Lung CT datasets.

Dataset	Type	Categories			Total
		Covid-19	Normal	Pneumonia
D${}_3$[24]	Train	1,001	–	984	1,985
	Test	125	–	123	248
	Validation	126	–	123	249
	Total	1,252	–	1,230	2,482
D${}_4$[25]	Train	109,250	67,029	111,791	288,070
	Test	23,410	14,364	23,955	61,729
	Validation	23,411	14,364	23,955	61,730
	Total	156,071	95,756	159,702	411,528

Dataset D${}_1$ consists of 13,511 images for training and 1489 images for testing. It can be seen that there is an imbalance among the categories. In particular, both categories Normal and Pneumonia contain a large number of images, while there are only 108 images in the Covid-19 category. We use D ${}_1$ as a means to compare the performance of our approach with some existing studies that exploited D ${}_1$ in their evaluation [22], [23]. Compared to D ${}_1$, D ${}_2$ is newly updated with more data for training and testing. This means that there are overlaps between the two datasets. In particular, D ${}_2$ consists of 14,324 and 3581 images for training and testing, respectively. We made use of D ${}_2$ to validate the performance of our approach on a larger amount of data, proving its feasibility in real-world settings.

Table 2 summarizes the two LCT datasets used in our evaluation. Dataset D ${}_3$ contains a small number of images in two categories, i.e., Covid-19 and Pneumonia. Finally, D ${}_4$ is much larger with 156,017 images for the Covid-19 category. In total, D ${}_4$ sums up to 411,528 images for training and testing. We believe that such a large dataset resembles a real scenario, and thus the evaluation on it helps to shed light on the performance of the approach in practice.

Concerning the settings, at the beginning of all experiments the datasets were randomly split using a random seeds function of the Python numpy library. With respect to the strategy it ensures that the test sets are independent and all the same for the other evaluations of each architecture. In other words, we can measure and compare prediction performance on the same criterion. This is done by following many existing studies [4], [26], [27], [28], which measured the prediction performance on independent test sets.

Fig. 1, Fig. 2 give examples of CXR and LCT images for the three categories mentioned above. By plain eyes, it is quite difficult to notice the difference between them, especially for non-expert users. In this respect, we assume that the application of deep learning will facilitate the recognition and effectively assist doctors in diagnosing the disease.

Fig. 1.

Examples of CXR images.

Fig. 2.

Examples of LCT images.

3.3. Settings

Deep neural networks such as EfficientNet and MixNet require a platform with powerful computational performance. To perform the evaluation, we used a server with the hardware and software configurations listed in Table 3.

Table 3.

Hardware and software configurations.

Name	Description
RAM	24 GB
CPU	Intel® Core™ i5-2400 CPU @ 3.10 GHz ×4
GPU	GeForce GTX 1080 Ti
OS	Ubuntu 18.04
Python	3.7.5
Pytorch	1.5
Torchvision	0.5.0
Numpy	1.15.4
Git	2.0
Timm	0.1.26

By means of an empirical study, we found out that two configurations, EfficientNet-B0 and EfficientNet-B3, are more effective than the others for the EfficientNet family, and thus we selected them for our evaluation. Weights for EfficientNet are obtained by all the transfer learning techniques mentioned in Section 2.2. In the MixNet family, we considered four different configurations, i.e., MixNet-Small, MixNet-Medium, MixNet-Large and MixNet-XL. We obtained weights coming from the available ImageNet dataset. At the end, by combining two network families with three learning strategies mentioned in Section 2.2, we got 10 experimental configurations C${}_1$, …, C₁₀, as shown in Table 4. Batch size corresponds to the number of items used for each training step; # of Params specifies the number of parameters used by each network; and finally Size is the file size needed to store the parameters. It is evident that EfficientNet-B3 is the largest network with respect to the number of parameters as well as the file size to store them. To be concrete, i.e., C${}_4$, C${}_5$, and C${}_6$ have more than 14 millions of parameters, accounting for more than 100MB of storage space each. In the evaluation, we employed the five-fold cross-validation methodology on the datasets. Namely, each dataset is split into five equal parts, and in each validation round one part is used for testing and the other four ones for the training.

Table 4.

Experimental configurations.

Conf.	Network	Batch size	# of Params	Learning method	Size (MB)
C${}_1$	EfficientNet-B0	110	7,919,391	ImageNet	53.1
C${}_2$	EfficientNet-B0	110	7,919,391	AdvProp	53.1
C${}_3$	EfficientNet-B0	110	7,919,391	NS	53.1
C${}_4$	EfficientNet-B3	64	14,352,075	ImageNet	106.9
C${}_5$	EfficientNet-B3	64	14,352,075	AdvProp	106.9
C${}_6$	EfficientNet-B3	64	14,352,075	NS	106.9
C${}_7$	MixNet-Small	110	6,253,449	ImageNet	41.8
C${}_8$	MixNet-Medium	90	7,133,225	ImageNet	48.9
C${}_9$	MixNet-Large	60	9,448,095	ImageNet	67.5
C10	MixNet-XL	60	14,015,611	ImageNet	104.2

3.4. Evaluation metrics

Each image in all the datasets has been manually labeled, i.e., either Normal or Pneumonia or Covid-19 , resulting in three independent groups, i.e., $G=(G_1,G_2,G_3)$, called ground-truth data. Using either EfficientNet or MixNet as the classifier on a test set, three predicted sets, i.e., $S=(S_1,S_2,S_3)$ of images are obtained. We measured the classification performance by evaluating the similarity of the classified categories with the ground-truth ones. To this end, three metrics, namely accuracy, precision and recall, and $F_1$ score are used [21]. We selected these metrics for the following reason: precision, recall and ${\mathrm{F}}_1$ are useful, given that the number of positive images accounts for a very small percentage of all the items in the dataset. In fact, in this case a classifier always providing a negative prediction would have a very high accuracy.

True positive $T{P}_i=|G_i\cap S_i|$, $i=1,2,3$ is defined as the number of items that appear both in the results and ground-truth data of class $i$. The metrics are defined as follows.

Accuracy: It is the fraction of correctly classified items with respect to the total number of images in the test set.

$$accuracy=\frac{\sum _i^{3}T{P}_i}{\sum _i^3\left|G_i\right|}\times 100\text{\%}$$ (1)

Precision and Recall:Precision is the fraction of classified images for S${}_i$ being found in the ground-truth data G${}_i$, while Recall the fraction of true positives being found in the ground-truth data.

$$precisio{n}_i=\frac{T{P}_i}{\left|S_i\right|}$$ (2)

$$recal{l}_i=\frac{T{P}_i}{\left|G_i\right|}$$ (3)

${\mathbf{F}}_1$score (F-Measure): It is calculated as the average of precision and recall using the following formula:

$$F_1=\frac{2\cdot precisio{n}_i\cdot recal{l}_i}{precisio{n}_i+recal{l}_i}$$ (4)

While accuracy is important, we are also interested in efficiency, considering the fact that the model needs to have a high recognition speed in practice.

Speed: The system presented in Table 3 is used to benchmark the processing time, i.e., the average number of predicted items in a second.

In the next section, we present in detail the experimental results by referring to the research questions introduced in Section 3.1.

4. Results

This section reports and analyzes the results obtained from the conducted experiments by referring to the research objectives presented in Section 3.1.

4.1. Performance on the CXR datasets

The results obtained by performing the experiments on D ${}_1$ are shown in Table 5. The accuracy for all configurations is always larger than 95%, and the maximum accuracy is 96.64% obtained by C${}_5$, i.e., EfficientNet-B3 using pre-trained weights with AdvProp. With respect to Precision, eight among ten configurations get 1.000 as precision for the Covid-19 category. This means that all images classified as Covid-19 by the classifiers are actually Covid-19 . For the other two categories, i.e., Normal and Pneumonia, the maximum precision is 0.968, achieved also by C ${}_5$ for Category Normal, and by C ${}_4$ for Category Pneumonia. Altogether, we see that all the classifiers are able to recognize the testing images, obtaining high precision.

Table 5.

Experimental results on dataset D${}_1$.

Concerning recall, all the configurations get a considerably low score for the Covid-19 category: the highest recall is 0.700, obtained by C${}_5$. This means that the models are not able to retrieve all the items in the ground-truth data, though they can yield good predictions for the category. We assume that this happens due to the limited data available in the training set. Referring back to Table 1, for the Covid-19 category there are only 98 images and 10 images for training and testing, respectively. Meanwhile, for other two image categories, the recall scores are substantially improved. The best performance is seen by category Normal, i.e., 0.985; while by Pneumonia, recall is 0.952. As depicted in Table 1, these categories consist of a larger number of training and testing images compared to the Covid-19.

For what concerns the $F_1$ scores, for the Covid-19 category the maximum $F_1$ is 0.778, obtained by C${}_5$. The classifiers obtain a low $F_1$ in the other configurations, and this happens due to the low recall scores as shown above. For the other two categories Normal and Pneumonia, the $F_1$ scores are improved considerably compared to Covid-19 . C${}_5$ is the most suitable configuration as it obtains the best $F_1$ scores for all the categories, i.e., also 0.973 for Normal and 0.958 for Pneumonia.

Similarly, let us analyze the results on dataset D ${}_2$ presented in Table 6 to ascertain the best network configuration. It is evident that C${}_1$ achieves most of the best scores with respect to various metrics as well as categories. For instance, C${}_1$ get 95.82% as accuracy, together with C${}_8$, which is the best over all the configurations.

Table 6.

Experimental results on dataset D${}_2$.

Altogether, through Table 5, Table 6 we can see that C${}_1$ and C${}_5$ are the configurations among the others that bring the best prediction performance.

Compared to existing work that performs evaluation on the same dataset [22], [29], our approach achieves a better performance with respect to accuracy, precision, recall, and ${\mathrm{F}}_1$-score. For instance, the work by Wang et al. [22], the maximum accuracy is 93.0% with similar experimental settings. In this respect, we conclude that application of the two network families EfficientNet and MixNet as well as the different transfer learning techniques brings a good prediction performance on the considered dataset.

We conclude that EfficientNet and MixNet can successfully predict Covid-19 from CXR images, obtaining a high accuracy and precision. Among others, EfficientNet yields the best prediction performance.

4.2. Performance on the LCT datasets

The experimental results for the two LCT datasets, i.e., D ${}_3$ and D ${}_4$ are shown in Table 7, Table 8, respectively. Table 7 demonstrates that EfficientNet-B0 trained with ImageNet is the most effective configuration, i.e., it obtains 97.99% as accuracy. Moreover, this configuration also gets the best ${\mathrm{F}}_1$ score for both categories.

Table 7.

Experimental results on dataset D${}_3$.

Table 8.

Experimental results on dataset D${}_4$.

Concerning D${}_4$, the dataset consists of a large number of images (cf. Table 2). This resembles a real-world scenario where images are collected and fed to the system on a daily basis, resulting in a big database. We see that both network families get a high performance with respect to different metrics. All the configurations obtain an accuracy larger than 98% with 99.66% as maximum accuracy.

In comparison to existing studies that performed evaluation on same datasets, we can see that our proposed framework achieves a better performance. For instance, He et al. [30] obtain 87.93% as maximum accuracy, while we get 97.99% and 99.66% for D ${}_3$ and D ${}_4$, respectively.

On the LCT datasets, EfficientNet is also the network family that helps to obtain the best prediction performance.

4.3. The benefit of transfer learning

We conducted experiments following the five-fold cross- validation technique. Moreover, to further investigate the applicability of the proposed approach, we made use of D ${}_2$ and D ${}_4$, which contain more images than D ${}_1$ and D ${}_3$ (cf. Table 1). Figs. 3(a), 3(b), and 3(c) depict the confusion matrices for EfficientNet-B0 using the three different transfer learning techniques mentioned in Section 2.2. The metrics for all the confusion matrices are shown in Table 6.

Fig. 3.

Confusion matrices of EfficientNet and MixNet using different transfer learning techniques on D${}_2$.

As we can see, each transfer learning method may have different effects on the different categories. For example, using EfficientNet-B0 with weights pre-trained by ImageNet is beneficial to Covid-19 and Pneumonia, but not to Normal. As shown in Fig. 3(a), 61 out of 66 images in Covid-19 are correctly classified, while for Pneumonia 1,392 out of 1,477. However, for the Normal category, only 1978 images are correctly classified over a total of 2,038 images, accounting for 97.05%. On the other hand, transfer learning with AdvProp (cf. Fig. 3(b)) induces a better performance for Normal, i.e., 1,981 among 2,038 images are classified to the correct categories. In Fig. 3(c), we see that compared to the other learning methods, NS has an adverse effect on the recognition of all the categories. Altogether, we come to the conclusion that training EfficientNet-B0 with weights from ImageNet yields the best prediction performance.

For EfficientNet-B3, we see that weights pre-trained with ImageNet are beneficial to the Normal category (cf. Fig. 3(d)). At the same time, AdvProp is the transfer learning method that is suitable for recognition of Pneumonia, i.e., it helps to detect 1,388 out of 1,477 pneumonia images, which is the best among the others.

Among the configurations, C${}_9$ is the best one for the Pneumonia category. Other MixNet configurations do not outperform the ones of EfficientNet-B0 and EfficientNet-B3. MixNet-XL obtains a considerably good performance with Category Normal, correctly classifying 1,984 images among 2,038 images, while it suffers of a low precision and recall for the other categories. For instance, with Pneumonia, only 1,377 out of 1,477 images are properly recognized by MixNet-XL with weights pre-trained with ImageNet. The experimental results demonstrate that, depending on the network family, each transfer learning technique has a diverse influence on the final outcomes. By considering the results in Table 6, it is evident C${}_1$, corresponding to training EfficientNet-B0 with weights by ImageNet, is the most effective configuration with respect to accuracy, precision, recall, and ${\mathrm{F}}_1$ for almost all categories. Moreover, together with the results obtained from Section 4.1, we conclude that ImageNet is the best transfer learning strategy for both network families on the two datasets D ${}_1$ and D ${}_2$.

Next, we consider the results obtained for the LCT dataset in Figs. 4(a)–4(j). In general, training the deep neural networks with ImageNet helps to obtain a better performance compared to the other transfer learning techniques in all the three categories. As an example, with C${}_1$ we get 23,281 correctly classified Covid-19 images, while the corresponding numbers for AdvProp and NS are 23,270 and 23,137, respectively. For other categories, training EfficientNet-B0 with ImageNet weights also gets a good performance.

Fig. 4.

Confusion matrices of EfficientNet-B0 and EfficientNet-B3 using different transfer learning techniques on D${}_4$.

The experimental results show that weights pre-trained from the ImageNet dataset contribute to the best prediction performance on all the CXR and LCT datasets.

4.4. Timing efficiency

We investigate whether the model is feasible in practice by measuring the time efficiency. The system presented in Table 3 is used to benchmark the processing time, i.e., the average number of predicted items in a second for the cross-validation experiments. For these experiments, D ${}_2$ and D ${}_4$ have been selected as they contain a larger number of images compared to D ${}_1$ and D ${}_3$. We counted the number of predictions returned by the classifiers in a second for the two datasets. The results are depicted in Fig. 5.

Fig. 5.

Recognition speed for the configurations on dataset D${}_2$ and D${}_4$.

Concerning D${}_2$, from the figure it is clear that the considered configurations have a comparable recognition speed. For instance, C${}_1$, C${}_2$, C${}_3$, and C${}_7$ corresponding to using EfficientNet-B0 and MixNet-S as the classification engine, are the most efficient configurations, as they return 206 and 235 images per second in average. EfficientNet-B3 also yields a good time performance, i.e., using C${}_4$, C${}_5$, or C${}_6$ as the experimental configuration, the system generates 165 and 183 predictions per second.

The recognition speed is improved a lot by D ${}_4$, i.e., on the LCT dataset with more images for training, compared to D ${}_2$. The maximum speed is 965 images per second and reached by C${}_1$, C${}_2$, i.e., using EfficientNet-B0 together with ImageNet and AdvProp. Among the MixNet configurations, MixNet-S is the most efficient one, as it classifies 921 images per second, which is much higher compared to 664, 510 and 367 by MixNet-M, MixNet-L, and MixNet-XL, respectively.

Together with the results obtained from Section 4.1 and Section 4.2, the results demonstrate that the proposed framework is feasible in real-world settings, i.e., it can be used to effectively and efficiently predict Covid-19 from CXR and LCT images. Moreover, EfficientNet-B0 is the most efficient model on both datasets, while the MixNet family yields a low timing efficiency.

4.5. Comparison with a baseline

ResNet is a modern deep neural network, and it has been widely used in different classification tasks [31]. For the detection of Covid, ResNet50 has been applied in various work [32], [33], obtaining a promising prediction performance. In this section, we run experiments to compare our approach with two baselines built on top of the ResNet50 architecture. For the comparison, we use the larger dataset among either the Chest X-ray and Lung Computed Tomography categories, i.e., D ${}_2$ and D ${}_4$, aiming to study the generalizability.

The experimental results obtained using ResNet50 are shown in Table 9. The approach gets an accuracy of 87.53% and 91.93% on D${}_2$ and D ${}_4$, respectively. ResNet50 performs the best on the Pneumonia, i.e., it always gets a maximum value for all the three metrics (Precision, Recall, and F1-score). Concerning Precision, ResNet50 gets the maximum value of 0.923 by the Pneumonia category on D ${}_4$. The corresponding scores for Recall and F1-score are 0.952 and 0.938, respectively.

Table 9.

Results obtained with ResNet50.

Configuration		D${}_2$	D${}_4$
Accuracy (%)		87.53	91.93
Precision	Covid-19	0.880	0.909
	Normal	0.843	0.922
	Pneu.	0.900	0.923
Recall	Covid-19	0.846	0.926
	Normal	0.902	0.813
	Pneu.	0.874	0.952
${\mathrm{F}}_1$-score	Covid-19	0.863	0.918
	Normal	0.872	0.864
	Pneu.	0.887	0.938

While ResNet50 obtains a good prediction, compared to our approach, its performance is considerably lower. For instance, as shown in Table 8, on the D ${}_4$ dataset, with Configuration C ${}_1$ our approach gets 99.66% as accuracy. The maximum Precision, Recall, and F1 scores are 0.999, 0992, and 0.999, respectively. Finally, we investigate if the gain in performance is statistically significant. To this end, we perform a Wilcoxon signed-rank test [34] with a significance level of $\alpha =5\text{\%}$ on the obtained Precision, Recall, and F1 scores from both approaches. The final $p$-value is $1.903\times 10^{-7}$, which means that the difference is statistically significant. Altogether, this demonstrates that compared to the baselines, we always achieve a better performance.

The original EfficientNet-B0 configuration contains 5.29M params, 385.89 MMac, and it obtains an accuracy of 76.3% on the ImageNet-1K dataset [9]. However, we noticed that it cannot distinguish well between Covid-19 with other pneumonia. Thus, we attempted to improve this by modifying the original configuration by adding 12 layers to boost up the prediction accuracy for the medical image classification for Sars-CoV-2 based on Chest X-ray (CXR) and Computed Tomography (CT). The new proposed model EfficientNet-B0 contains 7.92 M params, but still keeps the same number of MMac, i.e., 385.89 MMac. In summary, the modification of the original backbones is our key contribution for the classification task.

4.6. Threats to validity

We discuss the threats that may affect the internal, external, construct, and conclusion validity of our findings.

Internal validity. This concerns the internal factors that might have an adverse influence on the findings. A possible threat here could come from the results for the Covid-19 category, since they are obtained with a considerably low number of items for training and testing, i.e., D ${}_1$ with 98 and 10 images and D ${}_2$ with 327 and 98 images for training and testing, respectively. This threat is mitigated by the other two categories in the datasets, as they contain a considerably large number of items. To the best of our knowledge, there exists no CXR dataset with more images for the Covid-19 category. As a matter of fact, research in medical imaging on Covid-19 suffers a general lack of data. For this reason, unfortunately we are not able to study the models on a larger scale.

External validity. The main threat to external validity is due to the factors that might hamper the generalizability of our results to other scenarios outside the scope of this work, e.g., in practice we may encounter a limited amount of training data. The threat is moderated by evaluating EfficientNet and MixNet using the experimental settings following the five-fold cross-validation methodology. In particular, the dataset is split into five parts, and accordingly the evaluation is conducted in five rounds. By each round, four parts are combined to form the training data, and the remaining one is used for testing.

Construction validity. This is related to the experimental settings presented in the paper, concerning the simulation performed to evaluate the system. To mitigate the threat, the evaluation has been conducted on a training set and a test set, attempting to simulate a real usage where training data is already available for feeding the system, while testing data is the part that needs to be predicted. Depending on the given settings, we ran several rounds of training/testing, to make sure that the final results are generalizable, i.e., they are not obtained by chance. In the paper, we reported the most stable experimental results.

Conclusion validity. This is related to the factors that might have an impact on the obtained outcome. The evaluation metrics accuracy, precision, recall, ${\mathrm{F}}_1$ and execution time might induce a conclusion threat. To face the issue, we adopted such measures as recommended by the previous scientific literature related to our setting, and employed the same metrics for evaluating all the classifiers.

5. Related work

The recent months have witnessed a large number of studies related to the topic Covid-19 and Machine Learning, and multiple Covid-19/ML applications have been proposed. We summarize in Table 10 some of the most notable research, providing the number of considered images for each category as well as the prediction accuracy. In this work, since we support the recognition of Covid-19 from CXR and LCT images, in the remaining of this section we concentrate on analyzing these studies.

Table 10.

A summary of the related studies on CXR datasets.

Study	Number of images			Network	Acc. (%)
	Covid-19	Normal	Pneumonia
Ghoshal et al. [35]	68	1583	2786	ResNet	89.82
Abbas et al. [36]	105	80	11	DeTraC based on ResNet-18	95.12
Nari et al. [37]	50	50	–	ResNet-50	98.00
Apostolopous et al. [38]	224	504	700	VGG19	93.48
Luz et al. [39]	183	–	–	EfficientNet-B3	93.90
Zhang et al. [40]	100	1431	1531	ResNet18	96.00
Hemdan et al. [41]	25	25	–	VGG19, DenseNet121	90.00

Deep learning techniques have been exploited to predict which current antivirals might be more effective in patients infected with coronavirus [48], [49]. Similarly, a specific model has been developed to forecast if a Covid-19 patient has the chance to survive based on his personal data as well as other risk factors [50]. Jiang et al. [51] propose an approach to the identification of clinical characteristics of Covid-19, and develop an AI tool to recognize patients at risk of a more severe impact of the disease.

Ozturk et al. [52] proposed an approach to deliver accurate diagnostics for binary classification, i.e., Covid-19 vs. No-Findings and multi-nominal classification, i.e., Covid-19 and No-Findings and Pneumonia. The DarkNet model has been exploited as the classification engine, consisting of 17 convolutional layers and a different filtering in each layer. The proposed model has a prediction accuracy of 98.08% and 87.02% for binary and multi-nominal classification, respectively.

Various studies have demonstrated the usefulness of CXR exams in detecting Covid-19. Hall et al. [53] analyzed 135 CXR images confirmed as Covid-19 and 320 images of viral and bacterial pneumonia. A Resnet50 DNN was trained on 102 Covid-19 and 102 pneumonia cases by means of the ten-fold cross-validation technique. The experimental results showed an overall accuracy of 89.2% with a Covid-19 true positive rate of 0.804 and an area under the curve (AUC) of 0.95. As a matter of fact, the dataset used by Hall et al. [53] is quite small, and it is necessary to investigate the proposed model on a larger amount of data to see if it still achieves such a good performance.

Narin et al. [37] developed a system for the detection of coronavirus patients from CXR images. Three different CNN-based models have been exploited, i.e., ResNet50, InceptionV3 and Inception-ResNetV2. The results show that the ResNet50 model achieves the best prediction performance with 98.0% as the final accuracy. Although the approach obtains a good classification performance, it has been studied on a considerably small dataset. It is our belief that performance may substantially change on larger datasets like the ones used in our evaluation.

Apostolopoulos et al. [38] evaluated their solution to automatic detection of Covid-19, making use of a dataset of CXR images from patients with common bacterial pneumonia, confirmed Covid-19, and normal incidents. The datasets consists of 1427 CXR images including 224 images with confirmed Covid-19 cases, 700 images with common bacterial pneumonia, and 504 images of normal situations. The experimental outcomes demonstrate that deep neural networks can be exploited to extract important biomarkers related Covid-19. Nevertheless, like some other existing studies et al. [37], again the approach has been studied by means of a small amount of data. It is our assumption that such a good performance might considerably change with larger datasets.

COVID-Net [54] is a deep convolutional neural network design tailored for the detection of Covid-19 cases from CXR images. COVID-Net achieves an accuracy of 93.3%, with 98.9% positive predictive values that is related to the detection of false positives.

A deep learning model has been proposed [55] to detect Covid-19 and differentiate it from common acquired pneumonia and other lung diseases. The analyzed dataset consists of 4356 chest CT exams collected from 3322 patients. The per-exam sensitivity and specificity for detecting COVID-19 in the independent test set was 114 of 127 (90.0%) and 294 of 307 (96.0%), respectively, with an area under the receiver operating characteristic curve (AUC) of 0.96 (p-value $<$ 0.001). The per-exam sensitivity and specificity for detecting community acquired pneumonia in the independent test set was 87% (152 of 175) and 92% (239 of 259), respectively.

Abbas et al. [36] introduced Decompose, Transfer, and Compose (DeTraC), a deep neural network for automated recognition of Covid-19 from CXR images. An accuracy of 95% was achieved in the detection of Covid-19 CXR images from normal, and severe acute respiratory syndrome cases. COVID-CAPS [56] is a capsule Network-based Framework for Identification of Covid-19 cases from CXR Images. The approach yielded a good accuracy when working with small datasets.

Concerning LCT datasets, we review the most relevant studies to our work in Table 11. Overall, most of the studies use a small dataset for their evaluation. Only Gunraj et al. [44] made use of a considerably large dataset, however the obtained performance is much lower than our proposed approach, i.e., 95.90% compared to 99.66%. In this respect, we conclude that by combining different deep neural networks with transfer learning strategies, we can obtain a high prediction accuracy even on a very large dataset.

Table 11.

A summary of the related studies on LCT datasets.

Study	Number of images			Network	Acc. (%)
	Covid-19	Normal	Pneumonia
Rahimzadeh et al. [42]	465	7878	–	ResNet50V2	98.49
Anwar et al. [43]	98	–	105	EfficientNet	89.70
Gunraj et al. [44]	4346	9450	7395	COVID-Net-CT	99.10
Mobiny et al. [45]	47	58	–	DECAPS	87.60
He et al. [30]	25,442	14,471	28,160	ResNet3D34	95.90
Ardakanu et al. [46]	510	510	–	ResNet-101	99.63
Bai et al. [47]	5030	–	9152	EfficientNet-B4	96.00

To the best of our knowledge, compared to different existing studies [36], [38], our work is the first one that deals with big datasets. In particular, for the CXR datasets, there are 15,000 and 17,905 images in D ${}_1$ and D ${}_2$, respectively [57]. While for the LCT datasets, especially with D ${}_4$, there are more than 400K of images. This suggests that with the proposed model we can work with a huge number of images and still obtain a good timing efficiency. More importantly, the application of the two deep neural network families, i.e., EfficientNet and MixNet, allows us to build an expert system being capable of working with images coming from different sources.

6. Conclusions

We presented a workable solution for the detection of Covid-19 from chest X-ray and lung computed tomography images. We designed and implemented the models based on two building blocks: EfficientNet and MixNet as the prediction engine and effective transfer learning algorithms. The proposed models have been studied by means of four datasets which have been widely used in various papers. The experimental results show that our proposed approach obtains a better prediction performance compared to some relevant previous studies. To the best of our knowledge, our work is the first one that deals with images coming from different sources. As future work, we are going to refine and evaluate the approach by taking into consideration more datasets and tuning other deep neural network configurations.

CRediT authorship contribution statement

Linh T. Duong: Conceptualization, Software, Writing – original draft. Phuong T. Nguyen: Conceptualization, Methodology, Writing – original draft, Writing – review & editing. Ludovico Iovino: Validation, Visualization, Writing – review & editing. Michele Flammini: Writing – review & editing, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The authors do not have permission to share data.