An Analysis of Image Features Extracted by CNNs to Design Classification Models for COVID-19 and Non-COVID-19

Abstract

The SARS-CoV-2 virus causes a respiratory disease in humans, known as COVID-19. The confirmatory diagnostic of this disease occurs through the real-time reverse transcription and polymerase chain reaction test (RT-qPCR). However, the period of obtaining the results limits the application of the mass test. Thus, chest X-ray computed tomography (CT) images are analyzed to help diagnose the disease. However, during an outbreak of a disease that causes respiratory problems, radiologists may be overwhelmed with analyzing medical images. In the literature, some studies used feature extraction techniques based on CNNs, with classification models to identify COVID-19 and non-COVID-19. This work compare the performance of applying pre-trained CNNs in conjunction with classification methods based on machine learning algorithms. The main objective is to analyze the impact of the features extracted by CNNs, in the construction of models to classify COVID-19 and non-COVID-19. A SARS-CoV-2 CT data-set is used in experimental tests. The CNNs implemented are visual geometry group (VGG-16 and VGG-19), inception V3 (IV3), and EfficientNet-B0 (EB0). The classification methods were k-nearest neighbor (KNN), support vector machine (SVM), and explainable deep neural networks (xDNN). In the experiments, the best results were obtained by the EfficientNet model used to extract data and the SVM with an RBF kernel. This approach achieved an average performance of 0.9856 in the precision macro, 0.9853 in the sensitivity macro, 0.9853 in the specificity macro, and 0.9853 in the F1 score macro.

Introduction

The subtype 2 coronavirus, also known as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), belongs to a family of viruses that cause respiratory diseases in humans [1]. The outbreak of coronavirus disease 2019 (COVID-19) caused the collapse of health systems in several countries. The infrastructure of public and private health centers did not endure the number of patients with severe symptoms of COVID-19, which contributed to the increase in the number of victims. Thus, some governments have adopted large-scale tests to obtain early diagnosis and reduce the effects of the disease on health systems.

The reverse transcription polymerase chain reaction test (RT-PCR) of the combined oral/nasal swab confirms the diagnosis of COVID-19 by detecting the genetic material of the virus in respiratory samples [1]. However, there are factors that hinder the application of this test, such as the genetic material of the virus becomes detectable generally in the first four or eight days after the onset of symptoms, and undetectable after fourteen days [1]; and due to the high complexity of the test, not all centers have suitable infrastructure and technical expertise to perform [1, 2].

In general, different methods of artificial intelligence have successfully been applied in diverse medical research areas [3]. Medical images and videos have been adopted as a relevant information source for medical diagnostic purposes, in which diverse machine learning algorithms are used to classify images or detect and describe some regions of interest in the analyzed images. Also, different techniques are used to improve image quality [4, 5]. In [6], a denoising filter is designed, which is based on local statistics to improve images quality that is important for providing a proper medical diagnosis of a disease.

The use of radiological medical images, such as chest radiography (x-ray), chest ultrasound and chest computed tomography (CT), has been adopted to assist in the early diagnosis of COVID-19 and to rule out the suspicion of diseases [2, 7], because, in severe cases of the disease, pneumonia is developed [8]. According to [9], the experimental results obtained using CT images were superior than those obtained using x-ray images. In addition, it is important to note that there are several chest CT datasets widely used for research purposes, recently applied in the diagnosis of COVID-19. Some studies demonstrated that chest CT images show some evidences of the development of COVID-19 in patients, as bilateral changes with ground-glass opacity with or without consolidation and interlobular septal thickening [10]. The diagnosis of COVID-19 in symptomatic adult patients, through chest CT images, reaches 90% of the chances of positive evolution of the disease, that is, 90% of sensitivity [10]. According to Jaiswal et al. [11], only CT images cannot be used to identify which virus is causing viral pneumonia. However, when considering these images with the clinical symptoms, travel history and identification of contact with infected people, it can be verified that it is the SARS-CoV-2 [11, 12].

However, during an outbreak of a disease that causes breathing problems, radiologists may be overwhelmed with medical image analysis [2, 8, 13]. In this context, applying different machine learning models [14–17] during a pandemic is very useful, as it performs the automatic analysis of medical images. Deep learning models are the state of the art in identifying COVID-19 and other lung diseases by imaging [2, 11, 18–22]. In [23], a novel deep convolutional neural network method, named VGG16-T, is used to identify lung cancer in early stage using CT images.

In [24], the authors propose a method of deep learning called deep explainable neural networks (xDNN), which is efficient in terms of time and consumption of computational resources. In the first stage of the method, the authors applied a fully connected network layer visual geometry group-16 (VGG-16) [25] to extract the vectors of global characteristics of the images. The results obtained using a SARS-CoV-2 CT-scan dataset [26] and considering accuracy, F1 score and area under the ROC curve (AUC) were 0.9738, 0.9731 and 0.9736, respectively. In [7], authors proposed a model named PatchShuffle stochastic pooling neural network (PSSPNN) for classification between COVID-19, secondary pulmonary tuberculosis, community-captured pneumonia, and healthy subjects. This method is able to show the explainable heatmap, which displays association with lung lesions; their experimental results achieved a microaveraged F1 score of 0.9579.

This work uses transfer learning [27] to extract vectors of global characteristics of image, applying layers of interest from pre-trained convolutional neural networks (CNNs) of the Keras1 application programming interface (API): VGG-16, VGG-19, Inception V3 and EfficientNet-B0. In Pathak et al. [28], experimental results have shown that a COVID-19 classification model built with deep transfer learning achieves better results than other supervised learning models. In addition, the classification models are the result of the application of the xDNN algorithm and Scikit-learn2 API algorithms: k-nearest neighbor (KNN) and support vector machine (SVM) with linear kernels and radial base function (RBF). The main objective is to analyze the impact of features extracted by CNNs, in the design of COVID-19 and non-COVID-19 classification models. The analysis results are estimated by non-parametric statistical tests, specifically Friedman Aligned Ranks [29] and Finner [30].

The main contributions of this work are summarized as follows:

1. Implementation of different pre-trained CNN models to extract image features and identifying the most appropriated features using Keras API;

2. Implementation of a high precision classification models, which can assist the diagnostic of COVID-19 and non-COVID-19;

3. A comparative statistical analysis of the performance of the COVID-19 and non-COVID-19 classification models designed using image features extracted by CNNs;

4. A framework composed that permits select different combination of pre-trained CNN models and classification models, identifying the best pre-trained CNN and classification models. The implementation code is available to the research community.

Hence, the present research intends to be useful in the classification of CT images that present COVID-19 characteristics. Also, this work presented a detailed methodology in which transfer learning is used, and the best parameters are selected from a pre-trained CNN, and later used as input of a classification algorithm, obtaining high accuracy results. It is worthy to note that the code3 of the solution developed is free available for research purposes.

The article is organized considering the following structure. Section 2 covers studies related to the topic of this work. Section 3 describes the methodology of the objects to create and evaluate the proposed models. In Sect. 4, the results of the model validation are presented and compared. Finally, in Sect. 5, the final considerations of the work are presented.

Theoretical Reference

This section presents the characteristics that help to detect COVID-19 by CT scan and a summary of feature extraction techniques and machine learning applied in this work.

COVID-19 detection based on computed tomography

CT images of the chest with COVID-19 are generated from the emission of radiation beams in the human body [31]. These images show important radiological patterns for the diagnostic of a radiologist and classification based on CT [32, 33]: 1) the initial stage is defined in the period of 0–4 days after the onset of the first symptom, whose ground-glass opacity (GGO) [34, 35] can be seen subpleurally in the lower lobes of the lungs, distributed unilaterally or bilaterally [33]; 2) the progressive stage occurs in the period of 5–8 days with diffuse GGO [36], mosaic paving [37] and consolidation [38] distributed in multiple bilateral lobes [33]; 3) the peak stage appears in the period of 9–13 days with dense consolidation [39] most prevalent [33]; 4) the infection control stage usually occurs after 14 days with the permanence of GGO and the absorption of consolidation and mosaic paving [33].

According to [33], the related works in the literature are divided into three classification tasks: classification of COVID-19 and non-COVID-19; classification of COVID-19 and pneumonia; and classification of the COVID-19 severity assessment. Unlike the first two tasks, the COVID-19 severity assessment classification seeks to identify the degree of sequelae caused by the disease [40, 41], thus, all patients are infected with SARS-CoV-2. The classification of COVID-19 and pneumonia seeks to separate patients infected with SARS-CoV-2 from those with pneumonia caused by another agent [7, 42–50]. The classification of COVID-19 and non-COVID-19 seeks to distinguish patients diagnosed with infection caused by SARS-Cov-2, from those that present characteristics of classic pneumonia, healthy or with a diagnostic of other lung diseases [51–60]. In addition, some works presented different methodologies [7] that are able to detect the lesion of the image indicating its severity.

In this context, this paper applies the classification task of COVID-19 and non-COVID-19. The validation of the built models is based on the SARS-CoV-2 CT-scan data set. This data set includes CT scans of patients infected with SARS-CoV-2 and other lung diseases.

Feature extraction methods for transfer learning

The feature extraction pre-processor defines a set of data with the ability to influence the performance of a classification model [24, 61]. In traditional machine learning, an expert method is applied to extract a vector of features depending on a certain domain. With deep learning, this pre-processing became inherent to the classification algorithm and to extract vectors from abstract features [24, 62, 63].

According to [28], deep learning models are applied in medical imaging systems, extracting features automatically or using pre-trained networks. The transformation of a pre-trained neural network into a feature extraction method occurs after the removal of some layers. The application of this modified network configures a learning transfer process. The transfer of learning is the enhancement of a new machine learning task, using the transfer of knowledge from an analogous task that has already been perfected [64].

The use of pre-trained networks with the data set ImageNet is widespread in the literature [28, 65]. Thus, pre-trained CNNs with this data set are applied to extract feature vectors in the identification of COVID-19. In [66] pre-trained networks were applied to inception V3 and VGG-19 with the wide and deep neural network (WDNN) and both reached 0.9847 and 0.9904 accuracy, respectively. In [26] the VGG-16 network was applied with xDNN and reached 0.9738 accuracy. In [58] it was applied to network EfficientNet with a vote-based approach and reached 0.9899 accuracy of the test. Due to the good performances, these networks were applied in this work.

A description of each network is presented below:

• VGG: is an architecture that uses convolutional layers and the activation function rectified linear unit (ReLU). The VGG-16 and VGG-19 architectures are variations of the VGG with 16 and 19 layers, respectively. The filter dimension of the convolutional layers is 3x3 [25, 67, 68]. Different variations of the VGG network model, suc as 16 and 19, were applied in this work due to the simplicity of this architecture and because they are widely adopted in similar research; for instance, Soares et al. [26] used VGG-16 with xDNN.

• Inception: is a CNN architecture published as GoogLeNet, also known as inception V1 [69, 70]. The architecture was improved, and new versions were presented, such as inception V2 and V3. Inception V2 has added batch normalization layers to streamline training [69, 71]. In relation to inception V3, factoring convolutions were added with larger spatial filters to improve computational efficiency [69, 72]. The module inception is the main building block. The convolutional layers have different dimensions of the kernel, 1x1 and 3x3, and an operation of polling [69]. This architecture was utilized in our experimental tests, it was applied in parallel convolutional layers in its main block, as suggested in [73].

• EfficientNet: is a CNN architecture proposed by Tan and Quoc [74]. This architecture applies a scaling method that uniformly grows all dimensions of depth, width, and resolution efficiently, using a simple composite coefficient. Thus, a family of 8 convolutional networks was created, named EfficientNet-B0 to EfficientNet-B7. The main building block is the mobile inverted bottleneck convolution (MBConv). This architecture has few parameters, so it consumes less computational resources when compared to other CNNs [67, 74]. This architecture was used in our work, because it contains few parameters, and also because it obtained the best performance in the image classification from the ImageNet dataset, being 8.4 $\times$ smaller and 6.1 $\times$ faster on inference, these performance results were obtained in 2019 [74].

Supervised learning algorithms

Supervised learning is a machine learning task. This task uses essential information provided or acquired by human aid [75]. Thus, learning is obtained by classes (categories) identified by human specialists and examples of instances of each class. Classes with their instances make up the data set. The class and instance relationship is maintained in a subset of data called the training set, which is applied to learn a classification function. This classification function is used to classify unlabeled instances [76].

In the literature, different supervised learning techniques are applied in the identification of COVID-19 [24, 77, 78]: convolutional neural network (CNN); support vector machines (SVM); logistic regression (LR); Naive Bayes (NB); linear discriminant analysis (LDA); k-nearest neighbors (KNN); decision tree (DT) and random forest (RF).

In this work, the following algorithms were chosen:

• KNN: is a classifier where learning is by analogy. The training set has n-dimensional vectors with their respective classes, and each vector in this set represents a point in n-dimensional space [79]. The classification decision is based on classes of k-nearest neighbors of a new instance [80].

• SVM: is a binary vector classifier in an n-dimensional space. The objective of SVM is to find a hyperplane in an n-dimensional space that separates the examples into two distinct classes. The support vectors are the examples in the space (points) closest to the hyperplane. The SVM algorithm seeks to maximize the margin between the support vectors and the hyperplane [77, 81].

• xDNN: is a classifier formed by several layers with a clear semantic and functional meaning, according to Plamen and Soares [24]. The use of prototype-based “$IF\cdots THEN$” rules proposed by the authors makes it easier for users to understand how xDNN works. Prototypes are representative training images that help users visualize, understand, and appreciate similarity to other validation images. In other words, the prototypes are local density peaks and empirically derived probability distributions [26]. Thus, xDNN offers an association between statistical learning and reasoning [24].

Methodology

Figure 1 represents the proposed methodology to classify the diagnostic of a patient with suspected COVID-19. The methodology was implemented in python using the Scikit-learn and Keras machine learning algorithms APIs. Below, the steps applied to build the proposed model are presented:

Figure 1.

Proposed methodology to create and evaluate the classification model.

1. Data set: contains images of COVID-19 and non-COVID-19;

2. Feature extraction: CNN’s transformed, compressed, and reduced the information in the images. Thus, the Flatten layers transformed data matrices into vectors. The activation functions compressed (coded and transformed) the data to produce a reduced representation of these. In addition, the convolution and pooling operations layers reduced the dimensionality of the features map. The algorithms in this step return a features vector;

3. Training and test division: the set with feature vectors is randomly divided to perform stratified cross-validation with 10 folds of each model;

4. Machine learning: trains the model to classify the diagnostic of a patient with suspected COVID-19;

5. Classification model: it is the final result obtained in machine learning according to the training set. The samples from the test set, without the target classes, are classified by the model;

6. Diagnostic: it is the result of the classification model, based on an image of the test set with suspected COVID-19. This classification is compared with the target class to measure the performance of each model. Performance is compared in a statistical analysis to identify the best classification models, the best pre-trained CNNs for each model, and the best pre-trained CNN to extract characteristics from images in general.

Data set

The SARS-CoV-2 CT dataset [26] contains 2482 CT images. Of these, 1252 images belong to patients infected with SARS-CoV-2, and 1230 are from patients with other lung diseases. Thus, 50.44% of images belong to the class COVID-19 and 49.46% non-COVID-19. Therefore, the dataset is balanced and did not need any pre-processing.

Feature extraction by pre-trained CNN Models

From the data sets, the pre-trained models pre-processes the data, which the classification algorithms use to build the models and make the classifications. The CNN-based algorithms of the following functions were used to create the extraction models: VGG16, VGG19 and InceptionV3 (IV3) of the module tf.keras.applications; EfficientNetB0 (EB0) of model efficientnet.tfkeras. Two criteria were fundamental in the definition of the feature extraction layers: position closest to the layer that classifies and experimental tests considering the best performance in classification. The types of the layers chosen in the algorithms were:

• FC (Dense) is a fully-connected layer of the VGG16 and VGG19 algorithms. This layer returns a feature vector with 4096 dimensions for each image and is used to extract features [82];

• Flatten: it is a leveling layer of the algorithms VGG16 (VGG16 baseline), VGG19 (VGG19 baseline) and was applied to the block6d_add layer of EfficientNetB0 (EB0). This layer converts the data into a vector [83]. The sizes of the vectors returned from the baseline methods VGG16 and VGG19 were 25088 dimensions. Regarding the EB0 method, the returned vector has 9408 dimensions;

• GlobalAveragePooling2D: this parameter represents a global pooling operation for spatial representation of the data, which is applied to replace the stage of fully connected layers in conventional CNNs. This operation generates a resource map for each class that will be classified [84]. This operation was used to obtain a drastic reduction of the dimensions of the feature maps in the layers, top_activation, mixed9 and mixed10 of the respective methods of the algorithms: EfficientNetB0 (EB0 baseline) and InceptionV3 (IV3 and IV3 baseline). The sizes of the vectors returned for methods IV3 and IV3 baseline were 2048 dimensions. The vector size returned by the EB0 baseline method was 1280 dimensions Table 1.

Table 1.

Summary of the chosen layer of each applied CNN.

CNN	Layer type	Layer Name	Output Size
VGG16	FC (Dense)	FC2	4096
VGG16 baseline	Flatten	flatten	25088
VGG19	FC (Dense)	FC2	4096
VGG19 baseline	Flatten	flatten	25088
IV3	GlobalAveragePooling2D	mixed9	2048
IV3 baseline	GlobalAveragePooling2D	mixed10	2048
EB0	Flatten	block6d_add	9408
EB0 baseline	GlobalAveragePooling2D	top_activation	1280

Models

The classification models were generated using data obtained from the feature extraction. Three machine learning algorithms were used to create the classification models: KNeighborsClassifier of the module sklearn.neighbors; SVC module of sklearn.svm; and xDNN4. In general, the standard configuration parameters of the algorithms have not been changed, except for SVC which was applied two kernels, linear and RBF.

Classification measures

The applied measures belong to the module sklearn.metrics and were implemented in python. The metrics of precision (PR), sensitivity (SE), specificity (SP), F1 scores and their respective macro-averages, use the values of the variables in the confusion matrix of each model. Figure 2 presents a confusion matrix.

Figure 2.

Confusion Matrix.

The values of the variables true positives (TP) and true negatives (TN), quantify the hits. Errors are quantified by the values of the variables false positives (FP) and false negatives (FN).

The metrics are defined in equations. PR calculates the probability of patients with positive results, actually having the disease [85]. The SE calculates the probability of a positive outcome of the disease [86]. SP, on the other hand, calculates the probability of negative results in patients without the disease [86]. The F1 score combines the values of precision and sensitivity, it balances the relative importance of each metric [76]. Finally, the macro averages of the metrics calculate their respective averages across classes [76, 87]. Thus, Eqs. 5, 6, 7, and 8 define the macro precision, macro sensitivity (macro recall), macro specificity, and macro F1 score, respectively.

$$\begin{array}{c}\hfill PR=\frac{\mathit{TP}}{TP+FP}\end{array}$$ 1

$$\begin{array}{c}\hfill SE=\frac{\mathit{TP}}{TP+FN}\end{array}$$ 2

$$\begin{array}{c}\hfill SP=\frac{\mathit{TN}}{TN+FP}\end{array}$$ 3

$$\begin{array}{c}\hfill F1=\frac{2TP}{2TP+FN+FP}\end{array}$$ 4

$$\begin{array}{c}\hfill P{R}_{(Macro)}=\frac{1}{C}\sum _{i=1}^C\frac{\mathit{TP}}{(TP+FP)}\end{array}$$ 5

$$\begin{array}{c}\hfill S{E}_{(Macro)}=\frac{1}{C}\sum _{i=1}^C\frac{\mathit{TP}}{(TP+FN)}\end{array}$$ 6

$$\begin{array}{c}\hfill S{P}_{(Macro)}=\frac{1}{C}\sum _{i=1}^C\frac{\mathit{TN}}{(TN+FP)}\end{array}$$ 7

$$\begin{array}{c}\hfill F{1}_{(Macro)}=\frac{1}{C}\sum _{i=1}^C\frac{2TP}{(2TP+FP+FN)}\end{array}$$ 8

Validation of models and comparative statistical analysis of results

In the validation of the models, the set of features was divided into training and test data by the class StratifiedKFold of the module sklearn.model_se-lection, to apply stratified cross-validation. Stratified cross-validation guarantees the same class distribution in the evaluated sets. This validation divides the set of features into k subsets (folds). Subsets are used as sets of training and testing features, for each k different models.

The results of the F1 macro-average of the models, in each cross-validation fold, were analyzed on the Web platform Statistical Tests for Algorithms Comparison (STAC) [88]. Due to the 32 models created and considering the 10 result values of each, the application Assistant estimated a non-parametric test of aligned rank. Thus, the statistical test Friedman Aligned Ranks was applied with a level of significance $\alpha =0.05$, verifying the null hypothesis that the averages of the results of the models are the same. This hypothesis was rejected and then the post-hoc test was applied Finner with a level of significance $\alpha =0.05$, verifying the null hypothesis that the average results of each pair of models compared are equal. The level of significance, denoted by $\alpha =0.05$ indicates a 5% of risk to reject the null hypothesis when it is true. It means that there is a 5% of probability of predicting that the average performances of the algorithms are different when there is not a real difference [89].

Assessment and Results

In this section, the descriptions of the experimental evaluation and results of the pre-trained models of the classification of COVID-19 and non-COVID-19 will be presented, with the SARS-CoV-2 CT set. The objective is to validate the ability to generalize classification models with stratified cross-validation and to identify the best classification model in the applied data set. The experiments measured the performance of the proposed models using the confusion matrix, the metrics of precision, sensitivity, specificity, F1 score and their respective macro-averages, for each cross-validation fold. The average performances of the models in each macro-average and the result of their comparison, referring to the average of the F1 score, are shown in the Table 2.

Table 2.

Average performance of models in the classification of COVID-19 and non-COVID-19 - Cross-validation.

Extractor	Model	Macro averages for each metric				Rank
		PR	SE	SP	F1
EB0	SVC RBF	0.9856	0.9853	0.9853	0.9853	a
EB0	SVC linear	0.9855	0.9853	0.9853	0.9853	a
VGG16 baseline	SVC linear	0.9815	0.9813	0.9813	0.9813	ab
EB0	xDNN	0.9794	0.9787	0.9787	0.9788	abc
VGG19 baseline	SVC linear	0.9741	0.9737	0.9737	0.9737	abcd
EB0	KNN	0.9704	0.9696	0.9696	0.9697	abcde
VGG16 baseline	xDNN	0.9670	0.9661	0.9661	0.9662	abcde
VGG19 baseline	KNN	0.9655	0.9651	0.9651	0.9652	abcdef
VGG19	xDNN	0.9649	0.9646	0.9646	0.9647	abcdef
VGG16	xDNN	0.9630	0.9620	0.9620	0.9621	bcdefg
VGG16 baseline	KNN	0.9606	0.9600	0.9600	0.9601	bcdefgh
VGG19 baseline	xDNN	0.9612	0.9599	0.9599	0.9601	cdefgh
IV3	KNN	0.9605	0.9601	0.9601	0.9601	cdefgh
EB0 baseline	xDNN	0.9603	0.9590	0.9590	0.9591	defgh
VGG16	SVC RBF	0.9596	0.9585	0.9585	0.9586	defgh
VGG16	SVC linear	0.9581	0.9570	0.9570	0.9571	defgh
VGG19	SVC RBF	0.9538	0.9531	0.9531	0.9530	efgh
VGG16	KNN	0.9529	0.9519	0.9519	0.9520	efgh
IV3	xDNN	0.9517	0.9509	0.9509	0.9510	efgh
VGG19	SVC linear	0.9515	0.9506	0.9506	0.9505	efgh
VGG19	KNN	0.9483	0.9481	0.9481	0.9480	fgh
EB0 baseline	KNN	0.9461	0.9448	0.9448	0.9450	fgh
EB0 baseline	SVC linear	0.9428	0.9424	0.9424	0.9425	ghi
IV3 baseline	KNN	0.9353	0.9350	0.9350	0.9349	hi
EB0 baseline	SVC RBF	0.9310	0.9305	0.9305	0.9304	ij
IV3 baseline	SVC linear	0.9253	0.9238	0.9238	0.9238	ij
IV3 baseline	xDNN	0.9182	0.9172	0.9172	0.9172	ij
IV3 baseline	SVC RBF	0.9167	0.9159	0.9159	0.9157	ij
VGG19 baseline	SVC RBF	0.9110	0.8938	0.8938	0.8932	ij
IV3	SVC linear	0.8952	0.8934	0.8934	0.8929	ij
IV3	SVC RBF	0.8952	0.8934	0.8934	0.8929	ij
VGG16 baseline	SVC RBF	0.7903	0.6317	0.6317	0.5747	j

The averages of the F1 macro with the same letters do not differ at the probability level of 5% by the Finner test

The best performances in the F1 score are with the letter “a”

The worst performances in the F1 score are with the letter “j”

The highest averages for each metric are in bold

The Friedman Aligned Ranks test showed that the average of the F1 macro-average differs between the classification models with p-$value<0.001$. Models with the same letters present the same averages in the F1 macro-average, at the probability level of 5% by the Finner test. The models that achieved the best performances are identified with “a”. So, in descending order, the worst models are with the letter “j”. This assessment identified the best CNNs to build each classification model. The best CNNs to build the best KNN models were EB0 and VGG19. Regarding the best SVC models with linear kernel, they were obtained with features provided by EB0, VGG16 baseline and VGG19 baseline. The best xDNN models were built with features provided by EB0, VGG16 baseline and VGG19. The best SVC model with RBF kernel was obtained with EB0 features. Therefore, EB0 is the best algorithm to extract features for the classifiers, as it was the only one that managed to achieve the best performances with each classification algorithm.

In absolute values, the highest performance was for the model with EB0 and SVC RBF, which reached 0.9856 precision and 0.9853 sensitivity, specificity, and F1 score. The lowest performance was the model with VGG16 baseline and SVC RBF, which reached 0.7925 precision, 0.6382 sensitivity and specificity, and 0.5856 F1 scores.

Figure 3 and Table 3 results by cross-validation fold are presented to verify the best performance of the model with EB0 and SVC RBF. The performances measured in the metrics demonstrate that the model can infer a correct diagnosis in both classes of each cross-validation fold.

Figure 3.

Confusion matrices of the model with EB0 and SVC RBF - Cross validation.

Table 3.

Performance of the model with EB0 and SVC RBF in the classification of COVID-19 and non-COVID-19 by cross-validation fold.

Fold k	PR	SE	SP	F1	Category
1	0.9900	0.9801	0.9897	0.9850	covid
	0.9797	0.9897	0.9801	0.9847	non-covid
	0.9848	0.9849	0.9849	0.9849	general
2	0.9708	1.0000	0.9696	0.9852	covid
	1.0000	0.9696	1.0000	0.9846	non-covid
	0.9854	0.9848	0.9848	0.9849	general
3	0.9708	1.0000	0.9696	0.9852	covid
	1.0000	0.9696	1.0000	0.9846	non-covid
	0.9854	0.9848	0.9848	0.9849	general
4	1.0000	0.9800	1.0000	0.9898	covid
	0.9801	1.0000	0.9800	0.9900	non-covid
	0.9900	0.9900	0.9900	0.9899	general
5	0.9900	1.0000	0.9897	0.9950	covid
	1.0000	0.9897	1.0000	0.9948	non-covid
	0.9950	0.9948	0.9948	0.9949	general
6	0.9801	0.9900	0.9795	0.9850	covid
	0.9896	0.9795	0.9900	0.9846	non-covid
	0.9849	0.9847	0.9847	0.9848	general
7	0.9708	1.0000	0.9693	0.9852	covid
	1.0000	0.9693	1.0000	0.9844	non-covid
	0.9854	0.9846	0.9846	0.9848	general
8	0.9900	1.0000	0.9897	0.9950	covid
	1.0000	0.9897	1.0000	0.9948	non-covid
	0.9950	0.9948	0.9948	0.9949	general
9	0.9607	0.9800	0.9591	0.9702	covid
	0.9791	0.9591	0.9800	0.9690	non-covid
	0.9699	0.9695	0.9695	0.9696	general
10	0.9800	0.9800	0.9795	0.9800	covid
	0.9795	0.9795	0.9800	0.9795	non-covid
	0.9797	0.9797	0.9797	0.9797	general

The category values general are the macro-average for each metric

The matrices present a high frequency of correctness in the first diagonal, composed of the frequencies of the correct identifications of COVID-19 (TP) and non-COVID-19 (TN). Each matrix in the Fig. 3 was generated by package functions matplotlib.pyplot in Python.

The framework proposed using the model with EB0 and SVC RBF a-chieved similar performance to those obtained by the works and reported in [26, 90–92]. In [90] a cross-validation with ten folds was used, as well as this work, and applied a method based on generative adversarial network (GAN), obtaining 0.9879 of F1 score. The other works used different methods to analyse the data obtained in relation to the considerations applied in this work. In [26], the model with VGG-16 and xDNN achieved a performance of 0.973 in the F1 score. In [91], an optimized convolutional neural network model, called ADECO-CNN, reached 0.996 of sensitivity, 0.992 of precision, and 0.997 of specificity. In [92], a semi-supervised shallow neural network structure, called Parallel Quantum-Inspired Self-supervised Network (PQIS-Net), achieved a performance of 0.948 considering the F1 score. It is worthy to note that some recent methods also have the additional advantage of identifying the region of lesion and its severity using a heatmap [7].

Conclusion

This work implemented pre-trained CNNs to extract the vectors of global features from CT images. The vectors were used to design classification models for COVID-19 and non-COVID-19. The Algorithms of CNNs applied were VGG16, VGG19, InceptionV3 (IV3) and EfficientNetB0 (EB0). The classification algorithms applied were KNeighborsClassifier (KNN), SVC and Explainable Deep Neural networks (xDNN).

In the experimental evaluation, the performances of each model were compared in a statistical analysis using the Friedman Aligned Ranks and Finner tests. This analysis identified the best classification models, the best pre-trained CNNs to extract characteristics from images for each model, and the best pre-trained CNN in general. Thus, the KNN classifier achieved the best performance with EB0 and VGG19. The SVC linear classifier obtained the best performance with EB0, VGG16 baseline, and VGG19 baseline. The SVC RBF classifier achieved the best performance with the features extracted by the EB0 algorithm. Finally, the XDNN classifier performed best with the features provided by EB0, VGG16 baseline, and VGG19. The SVC RBF with data extracted by EB0 was the model that obtained the highest average performances: 0.9856 in the precision macro, 0.9853 in the sensitivity macro, 0.98553 in the specificity macro, and 0.9853 in the F1 score macro. In this scenario, the EB0 algorithm is the best to extract features for the classifiers, as it was the only one that achieved the best performances in each classification model.

Therefore, the use of algorithms based on transfer learning to extract characteristics from CT images was beneficial to identify COVID-19 and non-COVID-19, in the cross-validation tests. With the exception of one model, the models achieved excellent results. The inferiority of the model with VGG16 baseline and SVC RBF can be attributed to the mapping to another larger space, carried out by the kernel. Another justification is the lack of variation in the parameters of the classification algorithm. In addition, the better performances of the models with EB0, when compared with the others, can be justified by the lack of variation in the parameters of the feature extraction algorithms.

In future works, the parameters variation of the feature extraction and classification algorithms will be carried out to try to reach the maximum performance in the identification of COVID-19 and non-COVID-19. In addition, the same approach will be applied to classify the stages of COVID-19. In addition, different methods to improve image quality will be used in future experiments.