McS-Net: Multi-class Siamese network for severity of COVID-19 infection classification from lung CT scan slices

Abstract

Worldwide COVID-19 is a highly infectious and rapidly spreading disease in almost all age groups. The Computed Tomography (CT) scans of lungs are found to be accurate for the timely diagnosis of COVID-19 infection. In the proposed work, a deep learning-based P-shot N-ways Siamese network along with prototypical nearest neighbor classifiers is implemented for the classification of COVID-19 infection from lung CT scan slices. For this, a Siamese network with an identical sub-network (weight sharing) is used for image classification with a limited dataset for each class. The feature vectors are obtained from the pre-trained sub-networks having weight sharing. The performance of the proposed methodology is evaluated on the benchmark MosMed dataset having categories zero (healthy control) and numerous COVID-19 infections. The proposed methodology is evaluated on (a) chest CT scans provided by medical hospitals in Moscow, Russia for 1110 patients, and (b) case study of low-dose CT scans of 42 patients provided by Avtaran healthcare in India. The deep learning-based Siamese network (15-shot 5-ways) obtained an accuracy of 98.07%, the sensitivity of 95.66%, specificity of 98.83%, and F1-score of 95.10%. The proposed work outperforms the COVID-19 infection severity classification with limited scans availability for numerous infection categories.

1. Introduction

In late 2019, the novel Corona Virus Infection Disease (COVID-19) is detected in humans and declared a pandemic by World Health Organization (WHO) on 11 March 2020 [1]. The COVID-19 belongs to the virus family of Severe Acute Respiratory Syndrome-Corona Virus 2 (SARS-CoV2) and Middle East Respiratory Syndrome-Corona Virus (MERS-CoV). In the end of July 2021, globally there were 1,94,080,019 active cases of coronavirus, 4,162,304 deaths reported, and 3, 696,135,440 people were vaccinated [2]. In the year 2021, researchers have claimed the presence of numerous mutants of COVID-19 such as B.1.1.7, B.1.351, and P.1, etc. [3], [4]. The mutation of coronavirus is having a huge impact on the transmission scale of infection in humans, the efficacy of testing kits, vaccination, and treatment procedures [5].

In the year 2021, the highest number of coronavirus cases are found in India with an average of 0.35 million cases per day in April [6], [7]. The number of cases rapidly increased due to community transmission and the average time duration of the RT-PCR test result took 4–5 days with a very high false-negative rate. The coronavirus majorly affected the respiratory system, i.e., breathing issues, and severe lung infection. Thus, chest CT scans are used to evaluate the severity level of coronavirus infection in suspected patients. The COVID-19 infection spreads through saliva or cough droplets of an infected person through the air. Thus, with the increment in COVID-19 cases, researchers are focusing on faster and accurate techniques for severity of COVID-19 infection measurements in lungs.

In clinical practice, COVID is detected using swab tests, namely, RT-PCR, Rapid Antigen Test (RAT), and radiographic images of lungs (X-rays, CT-scans). The COVID-19 antigen tests identify the antibodies of coronavirus. However, it suffers from various disadvantages such as the complex procedure of taking samples from the nose/throat, time consumption in the generation of the diagnosis result, and a high false-negative rate [8]. The antigen kits are customized for only COVID-19 infection detection and are unable to evaluate the impact of infection levels in the respiratory system or various organs. To overcome these issues, clinicians prefer a chest Computed Tomography (CT) scan/X-ray scan of COVID-19 infected patients.

The chest X-ray of COVID-19 suspected patients provides a preliminary diagnosis of the dense tissues and the impact of infection in the lungs in a 2-D view [9]. However, in chest CT scans, patches due to infection are visible in 3-D views (axial, coronal, sagittal). Therefore, to speed up the COVID-19 infection detection process, radiologists prefer chest CT scans over X-rays. Doctors differentiate the healthy control CT scan from the coronavirus infection patient’s CT scan measurements based on consolidation, pleural effusion, and Ground Glass Opacity’s (GGOs) in the lungs as shown in Fig. 1. Further, the measurements of these patches in the left and right lung predict the severity of coronavirus infection. However, manual annotation of the patches due to COVID-19 infection is an error-prone and tedious task.

Fig. 1.

Biomarkers are identified by the experts in the chest CT scan slice of the COVID-19 infected patient.

In recent times, researchers focused on the development of a future prediction model for COVID-19 cases variants, and improvements in the available COVID-19 measurement equipment as well [10], [11]. In [12], low dose CT scan measurements are suggested for the severity of COVID-19 infection detection in mild COVID-19 infected patients along with children and pregnant women CT scans. The COVID-19 cases are predicted using time series models such as Eigenvalue Decomposition of Hankel Matrix (EVDHM) along with the Auto-regressive Integrated Moving Average (ARIMA) model [13], Composite Monte Carlo (CMC) simulation [14]. To monitor the lung condition of critical covid patients on ventilators, a data acquisition system, i.e., Electrical Impedance Tomography (EIT) system is proposed [15].

The major challenges associated with COVID-19 infection severity measurement using CT scan-based modality are (a) limited dataset is available with expert annotation, (b) multi-class classification of the level of COVID-19 infection is not given, etc. Thus few-shot learning frame-work is introduced for classification on a small sample dataset or no labeled dataset [16]. The Few-shot learning approach include metric-learning and meta-learning-based techniques. In the computer vision field, a meta learning-based approach is introduced for face identification [17], signature verification [18], and a variety of plant leaves classification tasks, etc. [19]. In medical field, numerous disease detection task with limited dataset availability is resolved used few-shot learning based Siamese networks [20], [21], [22]. The Meta-COVID networks (2-ways, 10-shot) are implemented for binary classification of COVID-19 and healthy X-ray images with an accuracy of 96.5%, specificity of 98.4%, and precision of 98% [23].

The main contributions of the proposed work are as follows:

1. The deep learning encoder-based Siamese network is proposed for the multi-class classification of COVID-19 infection from lung CT scan slices.

2. The P-shot N-ways (N denotes the number of classes, and P denotes the images per class) based meta-learning framework evaluated using contrastive loss.

3. The Gradient-weighted Class Activation Mapping (Grad-CAM) is used to visualize the multi-class infection features learned by the proposed McS-Net.

4. The proposed methodology is implemented on the Indian dataset of suspected COVID-19 patients with numerous infection levels.

5. Performance evaluation parameters such as sensitivity, specificity, accuracy, F1-score, precision, etc. are obtained for the considered datasets with numerous variations in the encoder network and some samples per class prove the superiority of the proposed framework.

The rest of the article is organized as follows: Section 2 presents the related work; Section 3 puts forth the brief details of the COVID-19 based CT scan dataset. Section 4 illustrates the materials and methods; Section 5 presents the experimentation and discussion. Section 6 concludes the proposed work along with the future scope.

2. Related work

Various Artificial Intelligence (AI) based techniques are implemented by researchers to auto-mate the COVID-19 infection detection, classification, segmentation, and future prediction task. In [24], X-ray modality is used for multi-class classification of COVID-19 infection with an accuracy of 97.72 ± 0.95% for severe infection, 86.90 ± 3.20% for moderate level, and 61.80 ± 5.49% for mild COVID-19 infection. The Internet of Things (IoT) based pre-trained deep learning model detected the COVID-19 with a sensitivity of 96.73% [25]. The IoT-based approach using a blood sample, X-ray images, body temperature of the covid suspected patient obtained the specificity of 97.95%. Gen-ProtoPNet based deep learning model obtained the classification accuracy of 87.27% on chest X-rays for three classes (normal, COVID-19, and Pneumonia) [26]. The ResNet-SVM obtained a classification accuracy of 93% and sensitivity of 88% on X-ray images of COVID-19 patients [27]. The Genetic Algorithm (GA) with specific hyperparameter optimization technique is used for binary classification of COVID-19 infection [28]. However, the technique takes more than 25 h to get the best results due to the specific optimization parameters requirement. Table 1 put forth the brief details of state-of-the-art techniques available for COVID-19 diagnosis using radiographic images.

Table 1.

Performance evaluation of AI-based techniques available in the literature for COVID-19 diagnosis using chest CT scan.

Sr. No.	Techniques	Augmentation	Dataset	Accuracy	Sensitivity	Specificity
1	7-layer CNN＋stochastic pooling [29]	14-ways	Normal = 320 COVID-19 = 320	94.03%	94.44%	93.63%
2	Joint Classification Segmentation (JCS) system [30]	Image mixing, random horizontal flip, and crop	Normal = 350 COVID-19 = 400	–	95.0%	93.0%
3	Few-shot COVID-19 (2- ways, 6-shot) [31]	Stochastic augmentation with random cropping (with resizing and color distortion)	Normal = 397 COVID-19 = 349	88.5%	88.6%	89.9%
4	Attention-based VGG-style network for COVID-19 (AVNC) [20]	15-ways multiple ways on original and Mirror image	Normal = 284 COVID-19 = 306	–	97.38%	97.06%
5	Truncated VGG-16 [32]	Rotation, shearing, zooming, and brightness	Normal = 358 COVID = 344	95.7%	96.3%	94.8%
6	ResNet＋DWT (Discrete Wavelet Transform) [33]	Scaling (0.85, 1.2), rotation (−20, 20), flipping, shearing (0, 50) and translation (−35, 35) in the $x$ and $y$ directions	OMNIAHCOV* Normal = 5152 COVID-19 = 6012 SARS-COV-2** Normal = 1230 COVID-19 = 1252	99.23%* 99.62%**	99.27%* 99.54%**	99.19%* 99.69%**

In [34], combination of transfer learning and shallow learning-based approach is used to detect the severity of the COVID-19 infection. The proposed method implemented on cloud-based server detects the binary class of COVID-19 infection severity. Hybrid system of the Sugeno fuzzy integral with ensemble of four pre-trained deep learning model (Squeeze Net v1.1, wide ResNet-50-2, VGG-11, and Google Net) is used for binary classification of COVID-19 [35]. In [36], customized DenseNet201-based models are used for COVID-19 diagnosis with an accuracy of 98.18%, a precision of 97.76%, and the specificity of 98.17%. Further, joint classification and regression method is used for binary classification of severity of COVID-19 infection [37]. In [38], fully automatic approach obtained a global accuracy value of 97.06%. In [39], ResNet50 with majority voting-based model is implemented for COVID-19 classification. The fine-tuning based ResNet50 model obtained Area Under the Curve of 0.90 and 3D-ResNet50 model obtained 0.67 of AUC.

In [40], two-stage deep learning based framework is proposed to discriminate between COVID-19 infection from normal pneumonia using chest CT scans. Transfer learning-based approach is implemented on chest X-ray images to detect COVID-19 [41]. For dataset-1, MobileNet+ support vector machine (SVM) classifier obtained a F1-score of 98.5% and DenseNet201+multi-layer perceptron (MLP) obtained a F1-score of 95.6% for dataset-2. The one shot [42] technique with augmentations of the classification branch in the RoI layer obtained sensitivity of 93.55% for COVID and 99.37% of F1-score. In [43], 5656 X-ray images of each healthy and covid patient is classified using VGG network. The proposed model obtains an accuracy of 96.41%, sensitivity of 96.6%, and the specificity of 96.2%. The explainable classification module (ECM) is proposed for COVID-19 classification using chest CT scan with an accuracy of 84.83% [44].

The hybrid (SpaSA and CNN) approach is implemented on chest CT scans of COVID 19 dataset [45]. The SpaSA algorithm is used to optimize the different Convolution Neural Network (CNN) and transfer learning hyperparameters. The proposed work obtained an accuracy of 99.74% on binary class (14,486 images) and 98% accuracy on three class (17,104 images) classification framework. The 2D-Empirical wavelet transform (EWT) along-with pre-trained transfer learning-based approach (DenseNet121) is used for binary classification of chest CT scans [46]. The TL＋DenseNet121 approach obtained an accuracy of 85.50%, F1-score of 85.28%, and AUC of 96.60%. In [47], a Parallel Quantum-Inspired Self-supervised Network (PQIS-Net) for automatic segmentation of lung CT images from hybrid dataset. The CNN based model is proposed for multi-class classification of chest CT scans into normal, COVID-19, and pneumonia. The proposed work obtained an average accuracy of 98% [48].

From the state-of-the-art, it can be inferred that chest CT scan-based COVID-19 diagnosis can identify the coronavirus infection with an accuracy of > 90% in comparison to X-rays or antigen-based tests [49]. The COVID-19 infection diagnosis using deep learning-based approach requires a large set of samples for each class and the model needs to be retrained for variation in the sample in a class. Also, the training on a small dataset may lead to overfitting. Further, the disadvantages of the state-of-the- art deep learning-based techniques for COVID-19 infection diagnosis using chest CT scan include (a) multiple levels of severity of the COVID-19 infection are not clear, (b) the COVID-19 infection is difficult to discriminate from certain pneumonia and lung infections, and (c) numerous techniques failed to differentiate the mutations of coronavirus.

3. Dataset

The important rationale of the proposed methodology is the dataset used for COVID-19 infection severity classification. This section discusses the details of the dataset used based on joint analysis of expert radiologist. In the proposed work, the multi-class classification framework for COVID-19 infection is inspired by the Coronavirus disease 2019 (COVID-19) Reporting and Data System (CO-RADS) [50] and CT severity scores. The CO-RADS score range is based on the patches available in the left lung and right lung [51], [52]. The CO-RADS annotation are as follows: (a) Healthy Control (HC) = CO-RADS-1, (b) less infection = CO-RADS-2, (c) Intermediate infection = CO-RADS-3, (d) High abnormalities suspicious of COVID-19 = CO-RADS-4, (e) Very high abnormality of COVID-19 = CO-RADS-5 and (f) PCR＋ = CO-RADS-6. The CT severity scores for COVID-19 infection analysis is having global range of 0–5 and global score of 0–25. In the considered MosMed dataset the CT scores are based on affected lung percentage lobes such as (a) 0% = 0 points (CT score 0: CT-0), (b) > 5% = 1 point (CT-1), (c) 5%–25% = 2 points (CT-2), (d) 25%–50% = 3 points (CT-3), (e) 50%–75% = 4 points (CT-4), and (f) > 75% = 5 points (CT-5).

The proposed deep learning-based Siamese networks are evaluated on a chest CT dataset with annotations provided by joint analysis of Expert-1 (E1) and Expert-2 (E2). Fig. 2 put forth the annotation details suggested by the experts for the considered dataset. The considered chest CT scans are provided by medical hospitals in Moscow, Russia for 1110 patients within the age of 18–97 years (duration: 1st of March to the 25th of April 2020) [53]. The chest CT scans with no lung infection are named as zero or healthy control. The chest CT scans are provided in NIFTI format [54]. The SMOTE based slice selection method is used to address the class imbalance issue with a considered dataset [55]. For class zero (healthy control), the first 15 axial CT scan slices are considered. Further for the suspected COVID-19 infected patients, the 15 slices with infection patches are considered by the joint decision of experts, i.e., E1$\cap$E2. However, only 2 patients’ CT scans are available for class CT-4, thus all the slices are considered.

Fig. 2.

Brief details of the chest CT scan from MosMed datasets used in the proposed work.

4. Methodology

This section discusses the detailed hierarchy of the proposed methodology, i.e., pre-processing of the dataset, brief architecture of pre-trained sub-network, contrastive loss function, and nearest neighbor classifier. The schematic architecture of the proposed methodology is shown in Fig. 3.

Fig. 3.

The schematic representation of the multi-class classification framework for COVID-19 infection severity level diagnosis using chest CT scans.

4.1. Pre-processing

In the considered dataset, chest CT scans are available in NIFTI format with a size of 512 × 512 and an average of 36 slices per patient. The input image dataset pre-processing is required to make the input dataset compatible with the proposed training network. The pre-processing pipeline includes the following steps: selection of axial slices, resizing operation, normalization, and contrast enhancement. To reduce the storage space issue and computational capacity, the input images are adjusted to the size of 256 × 256. Consider the input image dataset is ${\mathit{X}}_k$ = [(${\mathit{I}}_1$, ${\mathit{H}}_1$),(${\mathit{I}}_2$, ${\mathit{H}}_2$), …, (${\mathit{I}}_n$, ${\mathit{H}}_n$)], here, ${\mathit{I}}_k$ is input images and ${\mathit{H}}_k$ is respective labels for $k=(1,2,\dots ,n)$. The input images with pixels ‘$i$’ and ‘$j$’ are normalized to the range of [0, 1] and represented as $I_{\mathrm{norm}}$ $(i,j)$ in (1).

$$I_{\text{norm}\left(i,j\right)}=\frac{I_k(i,j)-min\left(I_k(:)\right)}{max\left(I_k(:)\right)-min\left(I_k(:)\right)}$$ (1)

The contrast of the CT scan slices is enhanced using a histogram equalization approach. For histogram equalization for the normalized input image, i.e., $I_{\mathrm{norm}}\left(k\right)$, consider ‘$t$ ₀’ as its cumulative histogram, ‘$tc$’ is the cumulative sum of the histogram. The histogram equalization is based on the transformation to map the grayscale to their new values. The monotonic transformation ‘$M_t$’ is minimized using $tc$ ($M_t$ ($r$)) — $t_0$ for all intensities $r$. Thus, $Ic$ ₁, $Ic$ ₂, …, $Ic$ ${}_n$ are the pre-processed CT scan images with corresponding labels.

4.2. Pre-trained sub-networks

In the P-shot, M-ways, based learning framework, for each infection class, a maximum of 15 images are used for training. Thus, pre-trained deep learning networks trained for 1000 classes classification on ImageNet dataset are considered as the base encoder for the considered dataset [56], [57]. As the depth of layer increases, the number of trainable parameters increases and that leads to complexity for metric-based learning space. Thus, to reduce the time complexity and metric space parameters ResNet-18 based pre-trained deep learning model is used as a base encoder. Fig. 4 put the architecture details of the base encoder network, i.e, ResNet18 for the proposed work. The ResNet18 with skip connection residual network is inspired by the VGG-19 architecture. The customized weight initializer function ‘He’ for CNN layers with ReLU layer is calculated with mean value zero and standard deviation, i.e., ${\mathit{S}}_d$ is defined using 2. The scale of ReLU layer is represented by ‘$b$’ and ‘$Fs$’ is the filter size and ‘$C$’ is number of channels.

$$S_d=\sqrt{\frac{2}{(1+b^2)(F_s\left(1\right)\ast F_s\left(2\right)\ast C)}}$$ (2)

Fig. 4.

The proposed schematic architecture of the deep learning-based sub-network.

The sub-networks in Siamese networks are having two different inputs and prediction scores are obtained based on the similarity of the encoded features as shown in Fig. 5. The Siamese batch pairs are obtained using Algorithm 1. The Euclidean distance is obtained between the obtained embedded vectors and the loss function is used to update the training model.

Fig. 5.

Flowchart of the multi-class classification framework for COVID-19 infection severity level diagnosis using chest CT scans.

4.3. Contrastive loss function

The generalized loss function is used to evaluate the prediction performance of the trained model. The loss value increases if the predicted value ‘${\mathit{y}}_p$’ is different from the true label ‘${\mathit{y}}_t$’. The contrastive loss functions are based on similar and dissimilar pair images. More similar feature embeddings are obtained for the same classes. The Euclidean Distance (ED) metric for the embedded feature values, i.e, $F1$ and $F2$, is obtained using (3). The 0 is an optimum value used to eliminate the undefined Euclidean distance measurement variables and set them to 1e-6.

$$\mathrm{ED}=\sqrt{(\left(sum{(F1-F2)}^2\right)+\Theta )}$$ (3)

Further, the loss function for similar ‘${\mathit{L}}_s$’ and dissimilar pairs ‘${\mathit{L}}_d$’ is calculated using (4), (5).

$${\mathit{L}}_s={\mathit{Y}}_P\ast {\left(\mathit{ED}\right)}^2$$ (4)

$${\mathit{L}}_d=(1-{\mathit{Y}}_P)\ast (max{(\mathit{m}-\mathit{ED,}0)}^2)$$ (5)

The ${\mathit{Y}}_P$ (m) = 1 if $F1(:,i)$ and $F2(:,i)$ are similar images, else ${\mathit{Y}}_P$ (m) = 0. The overall contrastive loss, i.e., CL is calculated using (6).

$$\mathit{CL}=0.5\ast \mathit{sum}({\mathit{L}}_s+{\mathit{L}}_d)$$ (6)

4.4. Prototypical nearest neighbor classification

The prototypical classification framework is based on embedded query points and finding the nearest class prototype as shown in Fig. 4. In the proposed P-shot M-ways classification framework, consider the support set with labeled sample $S_n$ = {$(x_1,y_1),\dots ,(x_{\mathrm{k}},y_{\mathrm{k}})$}, $k=\{1,2,\dots ,n\}$. The prototype of each class is computed in $S$ dimensional space such that $J_p\in {\mathbb{R}}^s$ and embedding function is $f\phi :{\mathbb{R}}^D⟶{\mathbb{R}}^s$ as defined in (7).

$$J_p=\frac{1}{\left|S_n\right|}\sum _{\left(x_{k,}{y}_k\right)\in S_n}f\phi \left(x_k\right)$$ (7)

The distribution over classes for a query point $\beta$ with learnable parameters $\varepsilon$ using (8). The prototype learning is based on minimizing the probability given in (9).

$$d\varepsilon (y=p|\beta )=\frac{exp(-d(f\phi \left(\beta \right),ck))}{\sum _{p^{\prime }}exp(-d(f\phi \left(\beta \right),c{k}^{\prime }))}$$ (8)

$$V\left(\varepsilon \right)=-log\left(d\varepsilon (y=p|\beta )\right)$$ (9)

5. Experimentation and discussion

5.1. System requirements

The deep learning-based Siamese networks used ‘Adam’ optimizer, the initial learning rate is 1e−4, gradient learning factor set to 0.9, mini-batch size is 4, the activation function is ReLU, output feature vector size is 1024D, and weight initializer is ‘He’. For the pre-trained deep learning-based encoder networks, the bias factor is set to 10. The maximum epoch is set to 50 with an early stopping criterion. The experiment is conducted on MATLAB R2021b software installed on Intel core i7 10th Generation processor (64 bit) with NVIDIA GeForce RTX2700 having 8 GB capacity and 16 GB RAM capacity.

5.2. Performance evaluation

To evaluate the performance of the proposed work, numerous performance evaluation parameters are considered such as Accuracy, sensitivity, specificity, AUC, and F1-score. The generalized confusion matrix A for the multi-class classification framework is given 10.

$$A=\left[\begin{array}{ccccc}\hfill A11\hfill & \hfill A12\hfill & \hfill A13\hfill & \hfill A14\hfill & \hfill A15\hfill \\ \hfill A21\hfill & \hfill A22\hfill & \hfill A23\hfill & \hfill A24\hfill & \hfill A25\hfill \\ \hfill A31\hfill & \hfill A32\hfill & \hfill A33\hfill & \hfill A34\hfill & \hfill A35\hfill \\ \hfill A41\hfill & \hfill A42\hfill & \hfill A43\hfill & \hfill A44\hfill & \hfill A45\hfill \\ \hfill A51\hfill & \hfill A52\hfill & \hfill A53\hfill & \hfill A54\hfill & \hfill A55\hfill \end{array}\right]$$ (10)

Here, the true-positive is ‘${\alpha }_p$’, the true-negative is ‘${\alpha }_n$’, the false-negative is ‘${\beta }_n$’, and the false-positive is defined as ‘${\beta }_p$’. Consider the High-Infection (HI) class label, to evaluate its performance evaluation parameters are obtained using 10–14. Here, ${\eta }_1$ $=A11+A12+A13+A14$, ${\eta }_2$ $=A21+A22+A23+A24$, ${\eta }_3$ $=A31+A32+A33+A34$, and ${\eta }_4$ $=A41+A42+A43+A44$.

$${\alpha }_p\left(HI\right)=A55$$ (11)

$${\beta }_p\left(\mathrm{HI}\right)=A51+A52+A53+A54$$ (12)

$${\beta }_n\left(\mathrm{HI}\right)=A15+A25+A35+A45$$ (13)

$${\alpha }_n\left(\mathrm{HI}\right)={\eta }_1+{\eta }_2+{\eta }_3+{\eta }_4$$ (14)

The mathematical expressions for performance parameters are given in 15–20. The Accuracy parameter for the different classes such as Extremely High Infection (EHI), High Infection (HI),Intermediate-Infection (II), Low-Infection (LI), and Zero (Z) is represented as $(A_{\mathrm{EHI}},A_{\mathrm{HI}},A_{\mathrm{II}},A_{\mathrm{LI}},A_{\mathrm{Z}})$, respectively. Here, sensitivity is denoted as $({\mathrm{SE}}_{\mathrm{EHI}},{\mathrm{SE}}_{\mathrm{HI}},{\mathrm{SE}}_{\mathrm{II}},{\mathrm{SE}}_{\mathrm{LI}},{\mathrm{SE}}_{\mathrm{Z}})$, specificity is represented as $({\mathrm{SP}}_{\mathrm{EHI}},{\mathrm{SP}}_{\mathrm{HI}},{\mathrm{SP}}_{\mathrm{II}},{\mathrm{SP}}_{\mathrm{LI}},{\mathrm{SP}}_{\mathrm{Z}})$, Precision measurement is denoted as (Pr${}_{\mathrm{EHI}}$, Pr${}_{\mathrm{HI}}$, Pr${}_{\mathrm{II}}$ Pr${}_{\mathrm{LI}}$ Pr${}_{\mathrm{Z}}$), F1-score is represented as ($\mathrm{F}{1}_{\mathrm{EHI}},\mathrm{F}{1}_{\mathrm{HI}},\mathrm{F}{1}_{\mathrm{II}},\mathrm{F}{1}_{\mathrm{LI}},\mathrm{F}{1}_{\mathrm{Z}}$), and Mathews Correlation Coefficient is expressed as (MCC${}_{\mathrm{EHI}}$, MCC${}_{\mathrm{HI}}$, MCC${}_{\mathrm{II}}$, MCC${}_{\mathrm{LI}}$, MCC${}_{\mathrm{Z}}$).

$$A_{\mathrm{HI}}=\frac{({\alpha }_p+{\alpha }_n)}{({\alpha }_p+{\alpha }_n+{\beta }_p+{\beta }_n)}$$ (15)

$${\mathrm{SE}}_{\mathrm{HI}}=\frac{{\alpha }_p}{({\alpha }_p+{\beta }_n)}$$ (16)

$${\mathrm{SP}}_{\mathrm{HI}}=\frac{{\alpha }_n}{({\alpha }_n+{\beta }_p)}$$ (17)

$${\mathrm{Pr}}_{\mathrm{HI}}=\frac{{\alpha }_p}{({\alpha }_p+{\beta }_p)}$$ (18)

$${\mathrm{F1}}_{\mathrm{HI}}=\frac{2.{\alpha }_p}{(2.{\alpha }_p+{\beta }_p+{\beta }_n)}$$ (19)

$${\mathrm{MCC}}_{\mathrm{HI}}=\frac{({\alpha }_p\ast {\alpha }_n)-({\beta }_p\ast {\beta }_n)}{\sqrt{({\beta }_p+{\alpha }_p)({\beta }_n+{\alpha }_p)({\beta }_p+{\alpha }_n)({\beta }_n+{\alpha }_n)}}$$ (20)

The global performance evaluation parameters, i.e., macro-accuracy (${\mu }_1$), macro-sensitivity (${\mu }_2$), macro-specificity (${\mu }_3$),macro-Precision (${\mu }_4$), macro-F1-score (${\mu }_5$), and macro-MCC (${\mu }_6$) are calculated using 21–26. Table 2 put forth the details of global performance evaluation parameters conducted on 5 random trials conducted on a 15-shot 5-ways multi-class classification framework. The global performance evaluation parameters ensure the stable performance of the deep learning encoder-inspired encoder-based multi-class classification framework. The proposed work on MosMed dataset obtained an average accuracy of 97.48%, specificity of 98.45%, the sensitivity of 94.35%, F1 score of 93.68%, the precision of 93.44%, and MCC of 92.27%.

Table 2.

Global performance evaluation parameters for five trials conducted with the base encoder ResNet18 on the MosMed dataset.

Trial	Performance parameters (%)
	${\mu }_1$	${\mu }_2$	${\mu }_3$	${\mu }_4$	${\mu }_5$	${\mu }_6$
1	97.24	93.86	98.31	92.78	93.05	91.53
2	98.07	95.66	98.83	94.90	95.10	94.03
3	97.99	95.78	98.78	94.86	95.01	93.99
4	97.57	94.18	98.49	93.70	93.88	92.40
5	96.57	92.28	97.88	90.96	91.37	89.40
Average	97.48	94.35	98.45	93.44	93.68	92.27

$${\mu }_1=\frac{(A_{\mathrm{EHI}}+A_{\mathrm{HI}}+A_{\mathrm{II}}+A_{\mathrm{LI}}+A_{\mathrm{Z}})}{P}$$ (21)

$${\mu }_2=\frac{({\mathrm{SE}}_{\mathrm{EHI}}+{\mathrm{SE}}_{\mathrm{HI}}+{\mathrm{SE}}_{\mathrm{II}}+{\mathrm{SE}}_{\mathrm{LI}}+{\mathrm{SE}}_{\mathrm{Z}})}{P}$$ (22)

$${\mu }_3=\frac{({\mathrm{SP}}_{\mathrm{EHI}}+{\mathrm{SP}}_{\mathrm{HI}}+{\mathrm{SP}}_{\mathrm{II}}+{\mathrm{SP}}_{\mathrm{LI}}+{\mathrm{SP}}_{\mathrm{Z}})}{P}$$ (23)

$${\mu }_4=\frac{(\underset{\mathrm{EHI}}{Pr}+\underset{\mathrm{HI}}{Pr}+\underset{\mathrm{II}}{Pr}+\underset{\mathrm{LI}}{Pr}+{\mathrm{Pr}}_{\mathrm{Z}})}{P}$$ (24)

$${\mu }_5=\frac{(\mathrm{F}{1}_{\mathrm{EHI}}+\mathrm{F}{1}_{\mathrm{HI}}+\mathrm{F}{1}_{\mathrm{II}}+\mathrm{F}{1}_{\mathrm{LI}}+{\mathrm{F1}}_{\mathrm{Z}})}{P}$$ (25)

$${\mu }_6=\frac{({\mathrm{MCC}}_{\mathrm{EHI}}+{\mathrm{MCC}}_{\mathrm{HI}}+{\mathrm{MCC}}_{\mathrm{II}}+{\mathrm{MCC}}_{\mathrm{LI}}+{\mathrm{MCC}}_{\mathrm{Z}})}{P}$$ (26)

To evaluate the robustness of the proposed work, the performance parameters are evaluated for the individual classes as well and given in Table 3. It can be inferred that less accuracy and sensitivity is obtained for ‘LI’ class, less specificity and precision is obtained for ‘II’ class. Also, F1-score is 90% and MCC is < 90% for both ‘LI’ and ‘II’ classes.

Table 3.

Performance evaluation parameters as per severity levels of COVID-19 infection in chest CT scans with fine-tuned base encoder ResNet18 encoder on MosMed dataset.

Classes	A	Se	Sp	Pr	F1	MCC
Z	100	100	100	100	100	100
LI	99.16	96.20	100	100	98.04	97.54
II	96.23	98.81	95.69	83.00	90.22	88.42
HI	95.82	83.90	99.72	99.00	90.83	88.65
EHI	98.74	100	98.52	92.31	96.00	95.36

5.3. Grad-CAM based infection visualization

The Grad-CAM technique is used to visualize the strongest activation features map learned by the proposed model. The Grad-CAM is based on the linear combination of weights of channel activation maps with an applied ReLU layer. It evaluates the network prediction using differential outputs (class scores) of the gradients. The activation function ReLU provides the feature map of the region of interest, i.e., (severity of infection) in the specified class. In a proposed multi-class infection classification task, consider the score of class is represented by ‘${\mathit{S}}_r$’ for output ‘${\mathit{Y}}_S$ r ’, and feature maps are denoted by ‘$f$’. The obtained score Map (:, :, $pc$) corresponding to categorical label ‘pc’ is the gradient of the final classification score for the particular class, w.r.t. each feature in the feature layer. The obtained Grad-Map is up-sampled to the size of 224 × 224 size, i.e., input image size. Fig. 6 shows the obtained Grad-MAP overlapped on respective input images to visualize the severity of infection in lungs CT scans of suspected COVID-19 patients.

Fig. 6.

Grad-CAM based severity of infection visualization using features learned by the proposed methodology on MosMed dataset.

5.4. Case study on Indian patients

The chest CT scans of 42 COVID suspected patients are collected in April 2021 for the patient (age 21–85 years). The low dose CT scan measurements are obtained on the multi-detector 16 slice CT Aquilion model from Toshiba. The high voltage circuit generates 60 kW power for a 7.5 MHU X-ray tube. The exposure time is 0.35 s for partial scans and 0.50 s for each full scan measurement. The length of the detector along the $z$-axis is set to 38 mm with 896 channels or columns across 40 rows. The slice thickness range varies from 16 × 0.5 mm to 16 × 1 mm. The average total time taken by the technician for chest CT slices measurements is 10 min for setup creation.

The CT scans are provided in DICOM 3.0 format having size dimensions 512 × 512 and an average of 150 frames are given for each patient. The input images are pre-processed to make them compatible with the proposed methodology. In the Indian dataset, for each patient 15 slices are considered. Further, Fig. 7 presents the performance evaluation details of the proposed work on the Indian dataset. Based on demographic region, an average recovery time period for COVID-19 patients was 14 days which is reduced to 7 days depending on COVID-19 infection severity [58]. In the proposed work, the chest CT scan of the dataset is collected for 30 days. However, the COVID-19 infection severity is detected for chest CT scan of an average 10 day period of COVID-19 detection to quarantine period. The proposed work obtained a precision of 94.0%, specificity of 98.50%, the sensitivity of 94.54%, F1 score of 93.32%, and MCC is 92.08%.

Fig. 7.

Performance evaluation along with infection visualization of the proposed work on follow-up patients from the Indian dataset.

5.5. Ablation study

The proposed methodology is investigated for the following queries: (1) How would different pre-trained CNN models affect the performance of multi-class classification of Siamese based network? (2) What is the impact of variation in P-shots on performance evaluation parameters of ResNet18 base encoder.

Table 4 puts forth the brief details of performance comparative analysis of the Siamese-based sub-networks on the considered dataset. It can be inferred that > 90% accuracy is achieved with the VGG19, DarkNets, and ResNet18. However, DarkNets and VGG-19 have higher trainable parameters that increase the complexity of parameters in metric space for similarity index. Thus, ResNet-18 based pre-trained model is a suitable encoder for the proposed work. The proposed ResNet18 encoder performance is evaluated with variation in N-ways and P-shots-based multi-class classification framework. Table 5 puts forth the details of global performance parameters with 5 classes and a P-shots learning framework. It is concluded that 5 class 15 shot training networks provide the optimum performance on the considered dataset.

Table 4.

Comparative analysis of global accuracy on the training dataset with 5-ways 2-shot Siamese network with different pre-trained encoder networks (Top-3 classification accuracy values are highlighted in bold).

Encoder	Layers	Parameters (in millions)	Size of the input image	${\mu }_1$
ResNet18	71	11.7	224 × 224	92.75%
ResNet50	177	25.6	224 × 224	88.10%
ResNet101	347	44.6	224 × 224	89.23%
DarkNet19	64	20.8	256 × 256	91.20%
DarkNet53	184	41.6	256 × 256	95.83%
DenseNet201	708	20	224 × 224	86.40%
VGG19	47	144	224 × 224	94.80%
Inceptionv3	315	23.9	299 × 299	85.20%
SqueezeNet	68	1.24	227 × 227	84.10%
GoogleNet	144	7	224 × 224	87.20%
Xception	170	22.9	299 × 299	77.78%

Table 5.

Comparative analysis of the P-ways M-shot Siamese network ResNet18 based encoder for multi-class classification of COVID-19 infection severity levels.

Techniques		Performance parameters (%)
P	M	${\mu }_1$	${\mu }_2$	${\mu }_3$	${\mu }_4$	${\mu }_5$	${\mu }_6$
5	15	95.68	91.40	88.64	88.60	89.04	87.06
5	15	97.07	94.58	98.27	92.70	92.75	91.59
5	15	98.07	95.66	98.83	94.90	95.10	94.03

Table 6 put forth the comparative analysis of the proposed methodology with other existing techniques available in literature on the considered MosMed dataset with 5 ways and 15 shots classification framework. The proposed methodology detects the numerous severities of COVID-19 infection with an accuracy of 98.07% in 0.8 seconds per chest CT scan. The few-shot learning with ResNet50 pre-trained network without augmentation obtained the classification accuracy of 87.23% on MosMed dataset in 15 seconds duration per scan [20]. The COVID-19 infection severity is classified with an accuracy of 96.20% in 12 s per scan using DenseNet [59].

Table 6.

Comparative analysis of the proposed techniques with the exiting state-of-the-art methods on considered MosMed dataset for multi-class classification (5 ways 15 shot).

Methods	ReCOV-101 [60]	ResNets＋Majority voting [39]	Deep COVID Detect (DCD) [61]	RF＋GLCM [62]	Proposed
Accuracy	94.9%	98%	74.70%	89.23%	98.07%

5.6. Limitations and future scope

Earlier we have worked on binary classification of chest CT scans of COVID19 patients from normal patients using deep learning-based techniques [63], [32]. For this, numerous statistical data augmentation techniques are implemented. But the earlier reported work is not able to detect the COVID-19 infection severity on smaller dataset. The proposed McS-Nets addresses the issue of COVID-19 infection severity classification with limited data availability in each class. It provides the accurate prediction of the disease in mean time with the minimum number of training samples. However, the proposed work is having certain limitations associated with it: (a) performance parameters are < 90% for the infection level with less than 25% infection or few GGO is visible, (b) it is costly as multiple CT scans are required for follow-up patients (average 2–3 scans during 14 days of tenure of COVID-19 infection first detected).

In future work, the proposed technique is to be implemented in a real-time environment for COVID-19 infection diagnosis. The proposed framework is based on image similarity; thus, techniques need to be explored on various variants of COVID-19 infection. Further, the proposed methodology will be implemented on the following conditions: (a) different variants of COVID-19 infections, (b) blood samples and other clinical parameters will also be used for COVID-19 infection identification. To make the proposed technique more effective, contrastive learning-based segmentation is to be explored. Also, a cloud computing-based framework is to be used for wider clinical applications.

6. Conclusions

The proposed work addresses the COVID-19 severity infection diagnosis using chest CT scans with limited samples available from each class. The COVID-19 infection in lungs is categorized as: 1. Zero — 0% infection (CT-0), 2. Low — < 25% infection (CT-1), 3. Intermediate — 25%–50% infection (CT-2), 4. High — 50%–75% infection (CT-3), 5. Extremely high infection — > 75% infection (CT-4). The proposed work learned different severity of infection features in the lung area in comparison to the best performing deep learning-based encoder network. In the proposed 15-shot 5-ways McS-Net an average of > 95% of accuracy, sensitivity, and specificity is obtained on testing data. The Grad-CAM-based functionality is used to investigate the infection map learned by the proposed networks. Further, the proposed work is evaluated on the Indian dataset with limited CT scan slices in each class and obtains average accuracy of 97.06%, a sensitivity of 94.54%. Thus, the proposed technique is useful in the infection severity identification on the follow-up low dose chest CT scan of the patients.

CRediT authorship contribution statement

Sakshi Ahuja: Conceptualization, Methodology, Software, Writing – original draft. Bijaya Ketan Panigrahi: Project administration, Supervision, Investigation, Visualization. Nilanjan Dey: Project administration, Methodology, Supervision, Investigation, Visualization. Arpit Taneja: Project administration, Medical assistance, Visualization. Tapan Kumar Gandhi: Supervision, Formal analysis, Investigation, Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.