Automatic Diagnosis of Stage of COVID-19 Patients using an Ensemble of Transfer Learning with Convolutional Neural Networks Based on Computed Tomography Images

Abstract

Background:

Diagnosis of the stage of COVID-19 patients using the chest computed tomography (CT) can help the physician in making decisions on the length of time required for hospitalization and adequate selection of patient care. This diagnosis requires very expert radiologists who are not available everywhere and is also tedious and subjective. The aim of this study is to propose an advanced machine learning system to diagnose the stages of COVID-19 patients including normal, early, progressive, peak, and absorption stages based on lung CT images, using an automatic deep transfer learning ensemble.

Methods:

Different strategies of deep transfer learning were used which were based on pretrained convolutional neural networks (CNNs). Pretrained CNNs were fine-tuned on the chest CT images, and then, the extracted features were classified by a softmax layer. Finally, we built an ensemble method based on majority voting of the best deep transfer learning outputs to further improve the recognition performance.

Results:

The experimental results from 689 cases indicate that the ensemble of three deep transfer learning outputs based on EfficientNetB4, InceptionResV3, and NasNetlarge has the highest results in diagnosing the stage of COVID-19 with an accuracy of 91.66%.

Conclusion:

The proposed method can be used for the classification of the stage of COVID-19 disease with good accuracy to help the physician in making decisions on patient care.

Introduction

New coronavirus 2019 (COVID-19) is an infectious disease that has accompanied by significant inflammatory symptoms in the respiratory system. This disease has involved more than 220 million individuals in the world until September 2021.[1] Computed tomography (CT) from the lungs is a fast, efficient, and available imaging modality that can be used for the diagnosis of this disease.[2,3] In this disease, ground-glass opacity (GGO), crazy-paving patterns, and subsequent consolidation patterns can be observed in lung CT images.[4,5] Based progression of this disease with time, there are four stages which include early stage (0–4 days after initiation of disease), progressive stage (5–8 days), peak stage (9–13 days), and absorption-stage (2 weeks after initiation of the disease).[6] There are differences in tissue patterns in these stages which can be observed from the lung CT images.[6] In lung CT images, the early stage is accompanied by GGO, the progressive stage with the crazy-paving pattern, the peak stage with consolidation, and the absorption stage with gradual resolution of consolidation without a crazy-paving pattern. Following that, the consolidation is slowly absorbed.

There are different studies that were focused on time changes in the lung CT images from COVID-19 patients.[7,8,9,10] Diagnosis of the stage of the COVID-19 disease by the use of chest CT can help the physician to predict the length of the time period of hospitalization of patients. This also will help to make proper decisions for the type of treatment and to manage the costs of maintenance and treatment of the patients. The diagnosis and interpretation of the disease patterns and their textural changes in the images require expert radiologists who are not available everywhere and are also tedious and subjective. Therefore, quantitative diagnosis of CT images by the use of machine learning methods can be useful for the early and adequate diagnosis and managing of this disease.

There are different machine learning methods for the purpose of image processing in the diagnosis of diseases in medicine. Among these methods, in recent years, deep learning methods, especially convolutional neural networks (CNNs) models, are proposed as the newest and most advanced methods.[11] However, these models require a large amount of training data. To overcome this problem, transfer learning methods were introduced. In these methods, a neural architecture is used with pretrained weights on a large data set and then applied to the target task with a limited number of training data.[12] In other words, a powerful deep neural network is learned on a number of large datasets for the extraction of comprehensive features and then is applied for a target task with a small-sized dataset. Deep transfer learning methods have been widely applied in medical image processing for the improvement of the performance of the methods.[13,14,15,16,17,18]

Deep learning methods were also applied on CT lung images for the diagnosis of COVID-19 disease or diagnosis of severity of the disease.[19,20,21,22,23,24,25,26,27,28] The results showed that the use of lung CT images and deep learning method has the potential to help to early diagnosis, isolation, and treatment of COVID-19 patients and can help to reduce the workload of health-care staff. Wang et al.[19] developed a weakly supervised deep learning method for the classification of COVID-19 disease and lesion localization using CT images. Their model could accurately predict the COVID-19 disease and localize lesion regions in chest CT images. Gifani et al.[20] introduced an automatic deep transfer learning with different pretrained CNN architectures for the detection of COVID-19 disease based on CT scan images. Chen et al.[21] constructed a deep learning system for the detection of COVID-19 pneumonia and the model showed comparable performance with the experienced radiologists. They concluded that the model can greatly improve the clinical efficiency of radiologists in practical situations. Huang et al.[22] evaluated burden changes in lungs of COVID-19 patients by an automated deep learning method based on CT images. The opacification in the lung CT images which was determined by a deep learning tool was significantly different in different groups with different clinical severities. Shan et al.[23] developed a segmentation system that was based on a deep learning system for automatic quantification of infection regions of interest and their volumetric ratios in the lung. The performance of the presented system was compared with those of manually-delineated systems on 300 CT images of COVID-19 patients. Shorten et al.[24] performed a literature review on the battle of deep learning against the COVID-19 pandemic. In that review, many examples of studies of deep learning systems on COVID-19 were presented. Zhang et al.[25] developed a deep learning integrated radiomics model for the identification of COVID-19 disease by using CT images. The clinical feasibility of the model was validated and the results showed that the model had an adequate performance for identification of this disease and can be a help for screening of suspected cases. Dou et al.[26] performed a study on multinational patients for the detection of COVID-19 abnormalities on CT images. Wang et al.[27] compared feature engineering and deep learning methods for feature extraction from CT images for the prediction of COVID-19 pneumonia prediction. They could successfully determine image features that are significant in COVID-19 pneumonia and this can be useful to increase the performance of humans in the diagnosis of the disease. Yasar and Ceylan[28] used three data augmentation methods and CT lung images for the automatic classification of COVID-19 disease. Amini and Shalbaf proposed texture feature and random forest classifier for assessment of the severity of COVID-19 patients.[29] Li et al. proposed a novel deep-learning framework for the severity assessment by jointly performing lung segmentation and lesion segmentation.[30] Aboutalebi et al. presented a new deep CNN specifically a three-dimensional (3D) residual architecture design which is leveraged to learn volumetric visual indicators for characterizing the severity of COVID-19 patients.[31] However, these mentioned recent studies were only on automatic detection of COVID-19 disease or severity of the disease, and to the best of our knowledge, there is no study on determination of stage of this disease using deep transfer learning methods from the CT images.

The purpose of this study is to diagnose the stage of COVID-19 patients including normal, early, progressive, peak, and absorption stages using an automatic ensemble method of deep transfer learning outputs with different powerful pretrained CNNs from the lung CT images without any need to image segmentation. This proposed assessment system can help physicians in making decisions on the time period required for hospitalization of patients and making better decisions to cure and finally control infections better for COVID-19 patients.

Materials and Methods

The computed tomography image database of patients

Lung CT images were collected for 689 cases (including 410 men and 279 women) suspected of COVID-19 disease in 3D format. Then, we decomposed the 3D CT scan into multiview slices for reducing the complexity of the 3D model. In 3D format, about 20–30 2D CT images are collected with a slice thickness of 10 mm for each case. In other words, the average number of 2D CT images for each patient in a 3D scan CT is 25. The 2D CT images were saved as 512 × 512 pixels images. The images were in the sagittal views and were acquired using a Hispeed CT dual slice scanner (GE Healthcare, USA). The CT scanner was in the Qaboos Teb Golestan Medical Imaging Center (Gonbad-e-Kavous, Iran). Images were acquired in the spiral mode and to avoid motion artifact, a single breath-hold was used. Slice thickness was 10 mm, with 10 mm intervals and a speed of 15 mm/rot.

Chest CT is used to assess the stage of the COVID-19 disease. In lung CT images, the early stage is accompanied by GGO, the progressive stage with the crazy-paving pattern, the peak stage with consolidation, and the absorption stage with gradual resolution of consolidation without a crazy-paving pattern. Following that, the consolidation is slowly absorbed. Diagnosis and labeling of the stage of COVID-19 disease were performed by consensus of two highly experienced radiologists to a normal, early-stage, progressive stage, peak stage, and absorption stage as the reference. The numbers of persons in the normal, early, progressive, peak, and absorption stages were 314, 80, 84, 110, and 101 cases, respectively. Sample CT images of lung slices for a normal case and the four stages are presented in Figure 1. The dataset which is provided in this research can be accessed by other investigators. For this purpose, the download link is as follows: “https://github.com/medical-dataset/CT-Images” which includes the path for Google drive for downloading the dataset of the normal and patients' cases. All procedures performed in this study which involved investigation on the CT images and information of human participants were in accordance with the ethical standards of Shahid Beheshti University of Medical Sciences.

Figure 1.

Lung computed tomography images illustrating histological pattern including (a): normal, (b): early stage (0–4 days) with ground-glass opacities, (c): progressive stage (5–8 days) with increase in the crazy-paving pattern (interloper septal thickening), (d): peak stage (9–13 days) with consolidation, (e): absorption stage (≥14 days) with gradual resolution of consolidation without crazy-paving pattern

Preprocessing of computed tomography images

As we mentioned earlier, the images of the lungs CT of COVID-19 patients were collected in 3D format. In 3D format, about 20–30 2D CT images are required for each patient with a slice thickness of 10 mm. In lung CT analysis, it is necessary to select appropriate 2D images that have the largest lung volume from these 2D images from a patient. The initial and final images of these 2D images have not enough information. For this purpose, in a preprocessing phase, each of these 2D images is evaluated to find the best images that completely contain the part of the lung volume [Figure 2]. For this work, the following operations are performed for each subject. In the first step of preprocessing, for each original 2D image, which is 512 × 512 in size, the central region was extracted with 333 × 333 pixels. The 2D images were then converted to binary images using the method of Otsu[30] and then complemented. In the following, the actual numbers of pixels set to 1 on the binary 2D image in the region (area) were extracted as scalar numbers. These pixels correspond to the part of lung volume in each image. Usually, in the 3D CT scan, the part of lung volume in these 25 of 2D images gradually increases and decreases again. By extracting the maximum curve of the pixels set to 1, we can obtain the 2D images corresponding to the maximum lung volume [Figure 2]. To increase the information, we also used 2D images adjacent to this 2D image. For this purpose, in addition to the 2D image that has the largest lung volume, we automatically extracted the previous two 2D images and the next two from the collection of 2D images of each subject [Figure 2]. Therefore, from about 20–30 2D images in 3D format, five 2D images were selected from each person.

Figure 2.

Preprocessing stages applied on the computed tomography images of a sample patient with the largest lung volume

Transfer learning based on pretrained convolutional neural networks

CNNs are accounted as one of the most popular deep learning methods which are used for the analysis of medical images.[32] In CNNs, by applying multiple building blocks, such as convolution layers, pooling layers, nonlinear layer, and batch normalization, spatial hierarchies of features can be extracted automatically. Finally, fully connected layers prepare extracted features to be classified by the softmax layer. Nonlinear layers (mostly ReLU function) can be utilized to strengthen the network to solve nonlinear problems. There are large numbers of parameters in the CNN models, and therefore, there is a demand for too much training data. To solve this problem, transfer learning methods are used by considering a neural architecture with pretrained weights on a large data set and then applying them to the target task with limited numbers of training data.[11,12] In these methods, a powerful deep neural network is learned on large datasets to extract comprehensive features and then applied or fine-tuned to the target task with the small-sized dataset. There are several special CNN architectures that are trained on very large numbers of images named ImageNet with different categories of animals and objects and then named pretrained CNNs model.[33] Some of popular pretrained CNNs which were utilized in this study are EfficientNets (B0-B5),[34] NasNetLarge, NasNetMobile,[35] InceptionV3,[36] ResNet-50,[37] SeResnet50,[38] Xception,[39] DenseNet121,[40] ResNext50,[41] and Inception_resnet_v2.[42] Some advantages of these networks are lower training time, lower computational time, and less and cheaper hardware requirements.

Ensemble of deep transfer learning method

A block diagram illustrating different processes of the proposed method which was used to determine the stage of COVID-19 disease from CT images is presented in Figure 3. The first step was data preprocessing which was described in detail before. Then, since the sizes of the inputs of the pretrained CNNs were different, all images were resized to be adequate to the sizes of the model inputs. In the following, the proposed deep transfer learning methods introduced in the previous section are applied. The pretrained CNNs were fine-tuned on the chest CT images and then the extracted features were classified by the softmax layer, separately. The pretrained CNNs models which were applied on the lung CT images included EfficientNets (B0-B5), NasNetLarge, NasNetMobile, InceptionV3, ResNet-50, SeResnet50, Xception, DenseNet121, ResNext50, and Inception_resnet_v2. Finally, the ensemble model was created by majority voting between all obtained basic transfer learning models to assign the final prediction.

Figure 3.

Overview of the proposed methods for detection of stage of COVID-19 disease based on computed tomography images by ensemble of deep transfer learning methods using the pretrained convolutional neural network models

To evaluate the performance of the developed system, the test data was used and accuracy, micro and macro precision, recall, and F1-score were selected as common classification metrics. The total dataset was split into three sets, including training, validation, and testing sets which included 80%, 10%, and 10% of the total dataset, respectively. The number of persons and frames in different stages of COVID-19 disease in the training, validation, and testing dataset are presented in Table 1. It should be noted that for each person, five frames are collected. Data partitioning was performed by cross-validation technique. In the fine-tuning of the CNN models, data augmentation was used during training to avoid overfitting. Ten percentage shift was randomly applied in the vertical and horizontal axes of the images to extend their original dimensions. Furthermore, a combination of small random zoom and 20° random rotation was applied to the training images. The size of the dataset was also increased by the horizontal flipping of the images. In the fine-tuning of the networks, all fully connected layers were removed. Furthermore, a global average pooling layer was applied on the top of the last convolutional layer. Then, after the flattening layer, softmax nonlinearity was applied in the final classification layer. For the purpose of fine-tuning the networks, the models were fine-tuned for 50 epochs by optimization of stochastic gradient descent including the following characteristics: an initial learning rate of 0.0001, Nesterov momentum of 0.9, and batch size of 32. In all cases, the categorical cross-entropy was selected as the loss function. Python programming language (version 3.5, Python Software Foundation, Beaverton, Oregon) and Keras software (version 2.1.5, GitHub, San Francisco, California) were used for the purpose of deep learning calculations, and a graphics processing unit (GeForce GTX 1080 Ti, NVIDIA, Santa Clara, California) was also used for these calculations.

Table 1.

Number of persons (frames) in different stages of COVID-19 disease in the training, validation, and testing dataset

Stage	All persons	Train	Valid	Test
Normal	314 (1570)	251 (1255)	31 (155)	32 (160)
Early stage	80 (400)	64 (320)	8 (40)	8 (40)
Progressive stage	84 (420)	67 (335)	8 (40)	9 (45)
Peak stage	110 (550)	87 (435)	11 (55)	12 (60)
Absorption stage	101 (505)	80 (400)	10 (50)	11 (55)
Total	689 (3445)	549 (2745)	68 (340)	72 (360)

It should be noted that for each person, five frames are collected

Results

The results of classification metrics (micro precision, micro recall, micro F1-score, accuracy) for the train and test dataset using deep transfer learning systems are reported in Tables 2 and 3, respectively. The data presented in these tables are based on the different architecture of pretrained CNNs. The accuracy and loss function values in the training sets for fine-tuning of different pretrained CNN models are presented in Figure 4. The validation accuracy and cross-entropy loss function versus a number of epochs for the pretrained CNN models are also presented in Figure 5. As it is shown in these figures, for the training process, the presented model is being converged after the 50^th epoch. Due to this result, after 50 steps of training, the test set was utilized and the diagnosis metrics were calculated. The obtained test results indicated that the NASNetLarge model has higher results than the other models with accuracy amounts of 91.27%. After this model, the EfficientNetsB4 and Inception_resnet_v2 had the highest accuracy with amounts of 89.42% and 88.87%, respectively. To improve the classification accuracy for the determination of the stage of COVID-19 patients, an ensemble model was created using majority voting among all possible basic transfer learning models to assign the final prediction to the test data. The experimental results in Tables 2 and 3 indicated that the ensemble of three pretrained CNN models named NASNetLarge, EfficientNetsB4, and Inception_resnet_v2 improved overall performance compared to each individual basic model and also all other possible ensemble models. The accuracy of this ensemble of three above transfer learning models has the highest results with a value of 91.66% in the test dataset. Micro and macro average ROC of the ensemble method are presented in Figure 6 and also the results of macro average values of the precision, recall, and F1-score of the ensemble method are 89.34, 90.99, and 90.07, respectively. Finally, the confusion matrix was calculated and presented based on the best ensemble model with the most accurate results in Table 4. Hence, this proposed method for classification of the stage of COVID-19 disease can help physicians to better treatment of patients and will have advantages in diagnosis, hospitalization, and cost management of the COVID-19 disease.

Table 2.

The results of classification metrics obtained for the train dataset using deep transfer learning systems based on different architecture of pretrained convolutional neural networks and also proposed ensemble method

	Micro precision	Micro recall	Micro F1-score	Accuracy
EfficientNetB0	91.07	90.94	90.49	92.25
EfficientNetB1	75.08	77.50	75.67	80.62
EfficientNetB2	93.52	92.99	93.09	94.83
EfficientNetB3	92.39	91.97	92.00	93.17
EfficientNetB4	94.44	94.84	94.60	95.75
EfficientNetB5	79.85	79.43	79.06	83.76
ResNet50	85.22	81.20	82.01	86.34
NASNetLarge	97.51	96.62	97.01	97.60
NASNetMobile	91.84	91.22	91.29	92.80
DenseNet121	87.52	89.30	88.05	90.03
Inception_resnet_v2	94.29	93.32	93.68	95.20
InceptionV3	94.04	92.94	93.36	94.46
Xception	93.18	92.12	92.37	94.09
ResNext50	89.91	88.5	88.74	91.69
SeResnet50	92.17	94.00	92.94	93.91
Proposed Ensemble model	98.27	98.27	98.27	98.27

Table 3.

The results of classification metrics obtained for the test dataset using deep transfer learning systems based on different architecture of pretrained convolutional neural networks and also proposed ensemble method

	Micro precision	Micro recall	Micro F1-score	Accuracy
EfficientNetB0	84.74	84.61	84.16	85.92
EfficientNetB1	68.75	71.17	69.34	74.29
EfficientNetB2	87.19	86.66	86.76	88.5
EfficientNetB3	86.06	85.64	85.67	86.84
EfficientNetB4	88.11	88.51	88.27	89.42
EfficientNetB5	73.52	73.1	72.73	77.43
ResNet50	78.89	74.87	75.68	80.01
NASNetLarge	91.18	90.29	90.68	91.27
NASNetMobile	85.51	84.89	84.96	86.47
DenseNet121	81.19	82.97	81.72	83.7
Inception_resnet_v2	87.96	86.99	87.35	88.87
InceptionV3	87.71	86.61	87.03	88.13
Xception	86.85	85.79	86.04	87.76
ResNext50	83.58	82.17	82.41	85.36
SeResnet50	85.84	87.67	86.61	87.58
Proposed Ensemble model	91.94	91.94	91.94	91.94

Figure 4.

The training accuracy and cross-entropy loss function versus number of epochs for the pretrained convolutional neural network models

Figure 5.

The validation accuracy and cross-entropy loss function versus number of epochs for the pretrained convolutional neural network models

Figure 6.

ROC of different stages of COVID-19 disease (class 0: absorption stage, class 1: early stage, class 2: normal, class 3: peak stage and class 4: progressive stage) and also micro and macro average ROC of the ensemble method

Table 4.

Confusion matrix obtained for the best ensemble model with fine-tuning and softmax for different stages of COVID-19 disease (normal, early stage, progressive stage, peak stage, and absorption stage) on the test data set (360 frames)

Reference (diagnosis of radiologist)	Estimated labels by the proposed method
	Normal	Early stage	Progressive stage	Peak stage	Absorption stage
Normal	150	1	7	0	2
Early stage	0	36	2	0	2
Progressive stage	1	2	39	3	0
Peak stage	1	1	1	54	3
Absorption stage	1	1	1	0	52

Discussion

In the present study, an automatic deep transfer learning system was developed which is based on pretrained CNNs for the diagnosis of stage of COVID-19 patients to normal, early stage, progressive stage, peak stage, and absorption stage by the use of lung CT images. The results of classifications showed that the developed deep transfer learning system could be successful to classify different stages of COVID-19 disease with an accuracy of 98.33% by the ensemble of three deep transfer learning outputs named NASNetLarge, EfficientNetsB4, and Inception_resnet_v2. Therefore, the proposed model can help physicians to make a better decision on the time period which is required for the hospitalization of patients and on other decisions regarding the type of treatment of the disease.

Based on the confusion matrices of the proposed algorithm which is presented in Table 4, the accuracy of the classifier was acceptable. As it is evident from these tables, the accuracy of classification for patients in the progressive stage had lower accuracy and in the normal stage had the highest accuracy. The higher accuracy for the normal stage may be due to the fact that the histological patterns in lung tissue are different in this stage, relative to the other stages. However, due to the correlation between the tissue characteristics in the progressive stage with early and peak stages, the accuracy seems to be slightly lower. In the early stage, the lung tissue pattern includes blurriness of ground-glass opacities and some squamous and consolidation tissues. In the progressive stage, the squamous tissue is observed to a higher extent and in the peak stage, the consolidation pattern is being dominant.

Ensemble of basic classifiers can perform better than every single classifier due to reducing variance in the final prediction and is a common way for increasing model generalization and prediction accuracy by utilizing capabilities of each classifier. In this research, an ensemble of deep transfer learning classifiers was built with the architecture of different pretrained CNN networks and the final classification was performed based on majority voting. Classification accuracy of the ensemble of the three deep transfer learning outputs (NASNetLarge, EfficientNetsB4, and Inception_resnet_v2) was improved compared to the single deep transfer learning classifiers [Table 2].

For training the CNN models, a large amount of image data which mostly belong to the general domain (for example cat, dog, and chair domains) are used. However, the lung CT images which were used in the present study had different visual appearances. As a result, the parameters which were learned on these large images could not represent the CT lung images adequately. To overcome this issue, the pretrained CNN models were fine-tuned and modified to be suited to the used dataset. As a general point, it should be mentioned that fine-tuning of pretrained CNN structures has much better results than the conventional training of the CNN model using random weights. The numbers of parameters in the CNN models are very large, and this requires more complex calculations and too much training data. The use of transfer learning methods can compensate for the lack of large datasets and achieving better outcomes. In this method, a powerful deep neural network is learned on large datasets for extraction of comprehensive features and then applied or fine-tuned to the target task with the small-sized dataset. In other words, we transfer the information of a huge database to our problem with an insufficient database with a transfer learning concept.

This study has the advantage of comparison of the deep transfer learning models with powerful pretrained CNNs and ensemble of a deep transfer learning system without any need for image segmentation in the determination of stage of COVID-19 disease. The main limitation of this study is the small number of patients evaluated in this study for the training of deep neural networks. Although by the use of regularization terms and simplification of deep models, it was tried to overcome this problem. Furthermore, the study population was only contained from only one site. Data input from various hospitals could be used to develop and incrementally train the proposed model to predict the stage of COVID-19 patients more precisely in clinical settings.

Conclusion

New ensemble of automatic deep transfer learning systems based on powerful pretrained CNNs proposed in this study to determine the five stages of patients suspected from COVID-19 as normal, early, progressive, peak, and absorption stages from the lung CT images and provide insightful findings without any need for image segmentation. Based on the results, application of the ensemble method based on majority voting of the three deep transfer learning outputs (NASNetLarge, EfficientNetsB4, and Inception_resnet_v2) with fine-tuning and softmax layer has most accurate results than the individual deep transfer learning models with an accuracy metric of 91.66%. Future studies should include larger subjects and a multi-site investigation to validate and improve the results in clinical settings. Furthermore, additional information from the patients such as clinical information, history of other diseases (such as diabetes, obesity, etc.), and the results of other tests of patients such as serological tests may be useful to determine the stage of the COVID-19 disease more accurately. Hence, the presented automated model can help physicians to make a better decision on the time period required for hospitalization of COVID-19 patients and better determination of how to treat them.

Financial support and sponsorship

None.

Conflicts of interest

There are no conflicts of interest.