COVID-19 Severity Prediction from Chest X-ray Images Using an Anatomy-Aware Deep Learning Model

Abstract

The COVID-19 pandemic has been adversely affecting the patient management systems in hospitals around the world. Radiological imaging, especially chest x-ray and lung Computed Tomography (CT) scans, plays a vital role in the severity analysis of hospitalized COVID-19 patients. However, with an increasing number of patients and a lack of skilled radiologists, automated assessment of COVID-19 severity using medical image analysis has become increasingly important. Chest x-ray (CXR) imaging plays a significant role in assessing the severity of pneumonia, especially in low-resource hospitals, and is the most frequently used diagnostic imaging in the world. Previous methods that automatically predict the severity of COVID-19 pneumonia mainly focus on feature pooling from pre-trained CXR models without explicitly considering the underlying human anatomical attributes. This paper proposes an anatomy-aware (AA) deep learning model that learns the generic features from x-ray images considering the underlying anatomical information. Utilizing a pre-trained model and lung segmentation masks, the model generates a feature vector including disease-level features and lung involvement scores. We have used four different open-source datasets, along with an in-house annotated test set for training and evaluation of the proposed method. The proposed method improves the geographical extent score by 11% in terms of mean squared error (MSE) while preserving the benchmark result in lung opacity score. The results demonstrate the effectiveness of the proposed AA model in COVID-19 severity prediction from chest X-ray images. The algorithm can be used in low-resource setting hospitals for COVID-19 severity prediction, especially where there is a lack of skilled radiologists.

Introduction

The novel Coronavirus (COVID-19) pandemic has become a grave public health concern worldwide. According to the World Health Organization (WHO) [1], there were more than 683 million confirmed COVID-19 cases worldwide and 6.8 million deaths as of now in 2023. With the increasing number of patients, the need for streamlined patient management has become extremely important. Disease severity monitoring can be of significant importance for the timely allocation of hospital resources and predicting which patients may need intensive care unit (ICU) support. Biosensors, Artificial intelligence (AI), and the Internet of Medical Things (IoMT) have paved the pathway of modern COVID-19 diagnostic methods [2]. Chest x-ray imaging is a widely available non-invasive tool used for frequent assessment of COVID-19 severity. Although the High-Resolution and Computer-Aided Computed Tomography methods are best suited for COVID-19 severity assessment [3, 4], CXRs can also be useful for regular monitoring due to their relative speed, low cost, availability, and portability, especially in low-resource settings. With the advent of portable equipment, CXR imaging is more reasonable for critically ill patients who cannot be transported for a CT scan.

During the peak of the pandemic, numerous admitted patients endured additional constraints in terms of medical image interpretation in the hospitals. In high-income countries, high patient volumes may result in a higher workload on the existing radiologists, whereas in low-income countries, there may not be a sufficient number of skilled radiologists to perform the necessary image interpretation. In these circumstances, automated analysis of CXRs for the severity prediction of COVID-19 can be of significant value, as the resulting analysis could be potentially used for making decisions regarding escalation or de-escalation of care and monitoring of treatment efficacy.

In recent times, Artificial intelligence (AI) algorithms, particularly deep learning, have demonstrated remarkable progress in the automated interpretation of medical images. During high workloads and in locations where the number of radiologists is limited, automated CXR analysis can play a vital role in efficient and effective patient management. Previous research already revealed that well-trained AI algorithms could perform at or above the level of third-year radiology residents when it comes to identifying various characteristics on radiographs [5]. Thus, effectively trained AI-based algorithms for automated CXR analysis tools can significantly benefit low-resource hospital systems.

Automatic assessment of COVID-19 using CXR images has been an active area of research since the inception of this pandemic [6–12]. Most previous work focuses on the detection of COVID-19 pneumonia from CXR images. However, in hospital settings, automatic detection of the presence of COVID-19 is not of significant value as most admitted patients are confirmed through RT-PCR tests. Methods for automatic assessment of COVID-19 severity can be more valuable as they can be used for monitoring treatment efficacy, especially in the ICU. Therefore, in this work, we focus on the assessment of COVID-19 severity using an existing dataset [13]. In Cohen et al. [13], a COVID-19 pneumonia severity prediction dataset was released including two different disease severity metrics, (i) the Geographic Extent Score, and (ii) the lung Opacity Score. The geographic extent score represents the extent of lung involvement, whereas opacity score signifies the degree of lung opacity.

The main contributions of this paper are as follows:

1. We propose a deep learning-based unified model that integrates anatomy information with the feature extraction scheme to increase the performance of the architecture for COVID-19 severity scoring

2. A regression model that uses the extracted features for COVID-19 severity scoring assesses lung involvement in COVID-19 patients.

3. Experimental evaluation on an available COVID-19 CXR dataset and validation on an in-house annotated CXR dataset proves the efficiency of this method.

4. This model can be incorporated into teleradiology platforms to reduce the number of severe COVID-19 patients by providing early predictive severity scores.

Motivation for Anatomy-Aware Analysis

COVID-19 pneumonia mainly increases lung density which is seen as whiteness in the radiography images depending on the severity of the pneumonia. When hazy gray areas partially obscure dark lung markings in CXR, the observation is referred to as the ground-glass opacity. A ground-glass opacity is a hazy increase in attenuation in interstitial and alveolar processes. Linear opacities (Peripheral, coarse, horizontal white lines, bands, or reticular changes) may also be seen along with ground-glass opacity. In severe cases, lung markings are completely lost because of whiteness, which is referred to as consolidation. These changes are more likely to be peripheral and lower zones, but the whole lung can also be affected. Bilateral lung involvement is most commonly observed. The appearance of nodules, pneumothorax, or pleural effusion might be incidental for COVID-19 cases [14]. Since the infected regions are always associated with the lungs, a trained radiologist naturally interprets a COVID-19 CXR image by first outlining the anatomical structure of the lung. Similarly, an anatomy-informed algorithm can analyze the CXR images for a superior analysis and subsequent severity prediction. For example, if the lung regions are pre-identified, the deep learning models can be trained using higher resolution images only of the relevant portion of the images. We refer to this type of method as being anatomy-aware.

This paper proposes a novel COVID-19 severity detection method for chest x-ray images using a lung-annotated anatomy-aware deep learning model. The proposed model integrates the generic x-ray image features with pre-identified anatomical information extracted using an auxiliary segmentation model. The proposed model generates both disease-level features and lung involvement percentage to estimate the severity of pneumonia in COVID-19 patients.

Related Work

Previous work on the automated assessment of CXR images for COVID-19 has focused on disease classification, segmentation, and severity prediction. Transfer learning techniques have proven to be effective ways to classify COVID-19 from CXR images. In [15], ResNet50, InceptionV3, and InceptionResNetV2 architectures are used to classify COVID-19 disease from CXR images, achieving accuracies of about 98.0%, 97.0%, and 87%, respectively. A number of deep learning architectures have been explored to classify COVID-19 disease from CXR images [6–11, 16–22]. Generative Adversarial Network (GAN)-based data augmentation to detect COVID-19 from CXR images has also been studied, and the result surpasses other augmentation methods [23, 24]. A densely attention-based deep network with high accuracy was proposed for automatic COVID-19 recognition [25]. In [26], a majority voting-based classifier ensemble of five benchmark algorithms was shown to provide promising results in detecting COVID-19 from CXR images. In [27], a wavelet-based depthwise convolution network with Grad-CAM visualization is used for the diagnosis of COVID-19. DMFL-Net, a deep learning framework is proposed ensuring latency, energy usage, and privacy of sharing data across hospitals [28]. A reliable COVID-19 detection architecture (ReCovNet) was developed to detect COVID-19 out of 14 different thoracic diseases using CXRs compiling QaTa-COVID-19 dataset of their previous study [29, 30]. DarkCovidNet, a deep learning model was proposed for both multi and binary classification of COVID-19 with an accuracy of 87.02% and 98.08% for multi and binary class, respectively [31]. Several approaches have been explored to reduce redundant input information using different feature extraction approaches [32–39]. In [40], pre-trained BiT models were analyzed, where DenseNet (using an additional Dense layer with 512 perceptrons) performed best with 92% classification efficiency for automatic COVID-19 prediction. However, none of these previous methods consider the analysis of lung anatomical structure for disease localization or classification. On a separate note, although the COVID-19 detection methods using CXR have achieved promising results, the detection method is not suitable for replacing a traditional RT-PCR diagnostic test. Several approaches have been explored to develop an architecture for both CXRs and CT scan images [41, 42]. In [43], a lightweight CNN-tailored deep neural network (DNN) is proposed to detect COVID-19 from both CT scans and CXRs using a small number of images. Transfer learning-based feature extraction with a capsule network has also been used on both CXRs and CT images to improve performance and avoid overfitting [44]. In our experience, radiologists and general physicians are capable of detecting COVID-19 markers from CXR images, so automatic detection of its presence may not provide a significant advantage. However, disease severity analysis may be more valuable in resource-constrained settings.

There has also been a considerable amount of work on COVID-19 diseased region segmentation in recent years. In [45], a deep neural network-based method is used for the detection of COVID-19 symptoms from CXR images, and gradient-guided class activation maps (Grad-CAM++) and layer-wise relevance propagation (LRP) are used to highlight class-discriminating regions. A segmentation-based deep fusion network is used to classify 14 thoracic diseases using a CXR image, where local lung region images are extracted with discriminative features for improved performance [46]. Anatomy-XNet, an anatomy-aware attention-based network incorporating the spatial features guided by the pre-identified anatomy regions proves the efficacy of utilizing the segmentation knowledge to improve the classification of 14 thoracic diseases [47]. Unsupervised clustering techniques are also used to localize the suspected abnormal regions in the lung field of the CXR images [48]. Additionally, integrating an attention gate (AG) into a standard convolutional neural network (CNN) increases the precision and predictive accuracy of automatic segmentation of COVID-19 regions in CT images using minimum computation [49].

Previous work on COVID-19 severity prediction from CXR images has mainly used variants of deep learning architectures for transfer learning [13, 50–53]. However, these methods also do not explicitly use anatomical information to interpret similar to a radiologist.

Methodology

This work proposes a unified framework in which an anatomy-aware (AA) deep learning model performs feature extraction while a regression model performs severity prediction. The overview of the entire architecture is illustrated in Fig. 1.

Fig. 1.

Overview of the proposed Anatomically Aware Network used for COVID-19 severity prediction. We use a Cycle-GAN trained image to the label generator for lung segmentation. The lung information is then incorporated in the original radiographs through the anatomy coding block which performs Eq. 2. This is followed by the Densenet-121 feature extraction and the gating function. In path I, we use Feature Pyramid Attention for improved pixel-level attention and in consequence, generating five diseases (Atelectasis, Consolidation, Edema, Pleural Effusion, and consolidation) level features. On the other hand, path II is used for class-wise heatmap generation with PCAM pooling. The lung involvement score is calculated using the generated heatmaps and lung masks. Finally, disease-level features and lung involvement scores are used for regression

The proposed AA model consists of a pre-processing model and a backbone model. The pre-processing model, named the Anatomy Annotation (ANT) model, infuses the anatomy structure information of CXRs, and the backbone model is used for dense pooling. Then, the feature map is passed through a gating system consisting of two paths. One path utilizes the Feature Pyramid Attention (FPA) module [54, 55] for improved pixel-level attention and extraction of features. This feature map is then pooled to obtain the disease-level features (feat 1). The other path uses Probabilistic Class Activation Map (P-CAM) pooling [55], which is known to have excellent localization ability. We generate disease-wise heatmaps with P-CAM pooling and calculate the lung involvement of each disease in decimal form (feat 2). The disease-level features and the percentage of lungs covered by the heatmaps are used as feature vectors and concatenated to generate the final feature vector. Finally, a linear regression model is applied to this feature vector to obtain the COVID-19 severity scores.

Anatomy Annotation Block

To incorporate the anatomical features in the AA model it is important to emphasize on different anatomical structures, e.g., lungs, heart, ribs, clavicles, diaphragm, etc. In this work, we present a novel pre-processing method termed as Anatomy Informed Annotation.

To perform lung segmentation, we employ a Cycle-GAN-based semi-supervised method that has shown superior performance compared to current methods for this task [56, 57]. The entire architecture is illustrated in Fig. 2. The first generator ($G_{\mathit{IS}}$), which corresponds to the segmentation network that we want to obtain, learns a mapping from an image to its segmentation labels. The first discriminator ($D_S$) attempts to differentiate these generated labels from real segmentation masks. Conversely, the second generator ($G_{\mathit{SI}}$) learns to map a segmentation mask to its original x-ray image. In our semi-supervised segmentation setting, this generator is only used to improve the training procedure. Similarly, the second discriminator ($D_I$) receives an image as input and predicts whether this image is real or generated. To enforce cycle consistency, the generators are trained such that feeding the labels generated by $G_{\mathit{IS}}$ for an image into $G_{\mathit{SI}}$ provides the same image, and passing back to $G_{\mathit{IS}}$ the image generated by $G_{\mathit{SI}}$ for a segmentation mask gives the same mask. We use the SCR dataset segmentation masks and 200 synthetic segmentation masks obtained from the Stanford Chexpert dataset (generated by a pre-trained U-Net [58]) as labeled chest X-rays. In this way, we generate segmentation masks for the entire Chexpert dataset in a semi-supervised way, using the anatomy-informed annotation.

Fig. 2.

The proposed Cycle-GAN-based architecture for semi-supervised lung segmentation. $G_{\mathit{IS}}$ and $G_{\mathit{SI}}$ are the two generators, where $D_I$ and $D_S$ work as discriminators. These four networks are trained simultaneously

Let $\mathbf{x}$ be a chest X-ray image and $\mathbf{y}$ be the generated segmentation mask. The output of the neural network, which performs anatomy-informed segmentation, can be expressed as a function:

$$\begin{array}{c}\hfill {\mathbf{y}}_{i,j}=\prod _{i,j}{\mathbf{f}}_{\mathit{ks}}({({\mathbf{x}}_{(s)i+(\delta )i,(s)j+(\delta )j})}_{0\le (\delta )i,(\delta )j\le k})\end{array}$$ 1

where k is the kernel size, s is the stride or subsampling factor, and ${\mathbf{f}}_{\mathit{ks}}$ is the layers of the neural network which is determined by the layer type. The output $\mathbf{y}$ is used for merging anatomy information in the original CXR image, $\mathbf{x}$. We can denote the RGB vector image as three separate column vectors, like, $\mathbf{x}=[{\mathbf{x}}^0{\mathbf{x}}^1{\mathbf{x}}^2]$. These column vectors can be denoted as three RGB channel column matrices. We can write as, $[{\mathbf{x}}^0{\mathbf{x}}^1{\mathbf{x}}^2]=[{\mathbf{X}}^R{\mathbf{X}}^G{\mathbf{X}}^B]$. Then this data vector is infiltrated by a gray-scale segmentation mask, $\mathbf{Y}=[{\mathbf{y}}_{\mathit{ij}}]$, which is composed of column vectors generated by the neural network shown in Eq. 1.

$$\begin{array}{c}\hfill \tilde{\mathbf{X}}=\mathbf{X}+\mathbf{I}·{\mathbf{P}}^T·\mathbf{Y}\end{array}$$ 2

We can find the anatomy-informed image using Eq. 2, where $\mathbf{I}$ is the identity matrix, $\mathbf{P}$ is a hyper-parameter matrix for controlling the measures of infiltration in a specific RGB channel. In order to keep the actual information of the chest radiographs as much as possible, only the Blue channel has been infiltrated with the anatomical information in a small amount, so the hyper-parameter matrix, $\mathbf{P}$ can be written as $\mathbf{P}=[11p],0<p<1$. This anatomy-informed image, $\tilde{\mathbf{x}}=[{\tilde{\mathbf{X}}}^R{\tilde{\mathbf{X}}}^G{\tilde{\mathbf{X}}}^B]$, is then fed into the backbone model.

Backbone Model

The backbone model is a traditional deep learning classification network used in transfer learning. In this case, we use Densenet-121 [59] as its dense block has been well-known for its feature-reuse capability during feature extraction. Training this model using the Anatomy-annotated images, $\tilde{\mathbf{x}}$, ensures that the model is aware of the chest radiograph anatomy.

Feature Pyramid Attention

To produce improved pixel-level attention, we incorporate a Feature Pyramid Attention (FPA) module [54] into the system. The Pyramid Attention module first fuses features from two different branches: three $n\times n$ pyramid scales convolution (n = 3,5,7) and origin features from CNNs going through a $1\times 1$ convolution. Then, a global average pooling branch feature is added with the output features to select the discriminative multi-resolution feature representation. When these features overlap significantly, final pooling is applied to extract the disease-level features (pre-softmax or pre-sigmoid output).

P-CAM Pooling

We use P-CAM pooling for improved heatmap generation. P-CAM pooling explicitly leverages the excellent localization ability of CAM [60] during training in a probabilistic fashion. The backbone network first processes the input CXR image and generates a feature map. Then, for a particular disease label, such as “Consolidation,” each feature embedding within the feature map goes through a fully connected (FC) layer implemented as a $1\times 1$ convolutional layer and generates the class activation score that monotonically measures the disease likelihood of each embedding. It is further bounded with the sigmoid function and interpreted as the disease probability of each embedding. Finally, the output probability map is normalized to attention weights of each embedding, following the multiple-instance learning (MIL) framework, which is used to pool the original feature map by weighted average pooling. The pooled embedding goes through the same FC layer introduced above and generates the image-level disease probability for training. During inference time, the probability map is directly used for localization. Then, we apply simple hyper-parameter thresholding to obtain disease regions. Finally, for the disease d, we define its activation score as follows:

$$\begin{array}{c}\hfill {\text{A}}_d=\frac{{\sum }_{0,0}^{h,w}[L_{i,j}\bigcap R_{i,j}^d]}{{\sum }_{0,0}^{h,w}{L}_{i,j}}\end{array}$$ 3

where $L_{i,j}$ is lung segment (right and left both stacked), $R_{i,j}^d$ is the d region of the disease created from heatmap. The summation is across the segment and region’s height and width. According to Eq. 3, disease activation will be between 0 to 1.

Data Sources

We use four different datasets for the training and evaluation of the proposed COVID-19 severity detection framework. These include chest x-ray image datasets including anatomy segmentation masks and disease labels. The datasets are as follows:

JSRT Dataset

The JSRT dataset [61] consists of 247 images (154 nodule and 93 non-nodule images), with a resolution of $2048\times 2048$. This dataset also includes patients information such as: age, gender, diagnosis (malignant or benign), X and Y coordinates of nodule, simple diagram of nodule location.

SCR Dataset

SCR dataset [62] is a database of posterior-anterior chest radiographs where the manual segmentation of lungs, heart, and clavicles is provided. This dataset includes chest radiographs of 247 subjects where annotations of the anatomical structures of the images of the JSRT database, e.g., left lung, right lung, heart, left clavicle, and right clavicle, are included.

Stanford Chexpert Dataset

The Chexpert dataset [63] is a large public dataset for chest x-ray interpretation. This dataset contains 224, 316 radio-graphic images of 65, 240 patients labeled with 14 observations such as the following: Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion, Pneumonia, Pneumothorax, Enlarged Cardiom., Lung Lesion, Lung Opacity, Pleural Other, Fracture, Support Devices, No Finding.

COVID-19 Pneumonia Severity

The COVID-19 Pneumonia Severity Dataset [13] is a small dataset with 94 images, where each corresponding image has two severity scores: the geographic extent score and the opacity score. The geographic extent score denotes the extent of lung involvement by ground-glass opacity or consolidation for each lung. Here, 0 = no involvement, 1 = <25% involvement, 2 = 25–50% involvement, 3 = 50–75% involvement, and 4 = >75% involvement. The total extent score ranges from 0 to 8, including both lungs. On the other hand, the opacity score signifies the degree of opacity for each lung. Here, 0 = no opacity, 1 = ground-glass opacity, 2 = consolidation, and 3 = white-out. This score ranges from 0 to 6, accumulating both lungs. The labeling of the geographical extent and opacity scores is shown in Fig. 3.

Fig. 3.

Labeling of geographical extent and opacity score

Dataset Organization

In this work, we only use lung annotations for our experiments. First, a U-Net model is trained on the JSRT dataset using SCR labels for lung segmentation. This U-Net is then applied for initially generating 200 lung masks from the Chexpert dataset. Due to dataset mismatch, the U-Net performance on CheXpert is inferior compared to the JSRT data. However, these particular 200 images and their corresponding lung annotations are selected through manual examination to provide an adequate lung mask. These 200 synthetically labeled images together with SCR clinical ground truth annotations are later used for the lung annotation of the entire Chexpert dataset using the Cycle-GAN-based semi-supervised segmentation approached discussed in the “Anatomy Annotation Block” section. Finally, these lung segments are used for appending anatomical structural information based on Eq. (2) on the original CheXpert images. These anatomy-annotated x-ray images are then used for training the Densnet121-FPA and Densnet121-PCAM models. The datasets utilized in this study are summarized in Table 1.

Table 1.

Datasets required for the training and evaluation of the proposed framework

Dataset	Number of CXRs
JSRT Dataset [61]	247 CXRs with patients information
SCR Dataset [62]	247 CXRs with annotated anatomical information
Stanford Chexpert Dataset [63]	224,316 CXRs with 14 observations
COVID-19 Pneumonia Severity Dataset [13]	94 images with geographic extent score and opacity score
In-house Dataset	12 independent CXRs annotated by experienced radiologist

To evaluate the performance of our framework, we use the COVID-19 Pneumonia severity dataset [13]. We perform prediction tasks on a held-out test set using a 5-fold cross-validation scheme. An overall flow diagram describing the data organization and experimental design of this paper is illustrated in Fig. 4.

Fig. 4.

Dataset organization and overall flow diagram in the feature extractor training and regression training/testing phases

Experimental Evaluation

Training Phase

In the training phase, we train a U-net model on the SCR dataset. We infer this U-net model on the 200 selected Stanford Chexpert images and generate lung segmentation masks. These 200 synthetic labels together with SCR ground truth labels are used to train the Cycle-GAN model. Next, we generate the lung segments for the entire Chexpert dataset and incorporate them within the x-ray images using Eq. (2). Afterward, we train our feature vector generation module using these lung-annotated radiographs. The Densenet121-FPA/PCAM model is trained using the following training protocols. Densenet initialization is done by ImageNet weights [64], the batch size is set to 24, images are resized to $256\times 256$, and the binary cross-entropy (BCE) loss is used. The training is done for 5 epochs on 3 RTX 2080ti graphics processing units (GPUs). We use this model for feature vector extraction.

Prediction Phase

In the prediction stage, we use two different types of features for the regression analysis. The first type consists of the outputs from the Densenet121-FPA classifier pipeline considering specific lung-related diseases. These diseases include Edema, Consolidation, Atelectasis, and Pleural Effusion. The second type of feature includes the activation values calculated from the disease activation as defined in Eq. (3) considering the same lung-related diseases. In total, we have a maximum of 8 feature parameters that are available for disease severity prediction via regression. During our experiments, we have selected different combinations of these features for linear regression. We report the mean squared error (MSE), mean absolute error (MAE), and correlation coefficient as performance metrics comparing the predicted COVID-19 severity and the ground truth for the geographic extent and opacity scores. The results are summarized in Table 2.

Table 2.

Performance evaluation of the proposed COVID-19 severity prediction architecture compared to existing models. In the table: 3 diseases include: Atelectasis, Edema and Consolidation related features. In 4 diseases features Pleural Effusion related features are added. Also results have been generated with the individual disease and the best one is shown as single class (like in case of Densenet-FPA with AA, Edema has the best correlation with the geographic extent. These results also compared with the baseline scores from Cohen et al. [13])

Task	Method	Features	MSE	MAE	R2
Geographical Extent Score	Densenet-FPA without AA	3 diseases	$2.25\pm 0.62$	$1.22\pm 0.12$	$0.59\pm 0.10$
		4 diseases	$1.93\pm 0.63$	$1.16\pm 0.15$	$0.63\pm 0.11$
		single disease	$3.16\pm 0.72$	$1.45\pm 0.18$	$0.41\pm 0.04$
	Densenet-FPA with AA	3 diseases	$1.90\pm 0.45$	$1.14\pm 0.12$	$0.64\pm 0.05$
		4 diseases	$1.85\pm 0.29$	$1.21\pm 0.16$	$0.63\pm 0.10$
		single disease	$3.45\pm 1.56$	$1.34\pm 0.20$	$0.38\pm 0.15$
	Densenet-FPA with AA and Disease Activation features	3 diseases	$1.87\pm 0.51$	$1.15\pm 0.09$	$0.63\pm 0.10$
		4 diseases	$1.90\pm 0.39$	$1.12\pm 0.11$	$0.63\pm 0.05$
		single disease	$3.52\pm 1.24$	$1.34\pm 0.20$	$0.35\pm 0.10$
	Cohen et al. [13]	lung opacity	$2.06\pm 0.34$	$1.14\pm 0.10$	$0.60\pm 0.09$
Opacity Score	Densenet-FPA without AA	3 diseases	$1.20\pm 0.14$	$0.76\pm 0.04$	$0.57\pm 0.09$
		4 diseases	$1.08\pm 0.22$	$0.80\pm 0.10$	$0.61\pm 0.07$
		single disease	$1.37\pm 0.47$	$0.94\pm 0.16$	$0.39\pm 0.10$
	Densenet-FPA with AA	3 diseases	$0.94\pm 0.20$	$0.82\pm 0.04$	$0.55\pm 0.09$
		4 diseases	$0.97\pm 0.23$	$0.85\pm 0.11$	$0.56\pm 0.11$
		single disease	$1.54\pm 0.47$	$1.10\pm 0.12$	$0.32\pm 0.08$
	Densenet-FPA with AA and Disease Activation features	3 diseases	$0.94\pm 0.21$	$0.81\pm 0.05$	$0.46\pm 0.20$
		4 diseases	$0.96\pm 0.24$	$1.15\pm 0.31$	$0.45\pm 0.20$
		single disease	$1.45\pm 0.50$	$1.52\pm 0.21$	$0.20\pm 0.36$
	Cohen et al. [13]	lung opacity	$0.86\pm 0.11$	$0.78\pm 0.05$	$0.58\pm 0.09$

Overall Results

Our baseline model using Densenet121-FPA without the proposed anatomy-aware (AA) block provides very similar results as reported in [13]. In terms of geographical extent score, Densenet121-FPA and [13] provide MSE scores of $1.93\pm 0.63$ and $2.06\pm 0.34$, respectively (p-value = $0.001<0.01$). In the case of lung opacity score, Densenet121-FPA and [13] provide MSE scores of $1.08\pm 0.22$ and $0.86\pm 0.11$ (p-value = $0.0<0.01$), respectively. Thus, we can conclude that the baseline model consisting of the Densenet121-FPA pipeline is equivalent to [13] in terms of performance and statistical significance. Table 2 also shows that the overall performance improves after including the anatomy-aware block.

The best-performing model is Densenet121-FPA, with the AA block included in geographic extent and opacity scores. In terms of geographical extent score, the MSE improves from $1.93\pm 0.63$ to $1.85\pm 0.29$ after including the AA block over the baseline architecture. On the other hand, for lung opacity score, the MSE improves from $1.08\pm 0.22$ to $0.97\pm 0.23$ after including the AA block along with the baseline model. However, Table 2 also shows that including the disease activation score does not provide the best result for the prediction of lung opacity score. Overall, analyzing the results of Table 2 implies that including additional anatomical information to the competitive baseline model further increases the ability of the system for disease severity prediction. In Fig. 5, we show a few example images along with predicted and ground truth COVID-19 severity scores. Figure 6 shows the comparison between our model’s performance with human annotation in a 5-fold cross-validation setup for geographic extent and lung opacity scores, respectively. The plots again indicate a high correlation between the ground truth disease severity and the predicted value.

Fig. 5.

Comparison of predicted (Pred.) and ground truth/actual (Act.) scores of selected x-ray images for geographical extent score (Geo.) and lung opacity score (Op.)

Fig. 6.

Model’s prediction comparison with ground truth (GT). Here different colored Cartesian coordinates represent different folds in 5-fold cross-validation settings

Evaluation on In-house Annotated Data

In order to validate the performance of the proposed method on an independent test set, we selected a small set of chest x-ray images and performed annotation by an experienced radiologist. In this study, 12 chest x-ray images were selected from a separate COVID-19 dataset. The same severity scoring method used in [13] was followed during the in-house annotation by the experienced radiologist. The geographic extent and lung opacity scores provided by our radiologist and the proposed anatomy-aware model are summarized in Table 3 and Fig. 7. We found significant improvement in the geographic extent score and opacity score (p-value $<0.01$) compared with [13] using the selected CXRs. As evident from the results, our model shows low MAE values for both geographic extent and lung opacity scores. However, the MSE is found to be lower for lung opacity score compared to that of the geographic extent scores. These results further demonstrate the effectiveness of the proposed method. However, we acknowledge that this independent validation set is small in size. Further evaluation on a larger independent dataset would provide more convincing evidence of the effectiveness of the proposed method.

Table 3.

Evaluation of the proposed COVID-19 severity prediction model on Selected Chest X-ray images annotated by an In-house experienced radiologist

Scoring Method	MAE	MSE
Geographical Extent Score	1.55±0.98	3.35$\pm 3.51$
Opacity Score	0.62$\pm 0.48$	0.59$\pm 0.89$

Fig. 7.

a Predicted value and ground truth of geographical extent score of selected chest X-ray images annotated by an In-house experienced radiologist, and b Predicted value and ground truth of opacity score of selected chest X-ray images annotated by an In-house experienced radiologist

Discussion

Most of the previous research on chest x-ray image analysis involves traditional deep learning architectures typically used for image classification. There are few studies that are involved with COVID-19 severity prediction. Basically, for severity prediction, these studies are involved with pre-trained models without anatomical information [13]. Additionally, there is a need for a case-by-case comparison to find out the possible causes of error to improve the image analysis architecture [53]. Also, there is a lack of CXRs for performing the segmentation tasks, which sometimes makes it difficult, and there may be anomalies in the findings. However, the existing models are not specifically aware of the anatomical structure present in typical chest x-ray images. In contrast, an experienced radiologist always first identifies the thoracic organs before looking for markers of the disease. Therefore, we hypothesized that adding anatomical information to existing models can help to improve the model’s performance in predicting disease severity. Also, anatomical information can reduce the computational complexity by introducing disease-specific features, which helps the model to learn precisely. The experimental results presented in this work demonstrate that including simple anatomical information, e.g., coloring the x-ray images by lung segmentation masks, can improve the performance of the COVID-19 severity prediction model. Our baseline architecture performs similarly compared to existing models on the COVID-19 pneumonia severity prediction task. We have also included a case-by-case comparison study with the annotated CXRs to quantify the performance of the proposed algorithm. However, the proposed anatomy-informed method yields about $4.1\%$ and $11\%$ relative improvement in MSE for the geographic extent and lung opacity scores, respectively. For the future perspective of this study, there is a need for further clinical validation on a larger-scale independent dataset and validation in a clinical workflow for effective deployment in low-resource hospital settings.

Conclusion

This paper has proposed a novel anatomy-aware deep learning framework for COVID-19 disease severity prediction from chest X-ray images. While traditional methods generally do not specifically consider anatomical information for medical image analysis, expert radiologists tend to always consider their human anatomy knowledge before making a diagnostic decision. In this work, we have utilized a semi-supervised model for automatically generating lung segmentation masks that are subsequently fused within the chest X-ray images. Our best-performing model has provided a relative improvement of $11\%$ in MSE compared to existing methods when evaluated on a COVID-19 pneumonia severity prediction dataset. Experimental comparisons between systems with and without anatomy information integrated clearly show the effectiveness of the proposed method. Our model also shows promising results on an unseen in-house clinical evaluation dataset that an experienced radiologist has annotated. The experimental evaluations demonstrate the effectiveness of the proposed anatomy-aware architecture for COVID-19 disease severity prediction. Though the COVID-19 transmission rate has significantly surged down, the proposed approach can be incorporated into other AI-based platforms for the severity prediction of other respiratory diseases. In different low-income settings and underprivileged areas, people with comorbid conditions are at risk of severe infections and there is a lack of medical facilities. Chest radiography is among the major radiological diagnostic methods in these regions. Hence, there is a need for computer-aided diagnostic methods to provide healthcare facilities in underserved communities where the proposed method can provide assistance to healthcare professionals. Thus, the extensions of the proposed method can be used for COVID-19 or other respiratory disease severity prediction in low-resource settings.

Author Contribution

Nusrat Binta Nizam: literature review, software/coding, experiments, writing manuscript, editing manuscript, preparing response to review document. Sadi Mohammad Siddiquee: conceptualization, methodology development, software/coding, experiments, writing first draft. Mahbuba Shirin: data collection and annotation. Mohammed Imamul Hassan Bhuiyan: conceptualization, supervision, writing — reviewing and editing. Taufiq Hasan: conceptualization, supervision, writing — reviewing and editing.

Funding

This work was supported by a Special Grant from the Information and Communication Technology (ICT) Division, Bangladesh.

Declarations

Ethics Approval

No human subjects were involved in this research. Existing open-access medical image datasets were used for this work.

Consent to Participate

Not applicable, since no human subjects were involved in this research.

Consent for Publication

Not applicable, since no human subjects were involved in this research.

Competing Interests

The authors declare no competing interests.