Deep Learning reveals lung shape differences on baseline chest CT between mild and severe COVID-19: A multi-site retrospective study

Abstract

Severe COVID-19 can lead to extensive lung disease causing lung architectural distortion. In this study we employed machine learning and statistical atlas-based approaches to explore possible changes in lung shape among COVID-19 patients and evaluated whether the extent of these changes was associated with COVID-19 severity. On a large multi-institutional dataset (N=3443), three different populations were defined; a) healthy (no COVID-19), b) mild COVID-19 (no ventilator required), c) severe COVID-19 (ventilator required), and the presence of lung shape differences between them were explored using baseline chest CT. Significant lung shape differences were observed along mediastinal surfaces of the lungs across all severity of COVID-19 disease. Additionally, differences were seen on basal surfaces of the lung when compared between healthy and severe COVID-19 patients. Finally, an AI model (a 3D residual convolutional network) characterizing these shape differences coupled with lung infiltrates (ground-glass opacities and consolidation regions) was found to be associated with COVID-19 severity.

1. Introduction

The on-going Coronavirus disease of 2019 (COVID-19) pandemic has prevailed in the form of multiple mutant variants including the most recent Omicron variants BA.4 and BA.5 (Rahman et al., 2022; Tuekprakhon et al., 2022). In spite of administration of multiple vaccine doses, COVID-19 has led to severe respiratory complications and numerous deaths (Jain et al., 2021). Chest computed tomography (CT) examination demonstrates pathological changes in the lung, mostly as lung infiltrates (ground-glass opacities and consolidations), especially in the peripheral regions of the lung (Kanne, 2020). Recently, studies have illustrated the importance of chest CTs not only for diagnosis of COVID-19 but also for staging and monitoring disease progression (Yang et al., 2020).

With COVID-19 cases continuing to remain high, artificial intelligence (AI) based approaches have been developed using CT or chest radiographs for COVID-19 diagnosis (Ren et al., 2022; Serte and Demirel, 2021) and distinguishing COVID-19 disease from other causes of pneumonia (Cardobi et al., 2021, p. 1). Additionally, a few studies have also developed AI-based biomarkers for COVID-19 prognosis identifying severe COVID-19 patients with high risk of mortality (Chamberlin et al., 2022) or those with need of mechanical ventilator (A.l. et al., 2021).

Recent studies have shown that severe COVID-19 disease can lead to complications such as pneumonia, acute respiratory distress syndrome (ARDS), and sepsis (SeyedAlinaghi et al., 2021). These complications may further cause acute lung damage followed by pulmonary fibrosis and chronic impairment of lung function (Wang et al., 2020, p. 19). While for some patients, damage caused to the lung may ameliorate over time, for others, lung damage may persistent and cause permanent loss of lung function (Wu et al., 2020). While it is known that severe COVID-19 disease may potentially cause extensive lung damage and persistent architectural distortion of the lung, to the best of our knowledge, there has been no quantitative effort to study impact of disease severity on lung structure, shape and morphology during early onset of the disease.

While previous studies have used deep learning approaches on CT scans for COVID-19 prognosis (Hiremath et al., 2021; Lee et al., 2021), most of these approaches rely on biomarkers derived from lung infiltrates for their analysis. However, the objective of this study was to explore the impact of severity of the disease on lung shape and morphology using baseline CT scans acquired at the time of COVID-19 diagnosis. On a large multi-institutional dataset (N=3443), at first, three different populations were defined; a) healthy (no COVID-19), b) mild COVID-19 (no ventilator required), c) severe COVID-19 (ventilator required) and separate statistical population-atlas models were built and compared to understand differences in shape and appearance of the lung. Subsequently, these statistical models were further integrated with deep learning to learn the extent of lung deformation associated with severity of the disease. Finally, deep learning derived features of lung deformation and shape differences were integrated with deep learning derived features of lung infiltrates to create a prognostic biomarker for COVID-19.

2. Methods and Materials

2.1. COVID-19 study population

Patients with COVID-19 disease as confirmed with reverse transcription polymerase chain reaction (RT-PCR) who underwent CT evaluation for diagnosis were identified for this study. This study included a retrospective cohort of N=3230 patients with data acquired from four different sources; N=835 patients from D₁ (Renmin Hospital of Wuhan University), N=113 from D₂ (University Hospitals Cleveland Medical Center), N=2000 from D₃ (a publicly available challenge dataset: Study of Thoracic CT in COVID-19 (STOIC)(Revel et al., 2021, p.)), N=282 from D₄ (a publicly available dataset from The Cancer Imaging Archive (TCIA) release by the Stony Brook University). The dataset from source D₁ was stratified based on the severity, and randomly split into 50% training (D₁^train: N=416) and 50% testing (D₁^test: N = 419), while data from the sources D₂, D₃ and D₄ were all used as an external independent test set to test our hypothesis. Summary characteristics with detailed information of cohorts D₁-D₄ including inclusion and exclusion criteria are provided in the supplementary section, S1. Additionally, details of CT acquisition parameters are presented in Table S1.

Additionally, a randomly selected set of N=213 patients from the National Lung Screening Trial (NLST) Spiral CT abnormalities dataset (Clark et al., 2013; National Lung Screening Trial Research Team, 2013; National Lung Screening Trial Research Team et al., 2011) marked as “Other minor abnormality noted” was used as part of a control cohort (D5). All patients from datasets D₁-D₅ were grouped into three categories based on severity; a) healthy patients (control cohort) with no COVID-19 disease), b) mildly affected COVID-19 patients (patients who did not require a mechanical ventilator), c) severely affected COVID-19 patients (patients who required a mechanical ventilator).

2.2. Manual delineations of lung and lung infiltrates

A board-certified cardiothoracic radiologist with 14 years of experience delineated lung regions and lung infiltrates (ground-glass opacities and consolidations) on the training set D₁^train (N = 416)_. The annotations were made using an open source software 3D Slicer v4.10 (Fedorov et al., 2012). For the rest of the patients in the test sets (D₁^test, D₂, D₃ and D₄), automated lung and lung infiltrate segmentations were obtained using a U-Net based convolutional neural network (CNN) trained from D₁^train.

2.3. Population based lung shape difference atlas (D_A) construction

2.3.1. Atlas construction with single split of training data

Difference atlases are often used to identify significant shape variations in anatomic regions between different populations using statistical approaches (Ashburner et al., 1998; Scher et al., 2007). The characteristics of shape differences as identified by difference atlases are further used to predict disease diagnosis and prognosis. In our study, we constructed a difference atlas (D_A) to identify significant shape differences between COVID-19 patients with mild and severe disease. Fig. 1 shows the flowchart explaining construction of a difference atlas (D_A) through a population-atlas model. The population atlas model takes CT scans of two sets of population separately as inputs and outputs D_A depicting statistically significant shape differences between the input populations.

Figure 1:

Population-based difference atlas construction identifying statistically significant shape differences between the two populations. At first, the lung regions in the chest CTs are manually annotated and a template is chosen based on the median lung volume of a particular population. As part of the second step, all the CT volumes are registered to the chosen template and corresponding transformations are applied to the lung masks. Subsequently, a signed distance function is calculated on all the lung masks in the template space to represent shape information as a volume. Finally, a difference atlas (D_A) is generated using a generalized linear model-based T-test with 500 permutations.

Detailed steps of generating D_A are given below;

Step 1: Population splits

COVID-19 CT scans were divided into two populations based on the severity of the disease; For example, patients with mild COVID-19 disease defined as one set of population and patients with severe COVID-19 disease requiring a mechanical ventilator as the other set of population.

Step 2: Template selection

A representative template T was chosen based on median lung volume from patients with milder disease among the two populations. For example, when constructing D_A between mild and severe COVID-19 patients, a template is chosen from mild COVID-19 cohort. For, D_A between healthy patients healthy and mild COVID-19 patients, template is chosen from the healthy cohort. Similarly, for constructing D_A between healthy and severe COVID-19 patients, template is chosen from the healthy cohort.

Step 3: Registration of all CT scans to the template space, T

All CT scans were registered to a common canonical frame of reference, T. A rigid followed by a b-spline deformable co-registration was performed. For rigid registration, mattes mutual information was used as a similarity metric with number of histogram bins set to 32. The similarity metric was optimized using gradient descent, a maximum of 500 iterations per resolution. A linear interpolator was used for rigid registration. Subsequently, to initialize deformable registration manually segmented lung masks were given as inputs to registration framework to provide anatomical constraints and improve registration accuracy. The anatomical constraints would help in making sure that the lung CT was sufficiently registered at the same time not introducing unnatural deformations in the lung. For the deformable registration, standard gradient descent optimizer and normalized correlation coefficient to measure the similarity between fixed and transformed volumes (Ghose et al., 2017) was used. The step size along the search direction for gradient descent was determined as a decaying function of the iteration loop with scalar constant set at 25000 to attain a close fit. A multi-resolution approach with 5 different levels was used with the number of iterations for each resolution set at 2500. Elastix tool v4.9 was used to perform both rigid and deformable registrations in this study (Klein et al., 2010).

Step 4: Volumetric representation of the shape of the lung using signed distance function

All the registered volumes were first isotopically scaled to 2mm³ resolution. A signed distance function (SDF) was applied to the corresponding binary lung masks in the template space. As opposed to a binary representation of a lung mask where each voxel within the lung is represented as 1 and voxels outside the lung are represented as 0, the SDF is defined as;

$$\text{SDF}=\{\begin{array}{ll}-d(x,ß)\hfill & (x\in ß+)\hfill \\ 0\hfill & (x\in ß)\hfill \\ d(x,ß)\hfill & (x\in ß-)\hfill \end{array}$$

Where ß represents the boundary of the lung, ß+ represents the voxels inside the boundary and ß− represents the voxels outside the boundary. −d(x1 x2) represents the Euclidean distance between the voxels x1 and x2. Python 3.11 with modules numpy (v1.26.4) and SimpleITK (v2.3.1) was used to apply SDF to volumetric images.

Step 5: Difference atlas (D_A) using a non-parametric Generalized Linear Model (GLM) based t-test

As described in a previous study (Ghose et al., 2017), at each voxel location in the template space (T), distributions of voxel values (based on signed distance function) between CT scans of two populations were tested for statistical significance using Generalized Linear Model (GLM) based t-test. The statistical testing was performed with 500 random permutation testing correcting for multiple comparisons (Winkler et al., 2014). Threshold free cluster enhancement (TFCE) method was used to correct for multiple comparisons (Winkler et al., 2014). The categorical variable, patient population (for example mild or severe COVID-19 disease) was considered as an exploratory variable and the signed distance function as a dependent variable for GLM based t-test. A “randomize” function from “fsl” library (Woolrich et al., 2009) was used to perform GLM based t-test. The method provides a test statistic image and sets of p-values. The p-values obtained from the statistical test were further represented as D_A.

2.3.2. Ensemble difference atlas (D^E_A) construction using multiple splits of training data

To make the process of D_A construction more robust, populations were into three randomly selected overlapping subsets (~67% of total samples in each subset) and three different D_As were constructed. All the three D_As were further thresholded at p>0.05. Subsequently, an ensemble method with majority voting technique on the thresholded binary masks of D_A to create a final ensemble difference atlas (D^E_A). The supplementary figure, Fig. S1, illustrates the process of generating D^E_A.

Considering pairs of populations, D^E_As were constructed identifying regions of statistically lung shape differences between; a) healthy patients from D₅ and mild COVID-19 patients from D₁^train, b) healthy patients from D₅ and severe COVID-19 patients from D₁^train and c) mild and severe patients from D₁^train_. Additionally, differences in lung volumes and mean intensities within the lung between the populations were also reported using violin plots.

2.4. AI based quantification of lung shape differences

At first, the ensemble difference atlas (D^E_A) constructed between mild and severe COVID-19 patients was transformed back from template space to the corresponding original image space using the registration approach as used for atlas creation. In this case, the moving image for registration was considered to be the ensemble difference atlas (D^E_A) while the corresponding input CT scans were considered as fixed images. Previous studies(Eppel, 2018) have shown that using binary mask of regions of interest (ROIs) as an additional channel (priors) along with an image input can help the network focus their attention on the ROIs while still extracting contextual information from the whole image. Similarly, in this study, a 3D CNN (M₁) was used to characterize population lung shape differences for COVID-19 prognosis. At first, D^E_A constructed between mild and severe COVID-19 patients was used as a shape prior S_P (Fig. 2). S_P was used as an auxiliary channel along with chest CT scans to train M₁ (Fig. 2). A 3-fold cross validation of M₁ was performed on D₁^train and ensemble predictions (average predictions) of M₁ from 3-fold cross validation was used to evaluate the model’s performance of the test sets D₁^test, D₂, D₃ and D₄. Three different base CNN architectures 3D-SEResNet-50 (Hu et al., 2019; Kataoka et al., 2020), 3D-DenseNet-121 (Kataoka et al., 2020), 3D-ResNeXt-50 (Kataoka et al., 2020) were tested for evaluating architecture sensitivity. The architecture yielding the best cross-validation area under the receiver operating characteristic curve (AUC) on D₁^train was chosen as the final base architecture for M₁. Data preprocessing and further implementation details of M₁ are provided in the supplementary section, S2, S3. PyTorch v2.1.2 python module was used to train all CNNs for AI based quantification of lung shape differences (Paszke et al., 2019).

Figure 2:

Flowchart of CovSafeNet. CovSafeNet consists of two parallel 3D CNNs (M₁, M₂). While M₁ encodes shape differences of the lung between mild and severe COVID-19 patients via a shape prior S_P (top row), M₂ encodes spatial information of automatically segmented lung infiltrates, I_P (bottom row). The decisions from M₁ and M₂ are fused at the decision fusion node (N_F) to obtain the final predictions predicting severity of COVID-19.

2.5. Automated segmentations of lung and lung infiltrates

A 3D U-Net (Isensee et al., 2018) model was used to segment lung and lung infiltrate regions on CT scans. All CT scans were first pre-processed by converting them from Hounsfield units to image intensities by assuming the air in the lungs corresponds to a zero-intensity value. To narrow the field of view to the lung within the chest CTs, intensities greater than −300HU were thresholded and subsequently a bounding box containing the largest connected component was extracted. The cropped region was further resized to 128×192×96 (x, y, z) and was provided as input to a 3D U-Net. PyTorch v2.1.2 python module was used to train all CNNs for 3D segmentations of lung and lung infiltrates.

2.6. CovSafeNet: Integrating lung shape difference biomarker with lung infiltrates

At first, separate 3D U-Net (Isensee et al., 2018) models were used for segmenting lung regions and lung infiltrates on CT. D₁^train was further divided into sub-cohorts D₁^train1 (N=367) and D₁^train2 (N=49) and the 3D U-Net models were trained on D₁^train1 and evaluated on D₁^train2 and were subsequently used to automatically segment the lung and lung infiltrate regions of D₁^test, D₂, D₃ and D₄.

The segmentation maps outputted by the networks were used as infiltrate prior (I_P) to evaluate detection performance of the network in detecting the infiltrate regions. A region was considered as being detected if ≥ 0.2 Dice similarity coefficient (DSC) overlap existed between the network segmentation map and the ground-truth delineation of that corresponding region. The segmentation accuracy of the detected regions was reported in terms of DSC on D₁^train2.

Subsequently, I_P was then used as an auxiliary channel along with chest CT scans to train M₂ (Fig. 2) to identify COVID-19 patients with severe disease. A 3-fold cross validation of M₂ was performed on D₁^train and ensemble predictions (average predictions) of M2 from 3-fold cross validation was used to evaluate the model’s performance of the test sets D₁^test, D₂, D₃ and D₄. Three different base CNN architectures 3D-SEResNet-50 (Hu et al., 2019; Kataoka et al., 2020), 3D-DenseNet-121 (Kataoka et al., 2020), 3D-ResNeXt-50 (Kataoka et al., 2020) were tested for architecture sensitivity. The architecture yielding the best cross-validation AUC on D₁^train was chosen as the final base architecture for M₂. Data preprocessing and further implementation details of M₂ are provided in the supplementary section, S2, S3.

Subsequently, CovSafeNet (Fig. 2) framework was built based on decision fusion of two different residual classifiers M₁ and M₂ (average predictions from M₁ and M₂). While M₁ encoded lung structure differences of the lung between mild and severe COVID-19 patients via a shape prior S_P (Fig. 2), M₂ encoded spatial information of automatically segmented lung infiltrates, I_P (Fig. 2). A similar 3-fold cross validation strategy on D₁^train was used on CovSafeNet and the ensemble predictions (average predictions) of CovSafeNet from 3-fold cross validation was used to evaluate the model’s performance of the test sets D₁^test, D₂, D₃ and D₄. PyTorch v2.1.2 python module was used to train CovSafeNet (Paszke et al., 2019).

2.7. Comparison of CovSafeNet with clinical and radiomics based machine learning model

Radiomics model, M_R

PyRadiomics (van Griethuysen et al., 2017) package was used to extract the radiomic features from regions of lung shape differences and lung infiltrates. Extracted features included first order statistics, 2D and 3D shape base features, Gray Level Cooccurence Matrix (GLCM), Gray Level Run Length Matrix (GLRLM), Gray Level Size Zone Matrix (GLSZM), Gray Level Dependence Matrix (GLDM), Neighboring Gray Tone Difference Matrix (NGTDM). All CT volumes were resampled to 2mm³ isotropic resolution prior to the feature extraction. Two different models M_R1 and M_R2 were trained on D₁^train with features extracted from regions of population lung shape differences and lung infiltrates, and subsequently evaluated on the test sets D₁^test, D₂, D₃ and D₄. The final radiomic classifier M_R was based on decision fusion of M_R1 and M_R2. (addition of predictions from M_R1 and M_R2). The following steps were involved in training the models M_R1 and M_R2 at each iteration on every cross-validation fold.

• Step 1: Correlated features with a Pearson correlation > 0.9 were removed.

• Step 2: Constant features which were repeated for more that 25% of the samples were removed.

• Step 3: On the remaining features an outlier removal step was performed, allowing for clipping of the feature values between 1^st and 99^th percentile and resultant features were normalized in the range [0,1].

• Step 4: A Least Absolute Shrinkage and Selection Operator (LASSO) (“Regression Shrinkage and Selection via the Lasso on JSTOR,” n.d.) feature selection strategy was applied to select the most optimal radiomic features.

• Step 5: XGBoost classifier was trained based on the features selected based on LASSO feature selection.

Python module scikit-learn v1.3.2 was used to train the radiomics model MR.

Clinical Model, M_C

Clinical data was not available for the dataset source D₃. Therefore, clinical features common to D₁^train, D₂ and D₄ were selected for the analysis. Creatinine (Cr), aspartate aminotransferase (AST), alanine aminotransferase (ALT), neutrophils count (Neu), lymphocytes count (LYM) and procalcitonin (PCT) information was available for D₁^train, D₂ and D₄ cohorts. Subsequently, a similar preprocessing and model training strategy used for training the radiomics based classifiers was used to develop the clinical model M_C. Python module scikit-learn v1.3.2 was used to train the clinical model M_C.

2.8. Statistical analysis

Registration accuracies for atlas construction was evaluated using DSC between binary masks of lung regions of the fixed image (template mask) and moving image (image registered to the template) respectively. Additionally, DSC was also used to evaluate the performance of the automated segmentation of the lung and lung infiltrate regions using a 3D CNN. Detection performance of the lung infiltrate regions was evaluated by sensitivity and positive predictive value.

The outcome of interest in this study for disease severity was invasive mechanical ventilation/death vs. no invasive ventilation support (no respiratory distress, oxygen supplementation, non-invasive ventilation). Mean and standard deviation of AUC was used to indicate the cross-validation performance of all AI models on training set D₁^train. Further, AUC with 95% confidence intervals (CI) was used to capture performance of all AI models on test sets (D₁^test, D₂, D₃ and D₄). The DeLong test (DeLong et al., 1988) was used to compare the statistical significance with respect to difference in AUCs between the two models. Optimal cut-off point on receiver operating characteristic curve (ROC) was chosen by maximizing F1 score on the D₁^train. Furthermore, sensitivity, specificity and F1 score were reported on test sets (D₁^test, D₂, D₃ and D₄) based off the optimal cut-off point. pROC and cutpointr packages of R statistical software was used to perform the DeLong’s test and calculate optimizal cut-off point respectively.

3. Results

3.1. COVID-19 study population

A total of N=3230 patients acquired from four different sources (D₁: N=835, D₂: N=113, D₃: N=2000, D₄: N=282) were used in this study. An additional N=213 patients from the National Lung Screening Trial (NLST) Spiral CT abnormalities dataset (Clark et al., 2013; National Lung Screening Trial Research Team, 2013; National Lung Screening Trial Research Team et al., 2011) marked as “Other minor abnormality noted” were used as a control cohort (D₅) for positive COVID-19 patients. Table 1 shows patient demographics. Median age with interquartile ranges among the cohorts D₁ and D₂ were found to be 59 (47 – 67) and 62 (53 – 72) respectively. Median ages for D₃ and D₄ could not be evaluated as accurate age information was not available for subjects within the cohorts D₃ and D₄. All patients from datasets D₁-D₅ were grouped into three categories based on severity; a) healthy patients (control cohort) with no COVID-19 disease, b) mildly affected COVID-19 patients (patients who did not require a mechanical ventilator), c) severely affected COVID-19 patients (patients who required a mechanical ventilator).

Table 1:

Patient demographics of datasets D₁-D₃. All the demographic information was available for cohorts D₁, D₂. Although cohort C had age information, they were specified as ranges. Hence median and interquartile range related statistics could not be calculated. No demographic information was available for dataset D₄. μmol : micromoles; U : units; L : liter; ng: nanograms; mL: milliliter.

Variables	D1	D2	D3
Age Median (IQR)	59 (47 – 67)	62 (53 – 72)	-
Gender Male (%)	46.30	45.31	-
Laboratory findings Median (IQR)
Creatinine (Cr)(μmol/L)	60.0 (50.0 – 72.0)	99.0 (75.4 – 142.3)	84.4 (69.0 – 110.9)
Aspartate aminotransferase (AST)(U/L)	24.0 (19.0 – 37.0)	30.0 (21.0 – 47.0)	38.0 (26.0 – 61.0)
Alanine aminotransferase (ALT)(U/L)	23.0 (16.0 – 39.0)	21.0 (15.0 – 33.0)	30.0 (19.0 – 54.0)
Neutrophils count (Neu) (109 L−1)	3.63 (2.25 – 5.11)	4.82 (2.75 – 6.17)	5.3 (3.8 – 7.0)
Lymphocytes count (LYM) (109 L−1)	1.21 (0.88 – 1.68)	1.05 (0.77 – 1.30)	0.89 (0.65 – 1.36)
Procalcitonin (PCT)(ng/mL)	0.16 (0.055 – 0.24)	0.17 (0.08 – 0.54)	0.17 (0.09 – 0.34)

The dataset from source D₁ was stratified based on the severity, and randomly split into 50% training (D₁^train: N=416) and 50% testing (D₁^test: N = 419), while data from the sources D₂, D₃ and D₄ were all used as an external independent test set to test our hypothesis. Supplementary table, Table S2 shows the distribution of mild and severe COVID-19 patients on the training (D₁^train) and test sets (D₁^test, D₂, D₃ and D₄).

3.2. Chest CT reveals significant volume and lung shape differences between mild and severe COVID-19 patients

The registration accuracy for atlas creation, D_A was observed to be high with a DSC of 0.91±0.08 between the template, T and the registered lung masks. Additionally, the supplementary figure, Fig. S2 depicts deformation fields of deformable registration performed by registering the CT scans to a template volume. One can notice that applying anatomic constraints helps in making sure that no unnatural deformations are introduced during the registration processing. Furthermore, from supplementary figure, Fig. S2, one can notice that the deformation fields are oriented towards the ground lung infiltrates (glass opacities and consolidation regions). Hence, this shows that a COVID-19 patient with severe disease with the existence of more lung infiltrates presents higher deformations as compared to patients with milder COVID-19 disease with little or no lung infiltrates. Statistically significant volume differences were found between healthy, mild and severe patients in D₁^train (Fig. 3). One can observe that lung volumes decreased with increasing severity of COVID-19 disease (Fig. 3). Similarly, it may also be observed that mean intensity values within the lung on CTs increased with increasing severity (Fig. 3). Additionally, we also conducted the same analysis separately for male patients and female patients and found the similar trends in lung volumes and mean CT intensities (supplementary figure, Fig. S3).

Figure 3:

Differences in lung volumes and mean intensities within in lung as seen on CTs between three different populations a) severe COVID-19 patients (patients who required a mechanical ventilator) from the training set D₁^train (green), b) mild COVID-19 patients (patients who did not require a mechanical ventilator) from the training set D₁^train (orange), c) healthy patients (no COVID-19) from D₅ (blue). The plot shows that as the severity of COVID-19 disease increases, the lung volume decreases. Similarly, it can be observed that mean lung intensities increases as the severity of COVID-19 disease increases.

Population difference atlases (D_A) and subsequently ensemble of population shape difference atlases (D^E_A) (Fig. 1 shows construction of D_A and supplementary figure, Fig. S1 depicts construction of D^E_A) were constructed between pairs of populations (healthy, mild and severe COVID-19 patients). The D^E_A between mild and severe COVID-19 patients (Fig. 4) shows that the majority of the lung shape differences were observed at the mediastinal surface of both lungs. Furthermore, it may be observed that lung shape differences between healthy and severe COVID-19 patients (supplementary figure, Fig. S4) was extended to the basal surfaces of both the lungs along with mediastinal surfaces. However, lung shape differences were found only on the mediastinal surfaces for D^E_A between healthy and mild COVID-19 patients (supplementary figure, Fig. S5). Additionally, when D^E_A between healthy and mild COVID-19 patients, was compared with D^E_A between mild and severe COVID-19 patients, one can notice that differences between healthy and mild COVID-19 patients was more significant as compared to further worsening of the disease between mild and severe COVID-19 patients.

Figure 4:

Ensemble difference atlas (D^E_A) between mild (patients who did not require a mechanical ventilator) and severe (patients who required a mechanical ventilator) COVID-19 patients. Majority of the lung shape differences between V⁻ and V⁺COVID-19 patients were found at the mediastinal surface of both lungs (Blue: No statistically significant shape difference between the populations. Red: Regions with statistically significant shape difference between the populations).

3.3. AI based quantification of lung shape differences associated with severity of COVID-19

A 3D CNN (M₁: Fig. 2) was used to quantify the lung shape differences for predicting COVID-19 severity (distinguishing between mild and severe COVID-19 patients). Two channels were given as input to M₁ (Fig. 2) with first being the lung CT volume and the second being a binary mask of regions with lung shape differences. The base architecture for M₁ was chosen based on 3-fold cross-validation performance (AUC=0.810±0.045) on D₁^train (supplementary table, Table S3). M₁ resulted in AUC of 0.850; 95% CI [0.814, 0.888], 0.714; 95% CI [0.612, 0.816], 0.709; 95% CI [0.681, 0.738] and 0.610; 95% CI [0.542, 0.678] on D₁^test, D₂, D₃ and D₄ respectively. Additionally, M₁ yielded AUC=0.655; 95% CI [0.633, 0.678] when evaluated on the combined dataset (D₁^test, D₂, D₃, D₄).

Fig. 5 (top 2 rows) shows Gradient-weighted Class Activation Maps(Selvaraju et al., 2020) (Grad-CAM) interpretability results for M_1. These activation maps depict that CNN is focus its attention on regions with significant lung shape differences during the classification phase.

Figure 5:

Grad-CAM interpretability results with binary masks, S_P and I_P encoded into M₁ (1a-1h) and M₂ (2a-2h) respectively as auxiliary channels to the network. These maps show that these auxiliary channels can aid the network in setting an attention region helping the network to focus on these regions, while at the same time, providing the context of the whole lung region. Blue regions indicate areas contributing to predictions as mild COVID-19 disease while red regions indicate areas contributing to severe COVID-19 disease. The color bar gradient corresponds to the strength of the contribution.

3.4. CovSafeNet: A novel framework integrating biomarkers of lung shape differences and lung infiltrates for COVID-19 prognosis

A 3D CNN architecture (M₂: Fig. 2) was used for characterization of lung infiltrates. Details of lung infiltrates segmentation and segmentation results are presented in the supplementary section, S4. The supplementary figure, Fig. S6 shows some exemplar segmentations of lung and lung infiltrate regions. M₂ resulted in a 3-fold cross validation AUC=0.815±0.005 on D₁^train (supplementary table, Table S3). Furthermore, M₂ resulted in AUC of 0.867; 95% CI [0.833, 0.901], 0.770; 95% CI [0.667, 0.873], 0.697; 95% CI [0.666, 0.728] and 0.650; 95% CI [0.579, 0.723] on D₁^test, D₂, D₃ and D₄ respectively. Additionally, M₂ yielded AUC=0.680; 95% CI [0.658, 0.702] when evaluated on the combined test set (D₁^test, D₂, D₃, D₄).

Gradient-weighted Class Activation Maps (Selvaraju et al., 2020) (Grad-CAM) (Fig. 5) results for M₂ are presented in Fig. 5 (bottom two rows). Similar to M₁, one can notice that the auxiliary channel can aid the network in focusing on regions of interest (here lung infiltrates) to make its final decisions. Additionally, from Fig.5 one may observe that M₂ successfully classified a patient as having mild or severe COVID-19 even though lung infiltrates had similar appearance. For instance, from Fig. 5, mild COVID-19 patients (2a,2b) and severe COVID-19 patients (2e,2f) had similar appearance and size (smaller lung infiltrates), and M2 correctly identified the severity of the patients’ disease. Similarly, although mild COVID-19 patients (2c,2d) and severe COVID-19 patients (2g,2f) have similar appearance and sizes (larger lung infiltrates), M₂ correctly identified them as mild versus severe COVID-19 patients.

Subsequently, CovSafeNet (Fig. 2) was constructed based on decision fusion of M₁ and M₂. CovSafeNet resulted in an improved performance over M₁ and M₂, yielding a 3-fold cross validation AUC of 0.838±0.015 on D₁^train. Similarly, from Table 2, it may be observed that CovSafeNet significantly outperforms M₁ and M₂ resulting in an AUC of 0.693; 95% confidence interval (CI) [0.671,0.716] on combined test set (D₁^test, D₂, D₃, D₄).

Table 2.

Test set performance (AUC with 95% confidence intervals (CI)) of M₁, M₂ and CovSafeNet on D₁^test_, D₂, D₃ and D₄ datasets. The performance of CovSafeNet was compared with models M₁ and M₂ using DeLong’s test.

Model AUC (95% CI)		D1test N=419	D2 N=113	D3 N=2000	D4 N=282	Combined N=2814
M1		0.850 (0.814, 0.888)	0.714 (0.612, 0.816)	0.709 (0.681, 0.738)	0.610 (0.542, 0.678)	0.655 (0.633, 0.678)
M2		0.867 (0.833, 0.901)	0.770 (0.667, 0.873)	0.697 (0.666, 0.728)	0.650 (0.579, 0.723)	0.680 (0.658, 0.702)
CovSafeNet		0.890* (0.860, 0.921)	0.769 (0.667, 0.870)	0.732 (0.704, 0.761)	0.654 (0.583, 0.724)	0.693* (0.671, 0.716)
	DeLong’s Test with M1	p=0.0342	p= 0.9877	p<0.0001	p=0.812	p=0.0123
	DeLong’s Test with M2	p=0.0001	p=0.0589	p=0.0548	p=0.0558	p<0.0001

^*indicates statistically significant improvement as indicated by DeLong’s test.

CovSafeNet’s performance evaluated on individual cohorts D1^test, D2, D3 and D4 was found to be 0.890; 95% CI [0.860,1.921], 0.769; 95% CI [0.667, 0.870], 0.732; 95% CI [0.704,0.761] and 0.654; 95% CI [0.583, 0.724] respectively. CovSafeNet resulted in statistically significant improvement over M₁ and M₂ on D₁^test (AUC = 0.890; 95% CI [0.860,0.921]). However, similar statistically significant improvement was not found between M₁ and M₂ on cohorts D₃ and D₄ (Table 2). Also, on D₂, CovSafeNet performed similar to M₂ yielding an AUC=0.769; 95% CI [0.667,0.870]. Furthermore, other performance metrics such as sensitivity, specificity and F1 score are presented in the supplementary table, Table S4. CovSafeNet resulted in a sensitivity, specificity and F1 score of 86%, 36% and 51% respectively on the combined test set (D₁^test, D₂, D₃, D₄).

3.5. Comparison of CovSafeNet with clinical and radiomics based machine learning model

Radiomics and clinical machine learning classifiers M_R and M_C were trained with radiomic and clinical features respectively. Clinical features were not available for cohort D₃. Therefore, comparisons of M_R and M_C with CovSafeNet were only made with D₁^test, D₂, and D₄. Among the clinical features, aspartate aminotransferase (AST), neutrophils count (Neu), lymphocytes count (LYM) were selected as the most prominent clinical features. From Table 3 one can notice that CovSafeNet with an AUC=0.688 95% CI [0.653,0.724] significantly outperforms both M_R (AUC= 0.651; 95% CI [0.614, 0.689]) and M_C (AUC=0.607; 95% CI [0.568, 0.646]) on the combined cohorts D₁^test, D₂, and D₄. Additionally, the combination CovSafeNet with the clinical model (M_C) by pooling the decisions (average of decisions from CovSafeNet and M_C) yielded AUC=0.685; 95% CI [0.649, 0.721] with no statistically significant difference (p=0.6694) found compared to the performance of the CovSafeNet model. Additionally, by performing a multivariable analysis (Supplementary table, Table S5) we also show that CovSafeNet was independently associated with other clinical variables (age, Cr, AST, ALT, LYM, Neu, PCM).

Table 3.

Test set performance (AUC with 95% confidence intervals (CI)) of the radiomics based machine model (M_R), and clinical based machine learning model (M_C) and CovSafeNet on D₁^test_, D₂, D₃ and D₄ datasets. The performance of CovSafeNet was compared with models M_R and M_C using DeLong’s test.

Model AUC (95% CI)		D1test N=419	D2 N=113	D3 N=2000	D4 N=282	Combined (D1test, D2, D3 D4) N=2814	Combined (D1test, D2, D4) N = 814
Radiomics, MR		0.893 (0.856, 0.931)	0.641 (0.523, 0.752)	0.723 (0.692, 0.742)	0.579 (0.514, 0.648)	0.662 (0.631, 0.684)	0.674 (0.631, 0.703)
Clinical, MC		0.686 (0.638, 0.726)	0.668 (0.549, 0.781)	-	0.664 (0.599, 0.732)	-	0.602 (0.568, 0.651)
CovSafeNet		0.890 (0.860, 0.921)	0.769 (0.667, 0.870)	0.732 (0.704, 0.761)	0.654 (0.583, 0.724)	0.693* (0.671, 0.716)	0.688* (0.653, 0.724)
	DeLong’s Test with MR	(p=0.6028)	(p=0.0485)	(p=0.4116)	(p=0.0323)	(p<0.0001)	(p<0.0001)
	DeLong’s Test with clinical model MC	(p<0.0001)	(p=0.0925)	-	(p=0.6321)	-	(p<0.0001)

^*indicates statistically significant improvement as indicated by DeLong’s test.

3.6. Ablation studies

3.6.1. Sensitivity of template selection to construct difference atlas

To understand the effect of template selection on quantification of lung shape differences, we constructed D^E_A using a randomly selected template (T^r), instead of template T chosen based off median lung volume. AI based quantification of lung shape differences using T^r as template was then compared to the quantification using T as the template. The resulting M₁ trained using T^r as the template yielded an AUC of 0.847, 0.728, 0.689 and 0.569 on D1^test, D2, D3 and D4 respectively. DeLong’s test indicated no significant differences (p>0.12) on D₁^test, D₂, D₃, except for D₄ (p=0.0350).

3.6.2. 3D-CNN without shape and infiltrate priors

To evaluate the significance of shape prior (S_P) or infiltrate prior (I_P) provided to CNNs as an auxiliary channel, we trained a 3D-CNN with only chest CT volumes as the only input channel (without S_P and I_P). The 3D-CNN trained without S_P and I_P resulted in an AUC=0.792; 95% CI [0.77–0.82], AUC=0.645; 95% CI [0.60– 0.68], AUC=0.660; 95% CI[0.62– 0.69] and AUC=0.589; 95% CI[0.53– 0.63] on D₁^test, D₂, D₃ and D₄ respectively. With respect to DeLong’s test, CovSafeNet (with SP and IP) performed significantly better than a 3D-CNN trained without S_P and I_P as auxiliary channels (p<0.0331).

3.6.3. Comparison of CovSafeNet architecture with vision transformers

We further compared the best 3D-CNN architecture (3D-Densenet-121) with 3D vision transformers (Dosovitskiy et al., 2021) (implementation from MONAI opensource package (Consortium, 2020)). The CovSafeNet with 3D vision transformers architecture achieved an AUC=0.697 on the combined test set (D₁^test, D₂, D₃ and D₄) showing no significant difference from the 3D-Densenet-121 architecture (p=0.4321).

4. Discussion

While a number of previous studies have developed and applied artificial intelligence (AI) predictors for COVID-19 diagnosis and prognosis (Hiremath et al., 2021; Lee et al., 2021), there has not been, to our knowledge, a rigorous, comprehensive, and quantitative assessment of lung shape and deformation on account of COVID-19 and COVID-19 severity. Understand and predicting lung deformation and changes in lung shape could have long term implications for lung function. For instance, a study by Osanlouy et al. (Osanlouy et al., 2020, pp. 20–90) depicted that changes in lung shape could in turn affect lung function by causing aging. In this work AI based approaches were combined with statistical shape modeling based atlas approaches to (1) quantitatively capture lung morphological changes on account of COVID-19, (2) relate the changes in the shape and volume of the lung with severity of the disease and (3) combine deep learning with atlas based approaches to create integrated predictors of lung shape and lung infiltrates for COVID-19 severity prediction.

Previous studies (Luger et al., 2022) have shown that severe COVID-19 disease can have long-term sequelae such as subpleural reticulation and ground-glass opacities. Additionally, recent studies have also shown the mechanical deformations are introduced in the lungs due to severity of COVID-19 disease (Dimbath et al., 2021; Tonelli et al., 2021). Furthermore, extensive and persistent lung damage was also observed in a study with postmortem samples of 41 patients (Bussani et al., 2020) in patients with severe COVID-19 disease. These findings can be explained by the intense disruption of the normal lung parenchyma and interstitium with subsequent fibrosis. These damages caused to the lung may even lead to permanent loss of lung function (Wu et al., 2020). However, as previously mentioned, a quantitative assessment of the nature, extent and precise location of the COVID-19 induced lung deformation and associated damage has not been previously attempted.

Previous studies have developed various population-atlas based models to quantify shape differences of different organs on imaging and have illustrated their association with severity of various diseases (Asaeikheybari et al., 2021; Ghose et al., 2017; Ismail et al., 2022). To our knowledge, this is the first study to explore the association of COVID-19 severity with population lung shape differences. With pulmonary fibrosis proven as a long term sequelae of COVID-19 (McGroder et al., 2021), there are several possible factors such as reticulations, traction bronchiectasis or honeycombing which might be associated with changes in lung shape and morphology. Additionally, a previous study also indicated increased pulmonary arterial medial wall thickness among patients who died from COVID-19 pneumonia (“Potential long-term effects of SARS-CoV-2 infection on the pulmonary vasculature: a global perspective | Nature Reviews Cardiology,” n.d.). A related previous study by Lin et al. (Lin et al., 2021) has showed that micro-alveolar ruptures were associated with COVID-19 disease. These data appear to suggest that micro-alveolar ruptures in the mediastinal surfaces of the lung could lead to preferential pleural defects affecting lung shape and morphology. However, these studies (Lin et al., 2021; McGroder et al., 2021; “Potential long-term effects of SARS-CoV-2 infection on the pulmonary vasculature: a global perspective | Nature Reviews Cardiology,” n.d.) have been mostly qualitative in nature. Our study represents the largest one to date in the use of AI and statistical atlas modeling for quantitative assessment of disease severity as also lung shape related changes using chest CT scans on account COVID-19.

The new study presented in this work revealed that majority of lung shape related differences between mild and severe COVID-19 patients were found at the mediastinal surfaces of both lungs. Furthermore, when lung shape differences were evaluated between a healthy control cohort and severely affected COVID-19 patients, lung shape differences were found to extend to basal surfaces (supplementary figure, Fig. S5) as also mediastinal surfaces. This can be explained due to the fact that bilateral lower lobes are more commonly involved with high occurrence of lung infiltrates in severe COVID-19 patients (“Time Course of Lung Changes at Chest CT during Recovery from Coronavirus Disease 2019 (COVID-19) - Google Search,” n.d.). Additionally, it was also noted that initial shape differences (between control and mild COVID-19 patients) were more prominent compared to further worsening of the disease (between mild COVID-19 patients and severe COVID-19 patients). Therefore, early characterization of these physiological based changes can portend the possible long-term sequela of COVID-19 disease and help in better prognosis of COVID-19 patients.

Recently, a few previous studies have developed artificial intelligence (AI) based prognostic models to predict the severity of COVID-19 disease using convolutional neural networks (CNNs) (Meng et al., 2020) and radiomics based approaches (Chen et al., 2021). Furthermore, some of the previous approaches have even integrated CT based biomarkers with clinical and biological variables such as prothrombin time, albumin, age, sex, neutrophil counts etc (Hiremath et al., 2021; Lassau et al., 2021; Vaidya et al., 2021). However, most of these studies have focused only on target regions of interest (ROIs) in lung parenchyma such as lung infiltrates (ground-glass opacities (GGOs) and consolidation regions) for characterizing severity of COVID-19 disease. The unique aspect of this study involved combining statistical population shape difference atlas models with deep learning for COVID-19 prognosis prediction. Additionally, the study was unique in the employment of novel combination of deep learning derived features from both lung shape differences as well as lung infiltrates. Furthermore, the study presented in this work represents the largest multi-institutional study (Lassau et al., 2021; Wang et al., 2021, p. 19) involving application of AI for COVID-19 severity estimation with N=3230 patients from multiple sites. The presented framework, CovSafeNet yielded an AUC=0.89 on D₁^test, a center on which the model was trained and AUC in the range of 0.66 – 0.77 on external test sets from other institutions. The study also demonstrated that the combined population-atlas and deep learning approach outperformed radiomics (M_R) and clinical based (M_C) machine learning models. Additionally, combining M_C with CovSafeNet resulted in a similar performance (AUC=0.685) as CovSafeNet with no statistically significant difference (p=0.6694). Although the study of post-acute sequelae of SARS CoV-2 (PASC), or “long COVID”, was beyond the scope of the study, the findings from our experiment also holds relevance for this long term complication of COVID-19 disease in light of its rising incidence and major healthcare consequences (Xie et al., 2021). Studies have implicated fibrosis as the inciting factor for lung disease in PASC, contributing to the long-term mortality. These findings support the results from our study (“SARS-CoV-2 infection produces chronic pulmonary epithelial and immune cell dysfunction with fibrosis in mice | Science Translational Medicine,” n.d.).

We acknowledge that our study did have its limitations. No improvement in CovSafeNet’s performance by integrating other clinical features may be attributed to the fact that many of the other important clinical factors such as lactate dehydrogenase (LDH), prothrombin time (PT), albumin (ALB) etc as indicated in the some of the previous studies (Hiremath et al., 2021; Vaidya et al., 2021) were not available across all the datasets (D₁^test, D₂, and D₄). Access to these additional clinical factors can further enhance the performance of CovSafeNet by providing valuable clinical information along with imaging biomarkers. Additionally, the drop in the performance of CovSafeNet on external test sets (D₂, D₃ and D₄) as compared to the test set A_V might be attributed to the fact that CT characteristics (mean intensities within the lung region) of mild COVID-19 patients from D₂, D₃ and D₄ which were found to be statistically different from D₁^test (Supplementary Fig. S7). Additionally, transfer learning techiniques such as fine-tuning or domain specific batch normalization(Liu et al., 2020) of CovSafeNet over a small subset of data from each of these independent test cohorts may also help in capturing distribution and CT characteristics across different institutions, and hence will be explored as part of future work. Additionally, this study mainly focuses on identifying the regions in CT scans with significant shape differences between mild and severe COVID-19 patients and further use the texture and intensity information from those regions via 3D CNN to predict severity of COVID-19 disease. However, characterizing the shape of the lung directly via shape features such as analyzing surface normals and curvature will be considered part of future work. Also, our study was retrospective in nature and to ensure the clinical practicality of CovSafeNet, the tool needs to be validated prospectively by following patients until discharge.

In conclusion, we demonstrated that there are significant differences in lung shape and morphology between COVID-19 patients with mild and severe disease. Further, we showed that AI based quantification of these lung shape differences can help in COVID-19 prognosis. Additionally, integrating these biomarkers with lung infiltrate-based biomarkers through a novel AI framework, we illustrated that COVID-19 prognosis can further be improved. Furthermore, we also showed that CovSafeNet was able to outperform other state-of-the-art approaches including radiomics and clinical based machine learning models. Lastly, fibrosis, as implied in this study is also the main pathophysiology behind PASC. Therefore, future studies would entail exploring the role of lung shape differences among COVID-19 patients as a biomarker in the context of long COVID.

Highlights.

• Lung volumes decrease with COVID-19 severity.

• Lung shape differences at mediastinal surfaces observed based on COVID-19 severity.

• New lung deformation-based imaging biomarker predicts COVID-19 severity.

• Combining lung deformation and lung infiltrate biomarkers; better predict COVID-19 severity.

Abbreviations

COVID-19

Coronavirus disease of 2019

CT

Computed Tomography

AI

Artificial Intelligence

ARDS

Acute respiratory distress syndrome

RT-PCR

Reverse transcription polymerase chain reaction

NLST

National Lung Screening Trial

CNN

convolutional neural network

SDF

Signed distance function

AUC

Area under the receiver operating characteristic curve

DSC

Dice similarity coefficient

GLCM

Gray Level Cooccurence Matrix

GLRLM

Gray Level Run Length Matrix

GLSZM

Gray Level Size Zone Matrix

GLDM

Gray Level Dependence Matrix

NGTDM

Neighboring Gray Tone Difference Matrix

LASSO

Least Absolute Shrinkage and Selection Operator

ROC

receiver operating characteristic curve