Highlights
-
•
Clinically available AI software allows automatic quantification of lung involvement on chest CT scans for COVID-19 patients.
-
•
CT findings combined with clinical variables allow better intensive care unit admission and death prediction.
Keywords: Prognosis, AI, COVID-19, Chest-CT, ICU
Abstract
Objectives
We evaluated the contribution of lung lesion quantification on chest CT using a clinical Artificial Intelligence (AI) software in predicting death and intensive care units (ICU) admission for COVID-19 patients.
Methods
For 349 patients with positive COVID-19-PCR test that underwent a chest CT scan at admittance or during hospitalization, we applied the AI for lung and lung lesion segmentation to obtain lesion volume (LV), and LV/Total Lung Volume (TLV) ratio. ROC analysis was used to extract the best CT criterion in predicting death and ICU admission. Two prognostic models using multivariate logistic regressions were constructed to predict each outcome and were compared using AUC values. The first model (“Clinical”) was based on patients’ characteristics and clinical symptoms only. The second model (“Clinical+LV/TLV”) included also the best CT criterion.
Results
LV/TLV ratio demonstrated best performance for both outcomes; AUC of 67.8% (95% CI: 59.5 - 76.1) and 81.1% (95% CI: 75.7 - 86.5) respectively. Regarding death prediction, AUC values were 76.2% (95% CI: 69.9 - 82.6) and 79.9% (95%IC: 74.4 - 85.5) for the “Clinical” and the “Clinical+LV/TLV” models respectively, showing significant performance increase (+ 3.7%; p-value<0.001) when adding LV/TLV ratio. Similarly, for ICU admission prediction, AUC values were 74.9% (IC 95%: 69.2 - 80.6) and 84.8% (IC 95%: 80.4 - 89.2) respectively corresponding to significant performance increase (+ 10%: p-value<0.001).
Conclusions
Using a clinical AI software to quantify the COVID-19 lung involvement on chest CT, combined with clinical variables, allows better prediction of death and ICU admission.
1. Introduction
The COVID-19 pandemic overwhelmed the healthcare system, and especially the intensive care units (ICU), with its magnitude and rapid spread [1], [2], [3]. Prognosis is known to be partially related to early identification of patients at risk in order to rapidly start the appropriate therapy [4], [5], [6], [7].
Chest CT scan has been playing a key role in diagnosis of COVID-19 [8], [9], [10], [11], [12], [13], [14], and was found essential in disease staging, patient prognosis, and treatment efficacy assessment [15], [16], [17], [18], [19], [20], [21]. In addition, multiple studies demonstrated the additional value of artificial intelligence (AI) algorithms in analyzing chest CT examinations in terms of positive diagnosis and lung involvement quantification. Severity scores related to disease progression [22], [23], [24], [25], [26], [27], [28] and prognostic scores based on clinical or biological variables associated to CT data [29], [30], [31], [32], [33], [34], [35], [36] have been introduced based on the outcomes of AI algorithms. However, few studies focus on the real impact of adding CT to simple clinical markers available directly at the emergency department and even fewer use a clinically available software. The latter is of major importance since it allows use of the results within the clinical workflow.
In this study, we evaluate the additional value of associating the quantification of the COVID-19 related lung involvement assessed by the software on chest CT to already established clinical risk factors and patient characteristics in predicting mortality and ICU admission.
2. Materials & methods
2.1. Population and study description
Based on our local radiologic information system, we retrospectively selected 349 patients hospitalized in one of the public health care facilities of the Hospices Civils Lyon (HCL), Lyon, France from 3rd March to 4th April 2020, presenting a positive COVID RT-PCR test and who underwent a chest CT examination. Pregnant women and children were excluded from the study.
For all patients the following clinical characteristics were recorded at admittance: chest pain, dyspnea (rated as "none", "moderate" or "severe"), ENT symptoms or asymptomatic (absence of the three previous symptoms). The following patient characteristics and comorbidities were also recorded: sex, age, obesity (defined as a BMI >30 kg/m2), hypertension, diabetes, chronic respiratory disease.
Chest CT examinations have been performed either on arrival at the emergency department or within five days after hospitalization.
Data usage policy of our institution in terms of confidentiality, anonymization and security was applied and approval for retrospective analysis of the patients’ data was obtained from the ethical committee.
2.2. CT scan acquisition
Various CT models were used. The scanning parameters were as follows: mean tube voltage of 120.4 ± 11.4 kVp (range: 100-140 kVp), mean slice thickness of 2.3 ± 0.8 mm (range: 0.6 - 3 mm). Scan length, mAs and field of view were adapted for each patient and clinical indication, including exploration of the abdomen when required.
92 out of the 349 examinations (26%) were obtained after intravenous administration of iodinated contrast material. All acquisitions were performed at the end of a deep inspiration.
2.3. AI based software for lung and lung lesion segmentation
All 349 chest CTs were analyzed using the CT Pulmo Auto Results software in the IntelliSpace Portal 11 platform (Philips Healthcare). This is a fully automatic image analysis application able to identify consolidations and ground-glass opacities (GGO) within the lungs, supporting the management of adult patients with suspected or diagnosed COVID-19 pneumonia. An example of the segmentation with quantitative results is shown in Fig. 1.
Fig. 1.
Example of the lung and lung lesion assessment as well as automatic measurements of the “CT Pulmo Auto” AI solution with the quantification of the overall lung volume, overall lesion volume, volume of the left lung, volume of the right lung, volume of ground-glass opacity (GGO) per lung, volume of consolidations per lung.
The automatic processing is performed in three cascaded steps as illustrated in Fig. 2. The left and right lungs are first segmented with exclusion of the main airways including trachea, stem, lobar bronchi, and the main vessels. COVID-19 related lesions are then segmented within the pre-segmented lungs with exclusion of the large bronchi and vessels. Finally, the detected lung lesions are classified voxel-wise as GGO or consolidation. Segmentations and classification steps are achieved using deep learning algorithms [37,38] trained on internal and public datasets [39,40] for which lungs, consolidation and GGO were annotated. The training dataset included both contrast and non-contrast enhanced studies in order for the algorithm to be robust to different protocols.
Fig. 2.
Overview on the full lesion classification pipeline.
To evaluate the performance of the CT Pulmo Auto Results software on the 349 cases of our study, two expert radiologists with more than 20 years of experience performed in consensus a subjective evaluation on 37 randomly selected cases out of 349 patients (approximately 10% of the cohort). Quality of lung and lesion segmentation and lesion classification was rated using five and two levels score respectively described in Table 1.
Table 1.
Quality scoring of lung and lung lesion segmentation and lesions classification.
| Quality scoring | Meaning for segmentation result | |
|---|---|---|
| 5 | Perfect | No manual correction required. |
| 4 | Good | Only minor errors that do not affect the measured anatomy and do not have to be corrected. |
| 3 | Acceptable | Small errors that only slightly affect the measured anatomy. In clinical routine, these errors would not be corrected, though. |
| 2 | Bad | Significant errors that need correction. |
| 1 | Unacceptable | Unusable segmentation even with (more) manual correction. |
| Lesion classification | Meaning for Segmentation Result | |
| Sufficient | The results of lesion classification algorithm (GGO and consolidation) are acceptable in the clinically relevant lesions | |
| Insufficient | The results of lesion classification algorithm (GGO and consolidation) are unacceptable in the clinically relevant lesions | |
2.4. Data analysis and prognostic score
To measure the additional value of the automatically extracted CT lung information in predicting death, the main outcome criterion, we performed a two-phase analysis. To select the best CT criterion, we first assessed the performance of each CT-scan related criterion in predicting death. Once the best CT criterion was identified, we evaluated the benefit on the predictive performance of incorporating this CT criterion to additional clinical factors.
During the first phase, data from all individuals of the study population (n=349) were used. ROC curves [41] were built for each of the following CT-scan criteria: GGO volume/ Total Lung Volume ratio (GGOV/TLV), consolidation volume/Total Lung Volume ratio (CV/TLV), lesion volume (GGO + consolidation)/Total Lung Volume ratio (LV/TLV), GGO volume/Lesion Volume ratio (GGOV/LV). AUC values were estimated, along with the associated 95% Confidence Intervals (CI) using the Delong method [42]. The CT-scan criterion with the highest AUC value was selected and used in the second phase of the analysis.
For this second phase, only individuals with complete data (i.e., with no missing information; n=346) were analyzed. In step 1, a clinical prognostic model for death prediction was developed. The following clinical factors were selected based on prior knowledge about factors associated with the severity of the COVID disease and on clinicians’ expertise: age, sex, obesity, diabetes, hypertension, chronic respiratory disease, dyspnoea, ENT symptoms, chest pain and asymptomatic. A logistic regression model including all the clinical factors listed previously was built and is referred as “Clinical” model in the next step.
In step 2, the previously selected CT-scan criterion was added to the Clinical model and, as recommended by Pepe et al. [43], a likelihood ratio test was performed to assess if the addition of the CT-scan criterion significantly improved the predictive performance (significance threshold set to 0.05). This final model is referred as “Clinical+LV/TLV” model. The predictive performance of the “Clinical” model and “Clinical+LV/TLV” model was quantified using AUC estimates called ‘naïve’ AUC estimates. Then a bootstrap resampling procedure (10000 bootstrap samples, stratified on living status) was applied to obtain optimism-corrected AUC estimates, as recommended by Steyerberg [44]. Finally 95% CIs of the corrected AUC estimates were derived using a two-stage bootstrap sampling procedure [45].
The same analysis strategy was carried out for the secondary outcome of our study corresponding to the admission in an Intensive Care Unit (ICU): the first phase to identify the CT scan criterion and to perform the univariate ROC-curve analyses and all 5 steps of the second phase
All analyses were performed using R software version 4.0.2 [46]. ROC analyses and logistic regressions were performed using the pROC package [47] and the mgcv package [48], respectively.
3. Results
3.1. Characteristics of the population
The baseline characteristics, CT-scan criteria, and outcomes of the 349 individuals of the study population are reported in Table 2. Median age of the patients was 71 years and 199 out of the 349 individuals (57%) were male. Ninety-three patients were admitted in ICU (26.6%) and fifty-three patients (15.2%) died in the hospital. Mean volumetric Computed Tomography Dose Index (CTDIvol) was 9.77 +/- 4.65 mGy (range: 2.1 – 29.5 mGy).
Table 2.
Baseline characteristics, CT-scan criteria, and outcomes. Quantitative variables were described as median (range), and qualitative variables as count (frequency in %). Volumes are measured in mL and volume ratios in percent (%).TLV: Total Lung Volume; LV: Lesions Volume; GGOV: Ground-Glass Opacity Volume; CV: Condensation Volume.
| Non-missing values | Description of the study population (N=349) | |
|---|---|---|
| Characteristics - Medical history | ||
| Age (years) | 349 | 71 (20-99) |
| Male | 349 | 199 (57%) |
| Obesity | 347 | 39 (11.2%) |
| Diabetes | 349 | 73 (20.9%) |
| Hypertension | 349 | 168 (48.1%) |
| Chronic respiratory disease | 349 | 48 (13.8%) |
| Symptoms | ||
| Dyspnea | 348 | |
| None | 96 (27.6%) | |
| Moderate | 171 (49.1%) | |
| Severe | 81 (23.3%) | |
| ORL symptom | 349 | 38 (10.9%) |
| Chest pain | 349 | 31 (8.9%) |
| Asymptomatic | 349 | 7 (2%) |
| CT scan criteria : | ||
| LV / TLV (%) | 349 | 10 (0-65) |
| GGOV / TLV (%) | 349 | 3 (0-41) |
| CV / TLV (%) | 349 | 5 (0-63) |
| GGO/ LV (%) | 349 | 37 (0-100) |
| Outcomes | ||
| Hospital Death | 349 | 53 (15.2%) |
| Intensive Care admission | 349 | 93 (26.6%) |
3.2. Performance of CT-scan criteria and clinical score in predicting death
The AUC values of the ROC curves were reported in the upper part of Table 3 and depicted in Fig. 3 (panel A). They ranged from 56.6% (95% CI: 47.7 - 65.6) for the ratio GGO/LV to 67.8% (95% CI: 59.5 - 76.1) for the LV/TLV ratio. The LV/TLV ratio was thus selected for the second phase of the analysis for the final prognosis score.
Table 3.
Estimation of Area Under the Receiver Operating Characteristic curve (AUC) with 95% Confidence Interval (95% CI), by CT-scan criterion, and for the “Clinical” model and the “Clinical+LV/TLV” model (LV/TLV used as CT-scan measurement). The “Clinical” and “Clinical+LV/TLV” model were obtained from multivariate logistic regressions. Naive AUC: AUC value not corrected for the optimism bias; Corrected AUC: AUC value corrected for the optimism bias. TLV: Total Lung Volume; LV: Lesions Volume; GGOV: Ground-Glass Opacity Volume; CV: Condensation Volume.
| Performance in predicting death AUC in % (95% CI) | Association with ICU admission AUC in % (95% CI) | |
|---|---|---|
| CT scan criteria : | ||
| LV/TLV | 67.8 (59.5 - 76.1) | 81.1 (75.7 - 86.5) |
| GGOV/TLV | 58.5 (48.7 - 68.3) | 67.6 (61.0 - 74.3) |
| CV/TLV | 63.2 (55.0 - 71.4) | 73.2 (66.8 - 79.6) |
| GGOV/LV | 56.6 (47.7 - 65.6) | 55.3 (48.3 - 62.4) |
| Results from multivariate model - Naive AUC | ||
| “Clinical” | 81.2 (75.7 - 86.8) | 78.9 (73.5 - 84.3) |
| “Clinical + LV/TLV” | 84.5 (79.3 - 89.7) | 87.7 (83.6 - 91.9) |
| Results from multivariate model - Corrected AUC | ||
| “Clinical” | 76.2 (69.9 - 82.6) | 74.9 (69.2 - 80.6) |
| “Clinical+ LV/TLV” | 79.9 (74.4 - 85.5) | 84.8 (80.4 - 89.2) |
Fig. 3.
Receiver Operating Characteristic ROC curve for the prediction of death at hospital. Left panel A: ROC curve by CT-scan criterion. Right panel B: ROC curve of the “Clinical” prognostic model (dashed red line) and of the “Clinical + LV/TLV” model using the ratio Lesions Volume/Total Lung Volume (solid black line). AUC were 76.2 and 79.9 respectively. TLV: Total Lung Volume; LV: Lesion Volume; GGOV: Ground-Glass Opacity Volume; CV: Condensation Volume.
The results of the multivariate logistic regressions for the “Clinical” and “Clinical+LV/TLV” models are shown in table 4. The main predictors of death for the clinical model were age with an Odds-ratio for a 10-year increase estimated at 2.27 (95%CI: 1.70-3.05) and dyspnea, for which the odds-ratios for moderate and severe symptoms (as compared to the absence of dyspnea) were 1.82 (95%CI: 0.75-4.44) and 2.88 (95CI%: 1.09-7.60), respectively. For the “Clinical+LV/TLV” model, the odds-ratio for a 10-year increase in age was estimated at 2.51 (95%CI:1.81-3.48) and age had a high prognostic value. However, the prognostic value of dyspnea was reduced as compared to the “Clinical” model. The odds-ratio for a 10-point increase in the LV/TLV ratio was estimated at 1.73(95% CI:1.35-2.22) and adding the LV/TLV ratio to the clinical factors significantly improved the predictive performance of the model (p-value <0.001). The ROC curves of the “Clinical” model and of the “Clinical+LV/TLV” model are provided in Fig. 3 (panel B). The optimism-corrected values of AUC (Table 3, lower part, left column) were 76.2% (95% CI: 69.9 - 82.6) and 79.9% (95% CI 74.4 - 85.5) for the “Clinical” model and “Clinical+LV/TLV “model, respectively.
Table 4.
Results from the multivariate logistic regressions to predict death at hospital, expressed as odds-ratio (95% Confidence Interval CI). Analyses were restricted to complete cases (n=346 individuals). “Clinical” model: multivariate model including clinical factors; “Clinical+LV/TLV” model: multivariate model including the Lesions Volume / Total Lung Volume ratio (LV/TLV) in addition to clinical factors. TLV: Total Lung Volume; LV: Lesion Volume.
| “Clinical” model Odds-ratio (95% CI) | “Clinical+LV/TLV” model Odds-ratio (95% CI) | |
|---|---|---|
| Intercepta | 0.09(0.04-0.22) | 0.04(0.02-0.12) |
| Characteristics - Medical history | ||
| Age for a 10-year increase | 2.27(1.70-3.05) | 2.51(1.81-3.48) |
| Female | 0.74(0.38-1.46) | 0.68(0.33-1.40) |
| Obesity | 1.73(0.59-5.12) | 1.58(0.48-5.15) |
| Diabetes | 1.35(0.62-2.92) | 1.48(0.65-3.36) |
| Hypertension | 0.82(0.41-1.63) | 0.90(0.44-1.83) |
| Chronic respiratory disease | 0.97(0.38-2.47) | 1.45(0.52-3.99) |
| Symptoms | ||
| Dyspnea | ||
| None | 1 | 1 |
| Moderate | 1.82(0.75-4.44) | 1.08(0.42-2.80) |
| Severe | 2.88(1.09-7.60) | 1.60(0.57-4.52) |
| ORL symptom | 0.25(0.05-1.24) | 0.26(0.05-1.26) |
| Chest pain | 0.68(0.14-3.38) | 1.07(0.21-5.49) |
| Asymptomatic | 0.00(0.00-Inf) | 0.00(0.00-Inf) |
| CT scan criterion | ||
| LV/TLV for a 10-point increase | - | 1.73(1.35-2.22) |
Values for Intercept corresponded to the odds for male individuals aged 70, without any symptom or comorbidity, and with a ratio LV/TLV equal to 0% (“Clinical+LV/TLV” model only).
Thus, the predictive performance significantly increased by + 3.7% when adding the LV/TLV ratio to the clinical factors.
3.3. Association with ICU admission
The ROC curves yielded by CT-scan criteria are reported in Fig. 4 (panel A). The association between CT-scan criteria and ICU admission varied greatly: the AUC ranged from 55.3% (IC 95%: 48.3 - 62.4) for the GGO Volume/LV ratio to 81.1% (IC 95%: 75.7 - 86.5) for the LV/TLV ratio (Table 3, right column). The LV/TLV ratio was thus selected for the second phase of the analysis.
Fig. 4.
Receiver Operating Characteristic ROC curve to evaluate the association with ICU admission. Left panel A: ROC curve by CT-scan criterion. Right panel B: ROC curve of the “Clinical” model (dashed red line) and of the “Clinical+LV/TLV” model using the ratio the ratio Lesions Volume/Total Lung Volume (solid black line). AUC were 74.9 and 84.8 respectively. TLV: Total Lung Volume; LV: Lesions Volume; GGOV: Ground-Glass Opacity Volume; CV: Condensation Volume.
The results for the clinical multivariate model for ICU admission are shown in Table 5 (left column). Age was globally associated with ICU admission, but the strength of this association varied across the range of ages. For example, ICU admission was similar for individuals aged 70 years old compared to 60 years old, but much less frequent when comparing ages 90 versus 80 years. The main factors associated with ICU admission were the presence of dyspnea with an odds-ratio for moderate vs. no symptom of 5.05(IC 95%: 2.34-10.90), and gender with an odds-ratio for female versus male of 0.51(IC 95%: 0.29-0.90). For the “Clinical+LV/TLV” model, the odds-ratio for a 10-point increase in the LV/TLV ratio was 2.52(1.94-3.29; p-value < 0.001).
Table 5.
Results from the multivariate logistic regressions to evaluate the association with admission in an Intensive Care Unit (ICU), expressed as odds-ratio (95% Confidence Interval CI). Analyses were restricted to complete cases (n=346 individuals). “Clinical” model: multivariate model including clinical factors; “Clinical+LV/TLV” model: multivariate model including the Lesions Volume/Total Lung Volume ratio (LV/TLV) in addition to clinical factors. TLV: Total Lung Volume; LV: Lesion Volume.
| “Clinical” model Odds-ratio (95% CI) | “Clinical+LV/TLV” model Odds-ratio (95% CI) | |
|---|---|---|
| Intercepta | 0.14(0.07-0.32) | 0.04(0.02-0.11) |
| Characteristics - Medical history | ||
| Age: 40y Vs. 50y | 1.44(0.86-2.41) | 1.36(0.76-2.44) |
| Age: 60y Vs. 70y | 1.02(0.61-1.70) | 1.04(0.58-1.89) |
| Age: 80y Vs. 90y | 0.19(0.07-0.48) | 0.11(0.03-0.36) |
| Female | 0.51(0.29-0.90) | 0.49(0.26-0.95) |
| Obesity | 1.01(0.44-2.33) | 0.91(0.35-2.39) |
| Diabetes | 1.67(0.85-3.30) | 2.15(0.98-4.72) |
| Hypertension | 0.79(0.43-1.45) | 0.87(0.43-1.74) |
| Chronic respiratory disease | 0.30(0.11-0.77) | 0.44(0.15-1.30) |
| Symptoms | ||
| Dyspnea | ||
| None | 1 | 1 |
| Moderate | 5.05(2.34-10.90) | 2.48(1.07-5.76) |
| Severe | 3.39(1.41-8.15) | 1.45(0.55-3.79) |
| ORL symptom | 0.58(0.23-1.46) | 0.48(0.17-1.37) |
| Chest pain | 0.79(0.28-2.25) | 1.49(0.49-4.55) |
| Asymptomatic | 0.00(0.00-Inf) | 0.00(0.00-Inf) |
| CT scan criterion | ||
| LV/TLV for a 10-point increase | - | 2.52(1.94-3.29) |
Values for Intercept corresponded to the odds for male individuals aged 70, without any symptom or comorbidity, and with a measure of the ratio LV/TLV equal to 0% (“Clinical+LV/TLV” model only).
The ROC curves of the “Clinical” model only and of the “Clinical+LV/TLV” model are provided in Fig. 4 (panel B). The optimism-corrected values of AUC (Table 3, lower part, right column) were 74.9% (IC 95%: 69.2 - 80.6) and 84.8% (IC 95%: 80.4 - 89.2) respectively.
Thus, the predictive performance significantly increased by +10% when adding the LV/TLV ratio to the clinical factors.
3.4. Performances of the CT Pulmo Auto Results software
Regarding lung segmentation, the median score value was 5 (25th-75th percentile: [4.5-5]). 36 out of 37 cases received a score of 3 or more, i.e. clinically acceptable values. Similarly, for lesion segmentation, the median score value was 4 (25th-75th percentile: [4], [5]). 35 out of 37 cases were scored larger than 3. Regarding lesions classification, 30 out of 37 cases were rated sufficient.
4. Discussion
In this study, we showed that adding automatically extracted quantification information of lung involvement on chest CT data to clinical variables improves death and intensive care admission prediction. Specifically, the results of the multivariate model showed an AUC increase of +3.7% (from 76.2% to 79.9%) in predicting death by adding the CT scan lung involvement quantification to the clinical data. Similarly, for ICU admission, the AUC increased by approximately 10% from 74.9% to 84.8%.
These data are consistent with other studies, including the one conducted by Lassau et al [49], demonstrating that adding CT data by using an AI tool to clinical and biological data increased the AUC by 3% as compared to the model including only clinical and biological features. In the same way, the study by Shiri et al [50] showed that using a combined model of clinical and radiomic data from the CT scan increased the AUC from 87% (clinical model) to 95% (combined model) for survival prediction. Finally, in the study by Zhang et al [51], the combination of the two models increased the AUC for estimating progression to critical disease from 84% to 91%. .
These differences concerning the added value of CT features to clinical variables across the different studies may be explained by differences in population characteristics. Indeed, in alignment with previous studies, we found a significant association between clinical risk factors and a severe form of the disease with a major effect of age [6,49,52,53] as a predictor of mortality and dyspnea [30,49,53] as a predictor of admission to ICU. Contrary to several other studies, we did not find any statistical association between disease severity and the presence of obesity, diabetes [54,55], hypertension [54], or chronic respiratory disease [54]. This may be due to differences in the studied population across studies. For example, the population in our study is mainly composed of elderly and hospitalized patients, whereas the evaluation population of the Lassau et al study has a large proportion of cancer immunosuppressed patients. This may result in a modification of the contribution of CT data to clinical data, which is positive but limited in the Lassau study, whereas it is clearly significant in our study.
Regarding lung lesion classification, on the ROC analysis, we highlight that the ratios GGOV/TLV and CV/TLV and the ratio GGOV/LV is less predictive than the ratio LV/TLV for mortality or ICU admission. This may be related to two different factors. Firstly, the CT attenuation values between GGO and consolidation are overlapping making even the manual classification subjective and extremely challenging. This may also explain the relatively low performance of the algorithm in lesions classification. Secondly, the evolution of the disease from GGO to consolidation is not linear. Indeed, in certain cases, the evolution of the disease is fast presenting a progression of the lesions directly from the early stage (GGO) to the late phase called fibrosis, skipping the consolidation phase [56,57].
The major strength of our study concerns the use of the CT Pulmo Auto Results software, a commercial AI tool released for clinical use, accessible via the Philips IntelliSpace Portal Platform. Having AI tools integrated into the clinical workflow is indeed of major importance, as specified in the CLAIM (Checklist for Artificial Intelligence in Medical Imaging, Mongan et al [58] item 39). This is not the case for several other studies [59], [60], [61], [62] that used AI lung segmentation tools difficult to access in daily hospital practice. Besides the prognostic value of automatic CT analysis that we reported in this study, access to a systematic quantification of lung involvement is also of importance for the overall management of COVID patients and to assess the severity and progression of the disease as stated by the Radiological Society of North America [63], the Fleischner Society [64], French Society of Radiology [65], Chinese National Health Commission [66], and The World Health Organization (WHO) [67] .
Our study has some limitations. First, a selection bias could be generated as the recruitment concerns severely ill patients admitted for hospitalization. Second, the retrospective nature of this study and the short period of patient inclusion at the beginning of the pandemic with a higher proportion of scans performed due to limited availability of PCR at some sites should be considered. Finally, our study concerns the original COVID-19 variant and might not be extrapolated to other variants.
In conclusion, the combination of clinical factors and the lung involvement ratio measured on regular chest CT examinations using a clinical AI software allows better prediction of death and ICU admission for COVID-19 patients than clinical variables alone.
5. Author contributions
Conceptualization: LB, MR, AD, Data curation: AG, AV, LB, SG, Formal Analysis: AG, AV, LB, SG, Funding acquisition: LB, SG, Investigation; AG, AV, LB, SG, Methodology: MB-D, MR, LB, Project administration: AM, SG, Resources: EG, LR, AV, ON, HC, A-R, PJ, PR, TK, JR, KT, MD, AS, MB-D, SAS-M, SG, AM, FT, JP, OR, LM, FC, PD, AD, MR, LB, Software: LB, AV, ON, HC, AD-R, PJ, PR, TK, SG, Supervision: LB, MR, Validation: EG, LR, AV, ON, HC, A-R, PJ, PR, TK, JR, KT, MD, AS, MB-D, SAS-M, SG, AM, FT, JP, OR, LM, FC, PD, AD, MR, LB, Visualization: SAS-M, LB, AV, MR, Writing – original draft: EG, LB, AV, MR, Writing – review & editing: EG, LR, AV, ON, HC, A-R, PJ, PR, TK, JR, KT, MD, AS, MB-D, SAS-M, SG, AM, FT, JP, OR, LM, FC, PD, AD, MR, LB
Declaration of Competing Interests
The authors declare the following competing interest: Anna Vlachomitrou, Olivier Nempont, Heike Carolus, Alexander Schmidt-Richberg, Peng Jin, Pedro Rodrigues are Tobias Klinder are employees of Philips Healthcare.
Acknowledgments
We acknowledge the “Consortium COVID HCL” for its support for this publication. We acknowledge Morgane Bouin, Cécile Rémy, Hayette Djouadi, Hind Behlouli, and Sabine Debeer for their help in data collection.
References
- 1.Zhu N, Zhang D, Wang W, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. 2020;382:727–733. doi: 10.1056/NEJMoa2001017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Fisher D, Heymann D. Q&A: the novel coronavirus outbreak causing COVID-19. BMC Med. 2020;18:57. doi: 10.1186/s12916-020-01533-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Khan M, Adil SF, Alkhathlan HZ, et al. COVID-19: a global challenge with old history, epidemiology and progress so far. Molecules. 2020;26 doi: 10.3390/molecules26010039. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wang D, Hu B, Hu C, et al. Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan, China. JAMA. 2020;323:1061–1069. doi: 10.1001/jama.2020.1585. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. 2020;395:497–506. doi: 10.1016/S0140-6736(20)30183-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Yang X, Yu Y, Xu J, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study. Lancet Respir Med. 2020;8:475–481. doi: 10.1016/S2213-2600(20)30079-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Phua J, Weng L, Ling L, et al. Intensive care management of coronavirus disease 2019 (COVID-19): challenges and recommendations. Lancet Respir Med. 2020;8:506–517. doi: 10.1016/S2213-2600(20)30161-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ai T, Yang Z, Hou H, et al. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology. 2020;296:E32–E40. doi: 10.1148/radiol.2020200642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fang Y, Zhang H, Xie J, et al. Sensitivity of chest CT for COVID-19: comparison to RT-PCR. Radiology. 2020;296:E115–E117. doi: 10.1148/radiol.2020200432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Huang P, Liu T, Huang L, et al. Use of chest CT in combination with negative RT-PCR assay for the 2019 novel coronavirus but high clinical suspicion. Radiology. 2020;295:22–23. doi: 10.1148/radiol.2020200330. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Tang Y-W, Schmitz JE, Persing DH, Stratton CW. Laboratory diagnosis of COVID-19: current issues and challenges. J Clin Microbiol. 2020;58 doi: 10.1128/JCM.00512-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Long C, Xu H, Shen Q, et al. Diagnosis of the coronavirus disease (COVID-19): rRT-PCR or CT? Eur J Radiol. 2020;126 doi: 10.1016/j.ejrad.2020.108961. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Xie X, Zhong Z, Zhao W, et al. Chest CT for typical coronavirus disease 2019 (COVID-19) pneumonia: relationship to negative RT-PCR testing. Radiology. 2020;296:E41–E45. doi: 10.1148/radiol.2020200343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ducray V, VLACHOMITROU AS, Bouscambert-Duchamp M, et al (2020) COVID-19: Role of chest CT in the triage of patients in multiple emergency departments: experience report from a large French university hospital. 10.13140/RG.2.2.31810.40649 [DOI] [PMC free article] [PubMed]
- 15.Bhandari S, Rankawat G, Bagarhatta M, et al. Clinico-radiological evaluation and correlation of CT chest images with progress of disease in COVID-19 patients. J Assoc Physicians India. 2020;68:34–42. [PubMed] [Google Scholar]
- 16.Abbasi B, Akhavan R, Ghamari Khameneh A, et al. Evaluation of the relationship between inpatient COVID-19 mortality and chest CT severity score. Am J Emerg Med. 2020 doi: 10.1016/j.ajem.2020.09.056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Francone M, Iafrate F, Masci GM, et al. Chest CT score in COVID-19 patients: correlation with disease severity and short-term prognosis. Eur Radiol. 2020;30:6808–6817. doi: 10.1007/s00330-020-07033-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Li Y, Xia L. Coronavirus disease 2019 (COVID-19): role of chest CT in diagnosis and management. AJR Am J Roentgenol. 2020;214:1280–1286. doi: 10.2214/AJR.20.22954. [DOI] [PubMed] [Google Scholar]
- 19.Ye Z, Zhang Y, Wang Y, et al. Chest CT manifestations of new coronavirus disease 2019 (COVID-19): a pictorial review. Eur Radiol. 2020;30:4381–4389. doi: 10.1007/s00330-020-06801-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Feng Z, Yu Q, Yao S, et al. Early prediction of disease progression in COVID-19 pneumonia patients with chest CT and clinical characteristics. Nat Commun. 2020;11:4968. doi: 10.1038/s41467-020-18786-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Zhao W, Zhong Z, Xie X, et al. Relation between chest CT findings and clinical conditions of coronavirus disease (COVID-19) pneumonia: a multicenter study. AJR Am J Roentgenol. 2020;214:1072–1077. doi: 10.2214/AJR.20.22976. [DOI] [PubMed] [Google Scholar]
- 22.Wang S, Kang B, Ma J, et al (2020) A deep learning algorithm using CT images to screen for corona virus disease (COVID-19). Infectious Diseases (except HIV/AIDS) [DOI] [PMC free article] [PubMed]
- 23.Meng L, Dong D, Li L, et al. A deep learning prognosis model help alert for COVID-19 patients at high-risk of death: a multi-center study. IEEE J Biomed Health Inform. 2020;24:3576–3584. doi: 10.1109/JBHI.2020.3034296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Wang S, Zha Y, Li W, et al. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis. Eur Respir J. 2020;56 doi: 10.1183/13993003.00775-2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Li L, Qin L, Xu Z, et al. Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: evaluation of the diagnostic accuracy. Radiology. 2020;296:E65–E71. doi: 10.1148/radiol.2020200905. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lessmann N, Sánchez CI, Beenen L, et al. Automated assessment of COVID-19 reporting and data system and chest CT severity scores in patients suspected of having COVID-19 using artificial intelligence. Radiology. 2021;298:E18–E28. doi: 10.1148/radiol.2020202439. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Kwon YJ (Fred), Toussie D, Finkelstein M, et al. Combining initial radiographs and clinical variables improves deep learning prognostication in patients with COVID-19 from the emergency department. Radiol Artif Intell. 2021;3 doi: 10.1148/ryai.2020200098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Gieraerts C, Dangis A, Janssen L, et al. Prognostic value and reproducibility of AI-assisted analysis of lung involvement in COVID-19 on low-dose submillisievert chest CT: sample size implications for clinical trials. Radiol Cardiothorac Imaging. 2020;2 doi: 10.1148/ryct.2020200441. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Levy TJ, Richardson S, Coppa K, et al. Development and validation of a survival calculator for hospitalized patients with COVID-19. MedRxiv. 2020 doi: 10.1101/2020.04.22.20075416. [DOI] [Google Scholar]
- 30.Liang W, Liang H, Ou L, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. 2020;180:1081–1089. doi: 10.1001/jamainternmed.2020.2033. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Ji D, Zhang D, Xu J, et al. Prediction for progression risk in patients with COVID-19 pneumonia: the CALL score. Clin Infect Dis Off Publ Infect Dis Soc Am. 2020;71:1393–1399. doi: 10.1093/cid/ciaa414. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with COVID-19 using the ISARIC WHO clinical characterization protocol: development and validation of the 4C mortality score. BMJ. 2020;370:m3339. doi: 10.1136/bmj.m3339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of COVID-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi: 10.1136/bmj.m1328. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Lippi G, Plebani M. Laboratory abnormalities in patients with COVID-2019 infection. Clin Chem Lab Med. 2020;58:1131–1134. doi: 10.1515/cclm-2020-0198. [DOI] [PubMed] [Google Scholar]
- 35.Chassagnon G, Vakalopoulou M, Battistella E, et al. AI-driven quantification, staging and outcome prediction of COVID-19 pneumonia. Med Image Anal. 2021;67 doi: 10.1016/j.media.2020.101860. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Bartoli A, Fournel J, Maurin A, et al. Value and prognostic impact of a deep learning segmentation model of COVID-19 lung lesions on low-dose chest CT. Res Diagn Interv Imaging. 2022;1 doi: 10.1016/j.redii.2022.100003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Brosch T, Saalbach A. Medical imaging 2018: image processing. International Society for Optics and Photonics; 2018. Foveal fully convolutional nets for multi-organ segmentation. [Google Scholar]
- 38.Milletari F, Navab N, Ahmadi S-A. 2016 Fourth International Conference on 3D Vision (3DV) 2016. V-net: fully convolutional neural networks for volumetric medical image segmentation; pp. 565–571. [Google Scholar]
- 39.Simpson AL, Antonelli M, Bakas S, et al (2019) A large annotated medical image dataset for the development and evaluation of segmentation algorithms. ArXiv190209063 Cs Eess
- 40.Armato SG, McLennan G, Bidaut L, et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans: the LIDC/IDRI thoracic CT database of lung nodules. Med Phys. 2011;38:915–931. doi: 10.1118/1.3528204. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press; Oxford; New York: 2003. [Google Scholar]
- 42.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–845. [PubMed] [Google Scholar]
- 43.Pepe MS, Kerr KF, Longton G, Wang Z. Testing for improvement in prediction model performance. Stat Med. 2013;32:1467–1482. doi: 10.1002/sim.5727. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Steyerberg EW. Clinical prediction models. Springer International Publishing; Cham: 2019. Validation of prediction models; pp. 329–344. [Google Scholar]
- 45.Steyerberg EW, Bleeker SE, Moll HA, et al. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441–447. doi: 10.1016/s0895-4356(03)00047-7. [DOI] [PubMed] [Google Scholar]
- 46.R Core Team . R Foundation for Statistical Computing; Vienna, Austria: 2020. A language and environment for statistical computing.https://www.R-project.org/ [Google Scholar]
- 47.Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform. 2011;12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Wood SN. 2nd ed. Chapman and Hall/CRC; 2017. Generalized additive models: an introduction with R. [Google Scholar]
- 49.Lassau N, Ammari S, Chouzenoux E, et al. Integrating deep learning CT-scan model, biological and clinical variables to predict severity of COVID-19 patients. Nat Commun. 2021;12:634. doi: 10.1038/s41467-020-20657-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Shiri I, Sorouri M, Geramifar P, et al. Machine learning-based prognostic modeling using clinical data and quantitative radiomic features from chest CT images in COVID-19 patients. Comput Biol Med. 2021;132 doi: 10.1016/j.compbiomed.2021.104304. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Zhang K, Liu X, Shen J, et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell. 2020;181:1423–1433. doi: 10.1016/j.cell.2020.04.045. e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Ruan Q, Yang K, Wang W, et al. Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China. Intensive Care Med. 2020;46:846–848. doi: 10.1007/s00134-020-05991-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Li K, Wu J, Wu F, et al. The clinical and chest CT features associated with severe and critical COVID-19 pneumonia. Investig Radiol. 2020;55:327–331. doi: 10.1097/RLI.0000000000000672. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Wang B, Li R, Lu Z, Huang Y. Does comorbidity increase the risk of patients with COVID-19: evidence from meta-analysis. Aging. 2020;12:6049–6057. doi: 10.18632/aging.103000. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Gupta R, Ghosh A, Singh AK, Misra A. Clinical considerations for patients with diabetes in times of COVID-19 epidemic. Diabetes Metab Syndr. 2020;14:211–212. doi: 10.1016/j.dsx.2020.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Li M, Lei P, Zeng B, et al. Coronavirus disease (COVID-19): spectrum of CT findings and temporal progression of the disease. Acad Radiol. 2020;27:603–608. doi: 10.1016/j.acra.2020.03.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Pan F, Ye T, Sun P, et al. Time course of lung changes at chest CT during recovery from coronavirus disease 2019 (COVID-19) Radiology. 2020;295:715–721. doi: 10.1148/radiol.2020200370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell. 2020;2 doi: 10.1148/ryai.2020200029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Amyar A, Modzelewski R, Ruan S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Epidemiology. 2020 doi: 10.1016/j.compbiomed.2020.104037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Fan D-P, Zhou T, Ji G-P, et al. Inf-net: automatic COVID-19 lung infection segmentation from CT images. IEEE Trans Med Imaging. 2020;39:2626–2637. doi: 10.1109/TMI.2020.2996645. [DOI] [PubMed] [Google Scholar]
- 61.Laradji I, Rodriguez P, Mañas O, et al (2020) A weakly supervised consistency-based learning method for COVID-19 segmentation in CT images. ArXiv200702180 Cs Eess
- 62.Shi F, Wang J, Shi J, et al. Review of artificial intelligence techniques in imaging data acquisition, segmentation, and diagnosis for COVID-19. IEEE Rev Biomed Eng. 2021;14:4–15. doi: 10.1109/RBME.2020.2987975. [DOI] [PubMed] [Google Scholar]
- 63.Simpson S, Kay FU, Abbara S, et al. Radiological Society of North America Expert Consensus Statement on reporting chest CT findings related to COVID-19. Endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA—Secondary Publication. J Thorac Imaging. 2020;35:219–227. doi: 10.1097/RTI.0000000000000524. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Rubin GD, Ryerson CJ, Haramati LB, et al. The role of chest imaging in patient management during the COVID-19 pandemic: a Multinational Consensus Statement from the Fleischner Society. Radiology. 2020;296:172–180. doi: 10.1148/radiol.2020201365. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Société d’Imagerie Thoracique (SIT) La société d’Imagerie Thoracique propose un Compte-Rendu Structuré de scanner Thoracique pour les patients suspects de COVID-19. SFR E-Bull. 20202020 https://ebulletin.radiologie.fr/actualites-COVID-19/societe-dimagerie-thoracique-propose-compte-rendu-structure-scanner-thoracique [Google Scholar]
- 66.National Health Commission Diagnosis and treatment protocol of COVID-19 pneumonia (Trial Version 8) Infect Dis Inf. 2020;33:289–296. doi: 10.3969/j.issn.1007-8134.2020.04.001. [DOI] [Google Scholar]
- 67.Akl EA, Blažić I, Yaacoub S, et al. Use of chest imaging in the diagnosis and management of COVID-19: a WHO rapid advice guide. Radiology. 2021;298:E63–E69. doi: 10.1148/radiol.2020203173. [DOI] [PMC free article] [PubMed] [Google Scholar]