Abstract
Background
There are conflicting data regarding the diagnostic performance of Chest computed tomography (CT) for COVID-19 pneumonia. Disease extent on CT has been reported to influence prognosis.
Purpose
To create a large publicly available dataset and assess the diagnostic and prognostic value of CT in COVID-19 pneumonia.
Materials and Methods
This multicenter observational retrospective cohort study (ClinicalTrials.gov: NCT04355507) involved 20 French university hospitals. Eligible subjects presented at the emergency departments of the hospitals involved between March 1st and April 30th, 2020 and underwent both thoracic CT and RT-PCR for suspected COVID-19 pneumonia. CT images were read blinded to initial reports, RT-PCR, demographic characteristics, clinical symptoms, and outcome. Readers classified CT scans as positive or negative for COVID-19, based on criteria published by the French Society of Radiology. Multivariable logistic regression was used to develop a model predicting severe outcome (intubation or death) at 1-month follow-up in subjects positive for both RT-PCR and CT, using clinical and radiological features.
Results
Of 10,930 subjects screened for eligibility, 10,735 (median age 65 years, interquartile range, 51–77 years; 6,147 men) were included and 6,448 (60.0%) had a positive RT-PCR result. With RT-PCR as reference, the sensitivity and specificity and CT were 80.2% (95%CI: 79.3, 81.2) and 79.7% (95%CI: 78.5, 80.9), respectively with strong agreement between junior and senior radiologists (Gwet’s AC1 coefficient: 0.79) Of all the variables analysed, the extent of pneumonia on CT (OR 3.25, 95%CI: 2.71, 3.89) was the best predictor of severe outcome at one month. A score based solely on clinical variables predicted a severe outcome with an AUC of 0.64 (95%CI: 0.62, 0.66), improving to 0.69 (95%CI: 0.6, 0.71) when it also included the extent of pneumonia and coronary calcium score on CT.
Conclusion
Using pre-defined criteria, CT reading is not influenced by reader’s experience and helps predict the outcome at one month.
See also the editorial by Rubin.
Summary
Using RT-PCR as the reference standard in 10735 subjects with suspected COVID-19 pneumonia, CT was 80% accurate and was also predictive of death or need for intubation.
Key Results
■ Using predefined criteria and RT-PCR as the reference standard in 10735 subjects with suspected COVID-19 pneumonia, CT diagnostic accuracy for COVID-19 was 80%, which increased to 86% after 5 days of symptoms.
■ The extent of pneumonia at CT (OR, 3.25) was the best predictor of severe outcome (intubation or death) at one month.
■ CT diagnostic accuracy was not influenced by reader experience (Gwet's AC1 coefficient, 0.79).
Introduction
The SARS-Cov-2 pandemic has caused more than 1.6 million deaths worldwide by the end of 2020 and has overwhelmed healthcare resources in most countries. SARS-Cov-2 infects the airway epithelial cells according to their expression of ACE2 receptors (1), with consequences ranging from no or few symptoms to acute respiratory distress, the main cause of death. COVID-19 pneumonia resulting from SARS-Cov-2 infection is characterized by ground glass opacities not always detectable on chest radiography (2). The advice from the European Society of Radiology was to use CT in subjects developing respiratory symptoms (3). The use of CT in managing SARS-Cov-2 pandemic has been variable around the world. CT has been used as a screening test in China, following initial reports of sensitivity as high as 97% (4). Conversely, the American College of Radiologists estimated that imaging findings in COVID-19 were not specific, overlapping with other infections (5). The risk that CT scanners could become vectors of infection and that the use of CT might have a disproportionate risk–benefit ratio was also raised (6,7). In northern Italy, CT helped patient triage by discarding from the COVID-19 protocol 29% of individuals who had normal or non-COVID-19 abnormalities on CT (8). To determine the diagnostic value of CT more clearly, one objective of the STOIC project, collecting CT scans and 1-month outcomes in more than 10,000 individuals, was to evaluate the sensitivity and specificity of CT interpreted by readers of differing levels of experience, with RT-PCR as the reference standard. Another objective was to assess the influence of the extent of pneumonia and CT features related to comorbidities, on subjects’ outcomes at 1-month follow-up.
Materials and Methods
Study Design and Participants
The protocol of this multicenter observational retrospective cohort study can be accessed via ClinicalTrials.gov with identifier NCT04355507. Its design and execution were in accordance with the Standards for Reporting of Diagnostic Accuracy (STARD) initiative (9).
The STOIC (Study of ThOracic computed tomography In Covid-19) project aimed to build a dataset of at least 10,000 CT scans from individuals with suspected COVID-19 pneumonia, evaluated during the first wave of the SARS-Cov-2 pandemic in France. The collected data and list of image annotations are described in Appendix E1-E3, together with the license agreement for data sharing. The project involved 20 university hospitals: 15 from Assistance Publique des Hôpitaux de Paris and 5 from other cities (Strasbourg, Lyon, Rennes, and Montpellier). Subjects were eligible if they had both thoracic CT scans and RT-PCR at initial presentation, between March 1st and April 30th, 2020. According to the recommendations of the French Health Authority (HAS), CT was not performed as a screening test in subjects with no thoracic symptoms but only in those presenting at the hospital with dyspnea and/or desaturation as measured by pulse oximetry (10).
This study was approved by the Ethics committee of Cochin hospital in Paris, which waived the need for written informed consent.
Procedures
Thoracic CT
All participating hospitals were equipped with multi-detector CT scanners from different manufacturers, allowing volumetric high-resolution CT acquisitions of the whole thorax. CT acquisitions were performed without contrast administration except when pulmonary embolism was suspected as a confounding diagnosis to COVID-19 pneumonia at presentation (11).
CT readings
After anonymized export, CT images were independently read using a dedicated 3D image viewer web application by 7 junior (IS, TL, AM, FB, SD, CH, CJ) and 13 senior (MPR, SBou, CDM, DM, GC, ML, SBen, SM, MPD, MO, SBom, MEH, IP) chest radiologists who were blinded to initial reports, RT-PCR results, demographic characteristics, clinical symptoms, and outcomes. Senior radiologists had at least 4-years of experience in chest imaging (Table E1). Readers were asked to classify CT scans as either positive or negative for COVID-19 based on the criteria of the COVID-19 structured report of the French Society of Radiology (12). Positive diagnosis required the presence of predominantly subpleural ground glass opacities and the absence of mucoid impactions, bronchiolar nodules, or focal consolidation suggesting bacterial infection. Only the initial CT, performed at presentation, was analyzed. CT scans were only read by one radiologist, except those which were unintentionally exported twice and received different anonymization numbers. Again, some CT scans not marked as read were re-annotated by the same reader. This provided the opportunity to evaluate inter and intra reader agreement in a post-hoc analysis. For CT scans they considered as COVID-19 positive, readers had to visually quantify lung disease extent using a 5-point scale. They also evaluated lung emphysema on a 5-point scale (0%, <25%, 25-50%, 50-75%, >75%), severity of coronary artery calcifications using a semi-quantitative visual method (13), and measured the amount of chest wall fat anterior to the sternum.
Data collected and reference standard
Patient demographic characteristics were retrieved from the electronic medical records, as was the time elapsed since the onset of symptoms, the need for oxygen supplementation at presentation and any pre-existing comorbidities such as diabetes, hypertension, coronary artery disease, and any other pre-existing cardiovascular or respiratory disease. Initial (or repeated) RT-PCR results were collected. For subjects with positive RT-PCR, the outcome at 1 month was analyzed, with unfavorable outcome being defined as death or intubation.
RT-PCR was used as reference standard. If the first RT-PCR result was negative but turned out to be positive in the following 7 days after CT, the patient was considered as positive for SARS-Cov-2 infection.
Statistical analysis
For the power analysis, we considered that with an enrollment target of 10,000 subjects, a disease prevalence of at least 50% and expected CT sensitivity and specificity between 80 and 85%, the width of the confidence intervals for both sensitivity and specificity would be less than 1.0%.
CT diagnostic performance was evaluated by its sensitivity, specificity, positive and negative predictive values, accuracy and area under the curve (AUC) with RT-PCR results as the reference standard. The McNemar test was used for paired comparisons. Cohen's kappa coefficient was used to measure intra-reader agreement for readers who re-annotated the same CT scan. Gwet's AC1 coefficient for multiple readers was used to evaluate inter-reader agreement on CT scans annotated by two different readers.
The analysis was conducted on all included subjects. Continuous data are presented as mean ± standard deviation, while categorical data are summarized as counts and percentages.
For modeling purposes, missing clinical data were handled with multiple imputations by chained equations. The risk model for severe outcome was developed on subjects positive for both RT-PCR and CT, who had unenhanced CT allowing quantifying coronary artery calcifications on CT.
Multivariable logistic regression was used to estimate the risk model. We selected 11 clinical and CT variables to serve as candidate predictors. These included: age, sex, need for oxygen supplementation at presentation, comorbidities (diabetes, hypertension, coronary artery disease, respiratory disease), chest wall fat thickness anterior to the sternum, disease extent, coronary artery ordinal calcium score (Figure 4) and emphysema score. Multiple imputations were performed to handle missing data using the SAS procedure PROC MI. Severity of the disease (subjects who were intubated at one point or deceased were considered as severe cases) was the dependent variable. Stepwise variable selection was applied to select predictors in the final model, with a 0.20 significance level for entry and a 0.05 significance level for retention. Discrimination was assessed by calculating the AUC and goodness of fit by the Hosmer-Lemeshow test. A simplified score was obtained by multiplying the regression coefficient by 5 and rounding to the nearest integer (14).
Figure 4:
CT annotations: Visual scoring of coronary artery calcifications. The coronary artery calcium score was evaluated according to the method by Shemesh et al. (13). Calcification in each of the four main coronary arteries (LM:left main, LAD:left anterior descending, LCX:circumflex, and RCA:right) was categorized as none, mild, moderate, or severe. Calcification was classified as mild when less than one-third of the length of the entire artery showed calcification, moderate when one-third to two-thirds of the artery showed calcification and severe when more than two-thirds of the artery showed calcification.
All statistical analyses were performed with SAS software Version 9.4 (SAS Institute Inc.). A P value of ≤.05 was considered significant for all statistical tests conducted.
Results
Study sample, initial characteristics, and outcome
Of the 10,930 subjects who were assessed for eligibility, 195 were excluded either because they did not fulfill the inclusion criteria or were erroneously included twice (Figure 1). A total of 10,735 subjects (median age 65 years, interquartile range, 51–77 years; 6,147 men) were finally included. The study demographics are presented in Table 1. RT-PCR was positive for 6,448 subjects, corresponding to a disease prevalence of 60.0% during the study period.
Figure 1:
Flow chart of the study sample. RT-PCR: Reverse Transcription–Polymerase Chain Reaction.
Table 1.
Patient Characteristics at Presentation
Of 4,557 subjects with an initial negative RT-PCR test, repeat tests were performed in 1,222 (26.8%). Repeat RT-PCT testing was positive in 271 subjects in the following 7 days, resulting in a sensitivity of first RT-PCR of 95.8% (6,176/6,447) (95% CI: 95.3, 96.3). At 1-month follow up, 84.3% (5,440/6,448) of the PCR positive subjects were alive and discharged from the hospital, 13.6% (881/6,448) had to be intubated at one point and 15.3% (988/6,448) died from complications of their COVID-19 infection.
CT results and diagnostic performance
Of the 6,448 subjects who had positive RT PCR results, 5,174 were judged COVID-19 positive on CT, resulting in an overall sensitivity of 80.2% (95% CI: 79.3, 81.2); 80.6% (95% CI: 79.4, 81.7) when based on senior radiologist reading and 79.6% (95% CI: 77.8, 81.3) for junior reading (p=0.34). Of the 4,287 subjects with negative RT PCR results, 872 were considered COVID-19 positive on CT. The calculated specificity of CT was 79.7% (95% CI: 78.5, 80.9) overall; 79.5% (95% CI: 78.0, 80.9) for senior and 80.1% (95% CI: 78.0, 82.3) for junior readings, respectively (p=0.61). Of the 1,041 subjects with a first negative PCR result but signs of pneumonia on CT, 430 (41.3%) had further RT-PCR tests which turned out to be positive in 169 (39.3%). After 5 days of symptoms, the sensitivity of CT was improved to 88.2% (95% CI: 87.0, 89.4). Excluding subjects with positive RT-PCR results but normal CT findings - thus not having pneumonia- increased CT sensitivity to 83.2% (95% CI: 82.3, 84.2).
These results and the corresponding AUC, accuracy and predictive values are presented in Table 2.
Table 2.
CT Diagnostic Performance
Inter and intra reader agreement
Annotations from different radiologists (Figure 2) were available for 235 CT scans exported twice and had received different anonymization numbers (Table E1).
Figure 2:
CT annotations: Classification as COVID+, COVID- or Normal CT. The readers had access to the CT scans using a 3D image visualization web application, allowing scrolling through the entire lung volume in the coronal, sagittal, or axial transverse plane. The CT scan shown here has been classified as COVID+, due to the presence of bilateral ground glass opacities and absence of features such as mucoid impaction, bronchiolar nodules, segmental or lobar consolidation.
Gwet's AC1 coefficient was 0.79, indicating strong interreader agreement for the classification of CT scans as COVID-19 positive or negative. Regarding disease extent, Gwet's AC1 coefficient was 0.38 for classification into one of the five categories (< 10%/10-24%/25-49%/50-74%/≥ 75%), but 0.85 for classification as < 50% vs ≥ 50%. (Figure 3). Different annotations from the same observer were also available for 324 CT scans. This happened when CT scans of a reading list were not labeled as “;already read”. Intra observer agreement was perfect for 16 of the 20 readers (κ=1.00), and almost perfect for the remaining four (κ ranging from 0.82 to 0.92).
Figure 3:
CT annotations: Visual quantification of lung disease extent. The readers had to visually quantify the extent of COVID-19 pneumonia on a 5-point scale. Here, it is estimated to be more than 50% and less than 75% (50-75%). Readers were also asked to manually contour COVID-19 pneumonia (area in blue in the right lung) on at least 2 CT images to later train deep learning algorithms for automated quantification of disease extent.
The proportion of double readings related to the whole number of readings was homogeneous amongst the readers (median 4.3%, IQR: 3.9, 5.2).
CT features reflecting comorbidities
Emphysema was rarely present, with 86.4% (3663/4238) of CT and RT-PCR positive subjects having no emphysema or, when present, affecting more than 25% of the lung in only 1.8% (79/4238).
Conversely, coronary artery calcifications were observed in 58.8% (2493/4238) of subjects with CT and RT-PCR positivity (Figure 4). Mean subcutaneous adipose thickness anterior to the sternum was 16.5 mm (±7.8) (Figure 5). This parameter showed good correlation with BMI (ρ=0.62, p<.001).
Figure 5:
CT annotations: Measurement of chest wall fat. The amount of fat in the chest wall was measured as shown here, in front of the sternum, on a mid-sagittal reformation of noncontrast CT. Here it is clearly increased in an obese 57-year-old man. This annotation served as a substitute for BMI.
Risk factors for severe outcome at 1-month follow-up
Risk factors for severe outcome were evaluated for the 4,238 subjects positive for both RTPCR and CT, who had unenhanced CT. At 1-month, 23.6% (1000/4238) of subjects positive for both CT and RT-PCR had developed severe disease. A comparison of the clinical data and imaging findings for subjects with severe and non-severe outcome is presented in Table 3. There was no evidence of an association between adipose thickness anterior to the sternum with disease severity (p=.10) contrary to emphysema percentage, coronary artery calcifications, and disease extent on CT (p< .001 for all 3 parameters).
Table 3.
Comparison of Characteristics in Severe and Non-Severe COVID Cases
A clinical model selection procedure retained age, sex, oxygen supplementation at presentation, hypertension and coronary artery disease as clinical risk factors of severe COVID-19 cases across pooled imputed datasets. The model achieved an AUC of 0.64 (95%CI: 0.62, 0.66) and no lack of fit was detected (Hosmer-Lemeshow test: p ranging from 0.79 to 0.96 across imputed datasets). When combining clinical variables and CT annotations, the model selection procedure retained age, sex, oxygen supplementation, hypertension, coronary artery disease, coronary artery calcium score and disease extent as risk factors. Table 4 displays the adjusted odds ratios of all significant predictors for severe outcome. The final risk model achieved an AUC of 0.69 (95%CI: 0.67, 0.71), without evidence for miscalibration (Hosmer-Lemeshow test: p ranging from 0.10 to 0.96 across imputed datasets) and was better than the clinical model (Likelihood ratio test p<.001) (Figure 6).
Table 4.
Adjusted Coefficients and Odds Ratios of Significant Predictors of Severe Outcome (n = 4085)
Figure 6:
Performance of the clinical and mixed (clinical and CT) models. The prediction model that included clinical features alone achieved an AUC of 0.64 while the use of both clinical features and CT improved discrimination between subjects with and without severe outcomes at 1- month follow-up (AUC: 0.69).
Taking into account the only two significant interactions (coronary artery disease and hypertension, oxygen supplementation and disease extent) did not improve the performance of the model (Table E2, Figure E1).
After multiplying the regression coefficient by 5 and rounding up to the nearest integer, a nomogram for severity prediction was derived (Figure E2), ranging from 0 (lowest risk) to 18 (highest risk), and achieved the same AUC.
Discussion
Despite abundant literature, CT diagnostic performance for COVID-19 pneumonia remains a subject of debate with diagnostic specificities ranging from less than 20% to 90%, depending on the diagnostic criteria and tests used for confirming infection (15–17).In this retrospective cohort study of 10,735 subjects suspected of COVID-19 pneumonia, the overall CT diagnostic accuracy was 80.0%, considering RT-PCR as the reference standard, and increased to 86.3% after 5 days of symptoms. Male predominance, increased BMI, diabetes and other characteristics of our study sample were in line with those already reported as risk factors for developing symptomatic COVID-19 pneumonia requiring hospitalization (18–20). Conversely the presence of emphysema was a rare finding in our study. An age- and sex-adjusted meta-analysis suggested that cigarette smoking might be protective against contracting COVID-19 (21). It can be argued that smoking is inconsistently reported, whereas emphysema was systematically assessed by the readers in our study.
We report high intra and inter-reader agreement with no influence of reader's experience on CT diagnostic performance, supporting the use of predefined criteria. CT sensitivity increased after 5 days of symptoms, in line with other reports of higher sensitivity of CT after 2 to 5 days of symptoms (22,23).
In our study, CT specificity, although insufficient for a confident diagnosis, was much higher than that of the first reports (4). A first Cochrane review based on studies mainly from Asian countries reported a pooled CT specificity of only 18.1% (24). Even though an update of this first review reported a pooled specificity of 61.1% (25), the authors estimated that the heterogeneity and insufficient quality of the included studies limited their possibility to draw conclusions and highlighted the need to conduct future diagnostic accuracy studies that should predefine positive imaging findings.
The structured reporting of COVID-19 pneumonia proposed by the French Society of Radiology, used in this study, requires that bronchiolar nodules, mucoid impactions, and focal consolidation should be absent for diagnosis (12). Other proposals for structured reporting also include CT features favoring diagnoses other than COVID-19 pneumonia (26–28). CORADS (COVID-19 Reporting and Data System) 2 category corresponds to a low level of COVID-19 suspicion when centrilobular nodules, lobar or segmental consolidation, or lung cavitation are observed. COVID-RADS includes pulmonary nodules and airway secretions as inconsistent with COVID-19 (27). The Radiological Society of North America consensus document suggested one atypical appearance category: bronchiolar nodules or cavitation (28).
As recommended by the American College of Radiology (5) RT-PCR is the only specific method of diagnosis even though it has imperfect sensitivity (29). The first RT-PCR showed a sensitivity of 95.8% in our study but not all negative tests were repeated. When RTPCR was negative and CT was interpreted as positive, 39.3% of the initially negative RTPCR results were positive on repeat testing, highlighting the need for further testing in cases of discrepancies.
In addition to the evaluation of diagnostic performance, another important objective of the STOIC project was to evaluate the risk factors for a poor outcome. As previously reported (18,30), we found that advanced age, male sex, and hypertension were risk factors for severe outcome, contrary to the amount of chest wall fat, a substitute for missing BMI. Emphysema was not found to be an independent predictor of severity in our study, in line with the report by Grasselli et al of COPD subjects representing only 4% of those admitted in ICU (18).
The risk model for severe outcome was improved when clinical variables were combined with CT annotations, the extent of lung parenchymal damage being the strongest predictor.
Artificial intelligence could help better predict disease severity in subjects with Covid-19 (31,32), but requires large datasets, one of the reasons the STOIC project was launched.
Our study had limitations. First, we did not repeat all first negative PCR tests, probably leading to an underestimation of CT specificity. Second, double readings were not systematically planned for evaluating intra and inter reader agreements. This evaluation resulted from post hoc analysis and was based on CT exams that received double annotations by chance, thus with a low risk of bias. Third, BMI data was missing in too many of our subjects to be able to be included in the clinical model. However, chest wall fat thickness could be used as a surrogate and was not retained in the final model. Fourth, we also did not include biological parameters, because the biological tests to be performed at presentation were not standardized during the first wave of the pandemic in our country. Lastly, we defined poor outcome as the risk for intubation at one point or death at 1-month follow-up. We know now that dexamethasone results in lower 28-day mortality (33,34), however subjects from our series who were intubated and/or died had developed a severe form of the disease. Thus, the definition of poor outcome in our study remains valid even though the therapeutic management of severe forms of COVID-19 has improved.
In conclusion, when using predefined criteria, CT interpretation performance was not influenced by reader experience and enabled a correct diagnosis of COVID-19 pneumonia with an overall 80.0% accuracy, increasing to 86.3% after 5 days of symptoms. For subjects with negative RT-PCR but signs of COVID-19 pneumonia on CT, RT-PCR testing should be repeated since it might become positive in 39.3% of such subjects. Lastly, the extent of lung disease on CT at initial presentation is a strong predictor of poor outcome within 1-month admission.
Acknowledgments
Acknowledgements
We thank all the clinical research technicians who collected the clinical data and exported the CT scans in an anonymized format, especially W Hadi, J Pierson, S Chenaf and A Bellino for coordination. We also thank C Villeret from the Clinical Research and Innovation Department in of Assistance Publique des Hôpitaux de Paris for her supervision.
We thank General Electric Healthcare for providing the 3D image viewer web application and Orange Healthcare for providing data storage resources.
STOIC
Study of ThOracic computed tomography In Covid-19
List of Steering Committee members
Marie-Pierre Revel1(chair) M.D., Ph.D., Guillaume Chassagnon1, M.D., Ph.D., Mathieu Lederlin2, M.D., Ph.D., Mickael Ohana M.D3., Ph.D., Sébastien Bommart4 M.D., Ph.D., Laure Fournier5M.D., Ph.D., Pascal Rousset6, M.D., Ph.D., Dominique Valeyre7 M.D., Raphael Porcher8, Hendy Abdoul9 M.D., Ph.D, ErikDomain10
Affiliations
1Université de Paris, APHP, Hôpital Cochin, Dept. of Radiology, Paris, France, 2 Université Rennes1, Hôpital Pontchaillou, Dept. of Radiology, Rennes, France, 3Université de Strasbourg, Nouvel Hôpital Civil, Dept. of Radiology, Strasbourg, France, 4 Université de Montpellier, Hôpital Arnaud de Villeneuve, Dept. of Radiology, Montpellier France, 5 Université de Paris, APHP, Hôpital Européen Georges Pompidou, Dept. of Radiology, PARCC UMRS 970, INSERM, Paris, France,6 Université Claude Bernard Lyon 1, Hospices Civils de Lyon, Hôpital Lyon Sud, Dept. of Radiology, Pierre-Benite, France, 7 Sorbonne Université, APHP, Hôpital Avicenne, Dept. of Pneumology, Bobigny, France, 8Université de Paris, APHP, Hôtel-Dieu, Dept. of Clinical Epidemiology, Paris, France, 9Université de Paris APHP, Clinical Research Unit Paris Centre, Paris, France; 10Dierction de la recherche clinique, APHP, Paris, France.
RULES TO ACCESS « STOIC » DATA FOR ACADEMIC RESEARCH
This document concerns the conditions of access to the data of the STOIC research «Study of ThOracic computed tomography In Covid-19», whose objective was to validate COVID-19 diagnosis criteria on computed tomography, based on a large multi-centric dataset associated with multi-expert annotation. The Principal Investigator of this research is Pr. Marie-Pierre Revel, Radiology Department of Cochin Hospital (AP-HP). Assistance Publique – Hôpitaux de Paris is the Sponsor of the STOIC research.
This document sets the conditions of access for all academic partners who wish to use the STOIC data for collaborative academic research.
Any and all academic partners who wish to access and use the STOIC data shall accept the following conditions.
Governance:
The STOIC research is overseen by a Steering Committee composed of:
– Pr Marie-Pierre Revel, Department of Radiology, Cochin Hospital, Paris, France Principal Investigator
– Dr Guillaume Chassagnon', Department of Radiology, Cochin Hospital, Paris, France
– Pr Mathieu Lederlin, Department of Radiology, Hôpital Pontchaillou, Rennes, France, president of the French Society of Thoracic Imaging
– Pr Mickael Ohana, Department of Radiology, Nouvel Hôpital Civil, Strasbourg, France
– Dr Sébastien Bommart, Department of Radiology, Hôpital Arnaud de Villeneuve Montpellier, France
– Pr Laure Fournier, Department of Radiology, Pompidou Hospital, Paris, France Scientific director
– Pr Pascal Rousset, Department of Radiology, Hôpital Lyon Sud, Hospices Civils de Lyon, Lyon, France
– Pr Dominique Valeyre, Department of Pneumology, Hôpital Avicenne, Bobigny France
– Pr Raphael Porcher, Department of Clinical Epidemiology, Hôtel-Dieu, Paris, France
– Dr Hendy Abdoul, Clinical Research Unit Paris Centre, Paris, France
– Erik Domain, Clinical research director for APHP, Paris, France
Any collaborative research project led by an academic partner who requires access to the STOIC data shall be analyzed, validated, and authorized by the Steering Committee of STOIC. To this end, the academic partner shall send a document describing the research project to the Stoic Steering Committee at the following email address: Marie-pierre.revel@aphp.fr, with the following subject: STOIC DATA ACCESS PERMISSION
Contractualisation:
After acceptance by the Steering Committee, the academic partner shall sign a specific agreement (Data Transfer Agreement - DTA) with AP-HP, who are legally responsible for the STOIC data as Sponsor of the STOIC research.
The DTA signed by the parties will use the AP-HP template and will be governed by applicable French laws.
Each collaborative research project shall be subject to a specific DTA.
Conditions of use of the STOIC data:
In addition to the previous indication, any and all access to the STOIC data is provided according to the following conditions:
– The STOIC data shall remain at all times under the governance of AP-HP.
– The academic partner will only use the STOIC data for the collaborative research project presented to the Steering Committee;
– The STOIC data shall be used by the academic partner in accordance with all applicable regulations and laws
– During the collaborative research project, STOIC data shall be under the sole responsibility of the academic partner and should not be shared with any other parties
– The academic partner shall inform the STOIC Steering Committee on a regular basis the progress of their collaborative research project
– At the end of the collaborative research project, the academic partner shall destroy the STOIC data in its possession, and shall not keep any copies;
– The intellectual property related to the results of the collaborative research on the STOIC data shall be shared with APHP, as defined on the specific DTA
– The academic partner shall respect and follow the « STOIC PUBLICATION POLICY / PROCEDURES » (Appendix 1) for any publication and communication on STOIC data;
– French Law and the GDPR shall be applicable, only non-identifying, retrospective data will be shared
Description on Appendices:
Appendix E2: STOIC PUBLICATION POLICY/PROCEDURES
Appendix E3: Description of the available STOIC data
APPENDIX E2. Publication Policy/Procedures
The STOIC Publication Policy/Procedures must be accepted by the principal investigator in charge of a project endorsed or accepted by the STOIC Steering Committee before any access to the STOIC research material.
Such procedures include the following items:
– The STOIC-Research Collaborative group will be included in the team in charge of the writing of the manuscript, for critical review and approval of the final version. For this objective, the draft of the manuscript must be submitted to the chair of the STOIC steering committee at least 2 weeks before the submission to the targeted journal
- – The STOIC-Research Collaborative group will be included in the list of co- authors, as having fulfilled the 4 following ICMJE criteria
- Substantial contribution to the acquisition of data for the work; AND
- Critical revision for important intellectual content; AND
- Final approval of the version to be published; AND
- Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved
– Refer to the acronym STOIC in the title or the sub-title of any publication. In case the policy of the journal does not accept this, refer to the acronym STOIC in the abstract of the publication.
– Inform the president of the STOIC Steering Committee of the status of the manuscript every 3 months (i.e., before each STOIC Steering Committee meeting) until the publication of the manuscript.
– Provide to the president of the STOIC Steering Committee an electronic copy of the manuscript once published.
– Mention in the “Acknowledgements section” the sponsor (AP-HP). The current proposal is to use the following sentences: « The STOIC study was sponsored by Assistance Publique Hôpitaux de Paris and was funded by Fondation APHP pour la Recherche, Guerbet, Innothera, Fondation CentraleSupélec. General Electric Healthcare provided a 3D image visualization web application and Orange Healthcare a data repository”
– Reference the publication which describes the main baseline characteristics of the patients (Revel et al, Radiology 2021).
– The ClinicalTrials.gov NCT04355507 should be mentioned in the manuscript.
STOIC/Steering Committee
APPENDIX E3. Description of the available STOIC Data
Format of data:
– NIfti for CT scan images
– The clinical data will be shared as age category, delay from symptoms onset, or binary information:
for instance: severe/non severe outcome, severe/non severe coronary artery calcium score so that any re-identification is made impossible, according to GDPR
The baseline data of the 10,735 patients of STOIC include the following information:
Age category, gender: all 10,735 patients
RT-PCR results: all 10,735 patients
Delay between CT and onset of symptoms: (available for 4,478 of the 6,448 RT-PCR positive patients)
Hypertension: information available for 6,119 of the 6,448 RT-PCR positive patients
Pre-existing cardiovascular disease: information available for 6,027 of the 6,448 RT PCR positive patients
Diabetes: information available for 6,101 of the 6,448 RT-PCR positive patients
Pre-existing pulmonary disease: information available for 6,092 of the 6,448 RT-PCR positive patients
Thoracic CT scanners: all 10,735 patients
Respiratory motion artifacts on thoracic CT scans (all 10,735 patients)
Pulmonary embolism on thoracic CT scans: information available for all contrast-enhanced studies (n= 2,084)
Pneumonia disease extent on thoracic CT scans: visual assessment using a 5-point scale (<10%, 10-25%,25-50%, 50-75%, > 75%), available for 5,174 patients with positive RT-PCR and signs of COVID-19 pneumonia on CT
Coronary artery calcification visual scoring (shemesh et al. doi: 10.1148/radiol.10100383) on thoracic CT scans: available for 4,238 patients with positive RT-PCR and signs of pneumonia on unenhanced CT
Emphysema visual scoring on thoracic CT scans using a 5-point scale: for 5,174 patients with positive RT-PCR and signs of pneumonia on CT
Chest wall fat thickness on thoracic CT scans measured on a midline sagittal reformation, middle part of the sternum: for 5,174 patients with positive RT-PCR and signs of pneumonia on CT
Pneumonia disease segmentation: manual segmentation of COVID-19 pneumonia on 8,476 CT images of patients with positive RT-PCR results and signs of COVID-19 pneumonia on thoracic CT
Outcome at 1-month: discharge/intubation/death: all 6,448 RT-PCR positive patients
Table E1.
Reader's Expertise and Number of Readings Years of experience Number of CT scans
Table E2.
Adjusted Coefficients and Odds Ratios of Significant Predictors of Severe Outcome Considering Two-Factor Interactions (n = 4,085)
Figure E1:
Performance of the mixed clinical and CT model with and without taking into account significant interactions. Taking into account interactions did not improve the performance of the model (AUC: 0.69).
Figure E2:
Summary of the simplified risk score.
Funding/Support: This Study received funding from Fondation APHP pour la Recherche, Guerbet, Inothera and Fondation Centrale Supélec. General Electric Healthcare provided the 3D image viewer web application and Orange Healthcare data storage resources.
Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Trial registration: This study is registered with ClinicalTrials.gov number NCT04355507.
Abbreviations:
- BMI
- Body Mass Index
- COVID-19
- Coronavirus Disease 2019
- OR
- Odds Ratio
- RT-PCR
- Reverse Transcription–Polymerase Chain Reaction
References
- 1.Hoffmann M, Kleine-Weber H, Schroeder S, et al. SARS-CoV-2 Cell Entry Depends on ACE2 and TMPRSS2 and Is Blocked by a Clinically Proven Protease Inhibitor. Cell. 2020;181(2):271–280.e8. 10.1016/j.cell.2020.02.052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Schiaffino S, Tritella S, Cozzi A, et al. Diagnostic Performance of Chest X-Ray for COVID-19 Pneumonia During the SARS-CoV-2 Pandemic in Lombardy, Italy. J Thorac Imaging. 2020;35(4):W105–W106. 10.1097/RTI.0000000000000533. [DOI] [PubMed] [Google Scholar]
- 3.Revel M-P, Parkar AP, Prosch H, et al. COVID-19 patients and the radiology department - advice from the European Society of Radiology (ESR) and the European Society of Thoracic Imaging (ESTI). Eur Radiol. 2020;30(9):4–4909. 10.1007/s00330-020-06865-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ai T, Yang Z, Hou H, et al. Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology. 2020;296(2):E32–E40. 10.1148/radiol.2020200642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection. American College of Radiology. https://www.acr.org/Advocacy-and-Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection. Accessed January 7, 2021. [Google Scholar]
- 6.Hope MD, Raptis CA, Shah A, Hammer MM, Henry TS, six signatories ., A role for CT in COVID-19? What data really tell us so far. Lancet. 2020;395(10231):4–1190. 10.1016/S0140-6736(20)30728-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Huang Y, Cheng W, Zhao N, Qu H, Tian J. CT screening for early diagnosis of SARS CoV-2 infection. Lancet Infect Dis. 2020;20(9):4–1011. 10.1016/S1473-3099(20)30241-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Sverzellati N, Milanese G, Milone F, Balbi M, Ledda RE, Silva M. Integrated Radiologic Algorithm for COVID-19 Pandemic. J Thorac Imaging. 2020;35(4):4–233. 10.1097/RTI.0000000000000516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. 10.1136/bmj.h5527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Réponses rapides dans le cadre de la COVID-19 - Place du scanner thoracique. Haute Autorité de Santé. https://www.has-ante.fr/upload/docs/application/pdf/2020-04/reponse_rapide_codid-19_indication_tdm_mel2.pdf. Accessed January 7, 2021.
- 11.Epidémie de Covid-19 :POINT SUR L’IMAGERIE. Société Française de Radiologie. 2020. https://ebulletin.radiologie.fr/actualites-covid-19/epidemie-covid-19-point-limagerie. Accessed January 7, 2021.
- 12.Compte-rendu structuré de scanner thoracique pour les patients suspects de COVID-19. Société Française de Radiologie. 2020. https://ebulletin.radiologie.fr/sites/ebulletin.radiologie.fr/files/medias/cr_type_covid-19_last.pdf. Accessed January 7, 2021.
- 13.Shemesh J, Henschke CI, Shaham D, et al. Ordinal scoring of coronary artery calcifications on low-dose CT scans of the chest is predictive of death fromcardiovascular disease. Radiology. 2010;257(2):4–548. 10.1148/radiol.10100383. [DOI] [PubMed] [Google Scholar]
- 14.Moons KGM, Harrell FE, Steyerberg EW. Should scoring rules be based on odds ratios or regression coefficients? J Clin Epidemiol. 2002;55(10):4–1055. 10.1016/s0895-4356(02)00453-5. [DOI] [PubMed] [Google Scholar]
- 15.De Smet K, De Smet D, Ryckaert T, et al. Diagnostic Performance of Chest CT for SARS-CoV-2 Infection in Individuals with or without COVID-19 Symptoms. Radiology. 2021;298(1):E30–E37. 10.1148/radiol.2020202708. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kovács A, Palásti P, Veréb D, Bozsik B, Palkó A, Kincses ZT. The sensitivity and specificity of chest CT in the diagnosis of COVID-19. Eur Radiol. 2020; 10.1007/s00330-020-07347-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Herpe G, Lederlin M, Naudin M, et al. Efficacy of Chest CT for COVID-19 Pneumonia in France. Radiology. 2020; 10.1148/radiol.2020202568. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Grasselli G, Zangrillo A, Zanella A, et al. Baseline Characteristics and Outcomes of 1591 Patients Infected With SARS-CoV-2 Admitted to ICUs of the Lombardy Region, Italy. JAMA. 2020;323(16):4–1581. 10.1001/jama.2020.5394. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Wu C, Chen X, Cai Y, et al. Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease 2019 Pneumonia in Wuhan, China. JAMA Intern Med. 2020;180(7):4–943. 10.1001/jamainternmed.2020.0994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Richardson S, Hirsch JS, Narasimhan M. Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized With COVID-19 in the New York City Area. JAMA. 2020;323(20):4–2059. 10.1001/jama.2020.6775. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Farsalinos K, Angelopoulou A, Alexandris N, Poulas K. COVID-19 and the nicotinic cholinergic system. Eur Respir J. 2020;56(1). 10.1183/13993003.01589-2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Bernheim A, Mei X, Huang M, et al. Chest CT Findings in Coronavirus Disease-19 (COVID-19): Relationship to Duration of Infection. Radiology. 2020;295(3):200463. 10.1148/radiol.2020200463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Guillo E, Bedmar Gomez I, Dangeard S, et al. COVID-19 pneumonia: Diagnostic and prognostic role of CT based on a retrospective analysis of 214 consecutive patients from Paris, France. Eur J Radiol. 2020;131:109209. 10.1016/j.ejrad.2020.109209. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Salameh J-P, Leeflang MM, Hooft L, et al. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev. 2020;9:CD013639. 10.1002/14651858.CD013639.pub2. [DOI] [PubMed] [Google Scholar]
- 25.Islam N, Salameh J-P, Leeflang MM, et al. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev. 2020;11:CD013639. 10.1002/14651858.CD013639.pub3. [DOI] [PubMed] [Google Scholar]
- 26.Prokop M, van Everdingen W, van Rees Vellinga T, et al. CO-RADS: A Categorical CT Assessment Scheme for Patients Suspected of Having COVID-19-Definition and Evaluation. Radiology. 2020;296(2):E97–E104. 10.1148/radiol.2020201473. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Salehi S, Abedi A, Balakrishnan S, Gholamrezanezhad A. Coronavirus disease 2019 (COVID-19) imaging reporting and data system (COVID-RADS) and common lexicon: a proposal based on the imaging data of 37 studies. Eur Radiol. 2020;30(9):4–4942. 10.1007/s00330-020-06863-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Simpson S, Kay FU, Abbara S, et al. Radiological Society of North America Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19. Endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA. Radiology: Cardiothoracic Imaging. 2020;2(2):e200152. 10.1148/ryct.2020200152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Green DA, Zucker J, Westblade LF, et al. Clinical Performance of SARS-CoV-2 Molecular Tests. J Clin Microbiol. 2020;58(8). 10.1128/JCM.00995-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Chen T, Wu D, Chen H, et al. Clinical characteristics of 113 deceased patients with coronavirus disease 2019: retrospective study. BMJ. 2020;368:m1091. 10.1136/bmj.m1091. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Chassagnon G, Vakalopoulou M, Battistella E, et al. AI-driven quantification, staging and outcome prediction of COVID-19 pneumonia. Med Image Anal. 2021;67:101860. 10.1016/j.media.2020.101860. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Vardhanabhuti V. CT scan AI-aided triage for patients with COVID-19 in China. Lancet Digit Health. 2020;2(10):e494–e495. 10.1016/S2589-7500(20)30222-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.WHO Rapid Evidence Appraisal for COVID-19 Therapies (REACT) Working Group , Sterne JAC, Murthy S, et al. Association Between Administration of Systemic Corticosteroids and Mortality Among Critically Ill Patients With COVID-19: A Meta-analysis. JAMA. 2020;324(13):4–1341. 10.1001/jama.2020.17023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.RECOVERY Collaborative Group, Horby P, Lim WS, et al. Dexamethasone in Hospitalized Patients with Covid-19 - Preliminary Report. N Engl J Med. 2020; 10.1056/NEJMoa2021436. [DOI] [PMC free article] [PubMed]