Temporal variations in the diagnostic performance of chest CT for Covid-19 depending on disease prevalence: Experience from North-Eastern France

Highlights

• •Chest CT accuracy was 85.4 % during ascending, peak and descending phases of the first Covid-19 wave.
• •It was not significantly different between chest (4 readers, accuracy 86.4 %) and general (6 readers, accuracy 85.1 %) radiologists, with a probability of equivalence ≥ 95 %.
• •There were important variations of CT predictive values whereas CT specificity remained stable over time.

Abstract

Rationale and Objective

The purpose of this work was to analyze temporal variations in the diagnostic performance of chest CT for Covid-19 throughout the first wave, depending on disease prevalence variations between the ascending, peak and descending phases of the epidemic in North-Eastern France.

Materials and Methods

From March 6th to April 22nd 2020, all consecutive adult patients referred to the “Covid-19 clinic” of our Emergency Department with the availability of chest CT and of at least one RT-PCR result were retrospectively included in the present study. Chest CT was considered positive when typical Covid-19 lesions were observed (bilateral and predominantly peripheral and sub-pleural ground glass opacities and/or alveolar consolidations). RT-PCR results were considered as the reference standard. Ascending, peak and descending phases were determined based on the number of CT scans performed daily. CT diagnostic performance were calculated and variations between phases were tested for equivalence or difference using Bayesian methods.

Results

2194 consecutive chest CT were analyzed. Overall CT diagnostic performance was Se = 84.2 [82.0 ; 86.3], Sp = 86.6 [84.5 ; 88.5], PPV = 86.1 [84.0 ; 88.1], NPV = 84.7 [82.6 ; 86.7] and accuracy = 85.4 [83.9 ; 86.8], with no significant differences between chest and non-chest radiologists. Variations between the ascending (11 days, 281 chest CT, disease prevalence 37.0 %), the peak (18 days, 1167 chest CT, disease prevalence 64 %) and the descending phases (19 days, 746 chest CT, disease prevalence 32.2 %) were highest for PPV and NPV with a probability of difference >99.9 %, and smallest for accuracy and specificity with a probability of equivalence >98.8 %.

Conclusion

In a homogenous cohort of 2194 consecutive chest CT performed over a 7-week epidemic wave, we observed significant variations of CT predictive values whereas CT specificity appeared marginally affected.

1. Introduction

Chest CT is the imaging modality of choice for diagnosis, severity evaluation and follow-up of Covid-19 patients [1,2]. Characteristic chest CT lesions have been broadly described in the recent literature [3], with significant differences from other viral pneumonia [4]. An initial Chinese study on 1014 patients reported a higher sensitivity as compared to RT-PCR [5].

However, the exact diagnostic performance of chest CT for Covid-19 remains debated, with a meta-analysis [6] reporting high sensitivity (94 %, 95 % CI 91 %–96 %) and low specificity (37 %, 95 % CI 26 %–50 %). Actually, changes in disease prevalence could also lead to a significant degradation of both negative and predictive positive values (NPV and PPV) [7], with concerns for a particularly low PPV that could hamper the use of CT as a screening tool in patients with moderate to severe symptoms [8]. Indeed, studies focusing on CT diagnostic performance were mainly retrospective [[9], [10], [11]], with no indication about the timing of inclusions with regards to the epidemic wave. To the best of our knowledge, no study has yet reported the potential temporal variation of CT diagnostic performance in a homogeneous cohort over the course of the epidemic, when disease prevalence varies significantly between the ascending, the peak and the descending phases.

The purpose of this work was therefore to analyze the temporal variations in diagnostic performance of chest CT for Covid-19 throughout the first epidemic wave, depending on disease prevalence variations between the ascending, peak and descending phases at a single tertiary clinical centre located in a severe SARS-CoV-2 cluster in North-Eastern France.

2. Materials and methods

The local ethics committee of Strasbourg University Hospital approved this retrospective study and waived the need of informed consent.

2.1. Patient population

From March 6th to April 22nd, 2020, all the consecutive adult patients with a clinical suspicion of Covid-19 admitted to the Emergency Department (ED) of a single tertiary care centre (Nouvel Hôpital Civil, Strasbourg University Hospital) were retrospectively analyzed and considered for inclusion. March 6th was the opening day of this dedicated Covid-19 ED pathway [12]. All patients referred for CT had moderate to severe symptoms, and chest CT was systematically used at ED arrival to expedite triage [8], due to the major delays (6–24 hours) in RT-PCR results at that time.

The inclusion criteria were:

• -clinical suspicion of Covid-19,
• -availability of chest CT at the time of ED admission,
• -availability of at least one RT-PCR result.

The only exclusion criterion was a delay of more than 6 days between chest CT and the first positive RT-PCR.

Gender, age and direct ICU transfer for acute respiratory insufficiency after ED passage were documented.

2.2. Chest CT

Non enhanced chest CT were acquired on an 80-row scanner (Aquilion Prime SP, Canon Medical Systems), with parameters optimized for the patient’s morphotype (tube voltage 100−135 kV and maximum mAs of 2–50). Images were reconstructed with a slice thickness of 1 mm in mediastinal and parenchymal windows using an iterative reconstruction algorithm (AIDR-3D, Canon Medical Systems) and read on dedicated workstations with multiplanar and maximum intensity projection reconstructions.

Ten consultant radiologists (4 specialized in chest imaging) with 5–30 years of experience were involved in the reading during the inclusion period. Rotations between readers changed significantly during the epidemic, thus the number of cases read by each radiologist could not be equally distributed between phases.

Standardized structured reporting was systematically used and examinations were adjudicated as either positive or negative for Covid-19:

• -Chest CT with typical Covid-19 appearance, i.e. bilateral and predominantly peripheral and sub-pleural ground glass opacities and/or alveolar consolidations, were classified as positive [1].
• -Alternative infectious findings (bronchiolitis with centrilobular nodules, lobar consolidation), non-infectious abnormalities (lung nodule, pneumothorax, effusion…) and normal examinations were classified as negative [4].

2.3. RT-PCR

A nasopharyngeal swab (Puritan Medical Products, Guilford, ME, USA) concomitant to the chest CT was systematically performed at ED admission. Some patients underwent multiple sampling with either nasopharyngeal swab, sputum or bronchoalveolar lavage during their stay.

RT-PCR for SARS-CoV-2 was chosen as the reference standard and determined the definite diagnosis: any positive result was adjudicated as a confirmed Covid-19 infection.

2.4. Temporality

The epidemic wave was arbitrarily divided into three phases: ascending, peak and descending, based on the number of CT scans performed daily and later confirmed by differences in prevalence of the disease, as assessed by the positivity rate of RT-PCR.

2.5. Statistics

Bayesian methods were used to estimate Sensitivity (Se), Specificity (Sp), Positive Predictive value (PPV), Negative Predictive Value (NPV) and accuracy with their 95 % credible interval. Posterior distributions were calculated using the Dirichlet distribution with Jeffrey’s prior. Differences between the phases ascending VS peak and descending VS peak were computed. Overall differences in diagnostic performances between the 4 radiologists specialized in chest imaging versus the 6 other readers were also computed.

A hypothesis of equivalence of the indicators was first tested. The equivalence was defined as an estimated difference (with its 95 % credible interval) included between +/- 10 %. A probability of equivalence (PrE) was therefore calculated, and equivalence was considered achieved when PrE ≥ 95 % (bilateral hypothesis).

When equivalence was not achieved, a hypothesis of difference was secondly assessed. Difference was confirmed when the 95 % credible interval of the difference didn’t include the value 0, corresponding to a probability of difference PrD ≥ 97.5 % (unilateral hypothesis).

3. Results

3.1. Patient population

The flowchart of the study is presented in Fig. 1 .

Fig. 1.

Flowchart of the study.

2278 chest CT were performed in 2164 patients over the inclusion period, with 2194 chest CT in 2080 patients having a RT-PCR result (of which 1091 were positive, 49.7 %). In the 1103 chest CT with a negative RT-PCR, 198 (18 %) had more than one assay – persistently negative – performed over the course of their hospital stay.

3.2. Chest CT

Chest CT was positive in 1067 cases, with 919 true positive and 148 adjudicated as false positive cases. Out of these 148 patients with positive chest CT and negative RT-PCR, 49 (33.1 %) underwent more than 1 RT-PCR assay, all remaining negative.

Chest CT was negative in 1127 cases (with 141 suggesting alternative infectious findings, 274 exhibiting non-infectious abnormalities and 712 normal chest CT), with 955 true negative and 172 false negative cases.

Overall CT diagnostic performance with 95 % credible interval were Se = 84.2 % [82.0 ; 86.3], Sp = 86.6 % [84.5 ; 88.5], PPV = 86.1 % [84.0 ; 88.1], NPV = 84.7 % [82.6 ; 86.7] and accuracy = 85.4 % [83.9 ; 86.8].

3.3. Effect of expertise

The 4 radiologists specialized in chest imaging read 588 examinations, with 247 true positive, 36 false positive, 261 true negative and 44 false negative cases, resulting in Se = 84.9 % [80.2 ; 88.8], Sp = 87.8 % [83.6 ; 91.4], PPV = 87.3 % [83.4 ; 90.3], NPV = 85.6 % [81.8 ; 88.7] and accuracy = 86.4 % [83.4 ; 89.1]. In comparison, the 1606 examinations read by the 6 other radiologists had 672 true positive, 112 false positive, 694 true negative and 128 false negative cases, resulting in non-significantly lower diagnostic performance, with Se = 84 % [81.3 ; 86.5], Sp = 86.1 % [83.5 ; 88.4], PPV = 85.7 % [83.4 ; 87.7], NPV = 84.4 % [82.2 ; 86.4] and accuracy = 85.1 % [83.2 ; 86.8]. The probability of equivalence between both groups of readers was ≥95 % for all variables.

3.4. Temporal variations

281 chest CT were performed during the ascending phase (March 6th to March 16th, 11 days), with a RT-PCR prevalence of 37.0 %, a male/female sex ratio of 0.52, a mean patient age of 59 ± 17.6yo (range 18–97) and 19 direct ICU transfers (6.8 %).

1167 were performed during the peak (March 17th to April 3rd, 18 days), with a RT-PCR prevalence of 64.0 %, a male/female sex ratio of 0.59, a mean age of 61.4 ± 18.1 (18–99) and 96 direct ICU transfers (8.2 %).

746 were acquired in the descending phase (April 4th to April 22nd, 19 days), with a RT-PCR prevalence of 32.2 %, a male/female sex ratio of 0.53, a mean age of 64.2 ± 16.3 (18–100) and 16 direct ICU transfers (1.4 %).

Temporal variations of diagnostic performances are summarized in Fig. 2 ; their differences between ascending VS peak and descending VS peak with probability of equivalence +/- probability of difference are given in Fig. 3 . Variations of both positive and negative predictive values were significant and related with disease prevalence: the probabilities of equivalence for PPV and NPV was never met and their probability of difference was always >99.9 %. Conversely, variations in accuracy and in specificity were considerably lower and their probability of equivalence between phases was >98.8 %.

Fig. 2.

Daily number of chest CT performed for suspicion of Covid-19 in ED patients (red line), and percentage of positive CT (green bars). Lighter green bars indicate week-end and bank holidays.

Chest CT diagnostic performance using RT-PCR as the gold standard are reported with 95 % interval confidence for each phase of the epidemic wave.

Fig. 3.

Differences in sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy between ascending VS peak (a) and descending VS peak phases (b) with probability of equivalence (PrE) +/- probability of difference (PrD) calculated.

* indicates differences that met the hypothesis of either an equivalence (PrE≥95 %) or either a difference (PrD≥97.5 %).

4. Discussion

In a homogenous population of 2194 consecutive chest CT with RT-PCR performed over a 7-week epidemic wave, we observed an equivalence in specificity and accuracy and a difference in predictive values, between the ascending, peak and descending phases. Variations in predictive values are connected to the variations in prevalence of Covid-19. The equivalence in accuracy support the use of CT as an efficient triage tool throughout the epidemic [8]. This study was carried out in a practical setting, pooling the readings of radiologists with various levels of expertise and in ED patients with a broad range of clinical manifestations.

When compared to the existing literature [9], while we reported a sensitivity similar to most publications, we noted a higher specificity than Ai et al. (0.25, 95 % CI 0.21−0.30) [5], Chen et al. (0.26, 95 % CI 0.11−0.46) [13], Zhu et al. (0.33, 95 % CI 0.23−0.44) [14] and Caruso et al. (0.56, 95 % CI 0.46−0.66) [15]. We report a higher specificity than in these papers, probably because exclusion criteria suggesting alternative infectious disease were considered, such as bronchiolitis with centrilobular nodules or lobar consolidation [1]. Our findings on a consecutive and homogenous ED cohort might be more representative of the “real-world” diagnostic performance of chest CT [16].

A potential “learning curve” effect must be raised. Indeed, new data on Covid-19 were published daily, and our knowledge of the disease grew along the epidemic wave [2]. Therefore, one might suggest that the variations in sensitivity and specificity were also secondary to a massive exposure to the disease. Yet, variations in sensitivity were lower between the ascending and peak than between the descending and peak phases, which does not support an improvement over time. In addition, we did not find a significant difference in performance between the radiologists specialized in chest imaging and the other readers. These observations could disfavor a potential learning curve effect in this setting.

Variations of PPV and NPV with regards to variations of prevalence are mathematical and expected [7], while the moderate temporal variations of sensitivity are more difficult to explain. We hypothesize that the decreased sensitivity in the descending phase might be related to a/ an older population, where the presentation might not be as typical as in younger patients [17,18] and b/ to an overall decrease in the severity of the disease as assessed by the decreased ICU transfer rate and an overall higher number of CT read as normal (72 % of all false negative examinations in the descending phase compared to 57–58 % in the peak and ascending phases).

4.1. Our work has several limitations

First, we chose RT-PCR as the gold standard for both positive and negative definite diagnosis, which is a debatable position for cases with positive chest CT and negative RT-PCR (148 in our cohort), considering that the risk of false negative of RT-PCR is non-negligible is these cases [5,19,20]. However, 33.1 % of these 148 patients with a positive chest CT and an initially negative RT-PCR had a repeat RT-PCR assay (up to 6 times) that remained negative.

Second, our study is retrospective, even though we included all consecutive cases from our ED Covid-19 clinic, and therefore can be subject to inclusion bias.

Third, only basic demographics (age and gender) were recorded, and useful clinical information such as Oxygen saturation, onset to symptoms and outcome could not be documented in the present study for the whole population. Therefore, a strict comparison of the disease severity between phases could not be achieved, and one can imagine that potential differences such as a less “aggressive” or respiratory-focused presentation in the decreasing phase could lead to potential variations in performance of chest CT. Yet, the clinical criteria for inclusion in the ED Covid-19 pathway were similar through the epidemic, and patients had a relatively consistent presentation, with clinical suspicion of Covid-19 and moderate to severe symptoms.

Fourth, we took into account only the initial reading of the chest CT at the time of ED admission, which involved 10 consultant radiologists with diverse experience, sub-specialty and training backgrounds. Even though a standardized report was used to mitigate inter-reader variations, a dedicated reading or second look by radiologists specialized in chest imaging might have further improved the performance of CT.

Finally, our results come from a single centre in an intense SARS-CoV-2 cluster and, as is, cannot be generalizable to other places with lesser prevalence of the disease or other medical systems.

To conclude, our study showed a significant difference of predictive values, a moderate variation of sensitivity and an equivalence of specificity and accuracy between the ascending, peak and descending phases in a homogenous cohort of 2194 consecutive chest CT performed over a 7-week epidemic wave. This could consolidate the use of chest CT throughout the epidemic, regardless of significant variations in disease prevalence.

CRediT authorship contribution statement

Mickaël Ohana: Conceptualization, Methodology, Validation, Writing - original draft. Joris Muller: Software, Formal analysis, Data curation, Writing - review & editing. François Severac: Formal analysis, Data curation, Writing - review & editing. Pascal Bilbault: Investigation, Data curation, Writing - review & editing. Martin Behr: Investigation, Data curation, Writing - review & editing. Mathieu Oberlin: Investigation, Data curation, Writing - review & editing. Pierre Leyendecker: Investigation, Data curation, Writing - review & editing. Catherine Roy: Supervision, Data curation, Writing - review & editing.

Declaration of Competing Interest

The author declares no sources of support or conflict of interest.