Improving CT accuracy in the diagnosis of COVID-19 in a hospital setting

Abstract

Objective

This study aimed to improve the accuracy of CT for detection of COVID-19-associated pneumonia and to identify patient subgroups who might benefit most from CT imaging.

Methods

A total of 269 patients who underwent CT for suspected COVID-19 were included in this retrospective analysis. COVID-19 was confirmed by reverse-transcription-polymerase-chain-reaction. Basic demographics (age and sex) and initial vital parameters (O₂-saturation, respiratory rate, and body temperature) were recorded. Generalized mixed models were used to calculate the accuracy of vital parameters for detection of COVID-19 and to evaluate the diagnostic accuracy of CT. A clinical score based on vital parameters, age, and sex was established to estimate the pretest probability of COVID-19 and used to define low, intermediate, and high risk groups. A p-value of <0.05 was considered statistically significant.

Results

The sole use of vital parameters for the prediction of COVID-19 was inferior to CT. After correction for confounders, such as age and sex, CT showed a sensitivity of 0.86, specificity of 0.78, and positive predictive value of 0.36. In the subgroup analysis based on pretest probability, positive predictive value and sensitivity increased to 0.53 and 0.89 in the high-risk group, while specificity was reduced to 0.68. In the low-risk group, sensitivity and positive predictive value decreased to 0.76 and 0.33 with a specificity of 0.83. The negative predictive value remained high (0.94 and 0.97) in both groups.

Conclusions

The accuracy of CT for the detection of COVID-19 might be increased by selecting patients with a high-pretest probability of COVID-19.

1. Introduction

Coronavirus disease 2019 (COVID-19) has challenged healthcare systems worldwide. In regions with rapidly increasing numbers of infections, resources such as hospital beds and ventilators became scarce, and healthcare providers were faced with the prospect of having to make difficult decisions about triage and allocation [1,2]. Therefore, tools for early and precise diagnosis as well as outcome predictors for therapeutic decisions and targeted use of resources are needed.

Real-time reverse-transcription-polymerase chain reaction (RT-PCR) from deep nasal or pharyngeal throat swaps is currently considered the gold standard for the diagnosis of COVID-19. However, chest computed tomography (CT) is often used in hospitals, as it can provide immediate results and has shown high diagnostic accuracy [[3], [4], [5]]. Besides, CT imaging was reported to identify COVID-19 patients at earlier stages than RT-PCR [5]. This has been attributed to possible migration of the virus from the upper respiratory tract to the lungs or faulty execution of throat swaps.

Despite its novelty, the imaging characteristics of COVID-19 are well described, most commonly including ground glass opacities in a peripheral, bilateral dissemination pattern [6., [7], [8]]. Still, the role of CT as a screening tool remains controversial due to its limited ability to distinguish between COVID-19 and other types of viral pneumonia [9,10]. In a recent statement from the Fleischner Society, it was hence recommended that only patients with a high pretest probability of disease (moderate to severe clinical features, mild symptoms with risk factors for progression, or high pretest probability only) should undergo CT imaging [11].

Our aim was therefore to evaluate the accuracy of CT as a function of clinical symptoms and to estimate pre- and posttest probabilities of COVID-19 after CT imaging for different risk groups.

2. Material and methods

We retrospectively included 269 patients who underwent diagnostic chest CT for suspected COVID-19 between March 27th and April 27th, 2020. COVID-19 was confirmed or ruled out by RT-PCR. Repeat PCRs were performed in case of a discrepancy between CT and PCR, usually on the same day, with the new test initiated immediately after both CT and initial PCR results were available. This was true for both negative PCR and positive CT and vice versa, provided that patients had not left our institution in the meantime. Similarly, repeat RT-PCR was performed, when RT-PCR results were negative or indeterminate but suspicion of COVID-19 remained. This study was approved by the Institutional Review Board of Charité Universitätsmedizin Berlin.

2.1. CT protocol

Low-dose chest CT scans for suspected COVID-19 were acquired on two different CT scanners of our hospital: an 80-slice scanner (Aquilion Prime, Canon Medical Systems Cooperation, Otowara, Japan) and a 64-slice scanner (Lightspeed VCT, General Electric, Boston, Massachusetts, United States).

For the Canon Aquilion Prime, imaging parameters were set as followed: tube voltage 100 kV, tube current modulation between 10 and 100 mA, maximum resolution time 0.27 s, noise index 27, pitch factor 1.388 and slice thickness 0.5 mm. Parameters for the Lightspeed VCT were as follows: tube voltage 100 kV, tube current modulation between 10 and 100 mA, 0.35 s maximum resolution time, noise index 39, pitch factor 1.375, and slice thickness 0.625 mm. Iterative reconstruction was used (Canon Aquilion Prime: AIDR 3D; Lightspeed VCT: ASIR) with a lung and a soft tissue kernel (Canon Aquilion Prime: Fc01 and Fc85; Lightspeed VCT: “standard” and “lung”).

2.2. Data acquisition

All CT images were evaluated manually and data on presence/absence of COVID-19 was assessed. Vital parameters (O₂ saturation, respiratory rate, and body temperature) were taken from the initial admission form or request form for chest CT. Further patient characteristics, such as sex or age, were extracted from our Radiology Information System database.

2.3. Statistical analysis

All statistical analysis was conducted using the “R” statistical programming language including the “tidyverse” and “lme4” libraries [12., 13., 14.]. Variables were expressed as means +/− standard deviation if normally distributed and as median and interquartile range (IQR) if not. Categorical variables were expressed as frequencies and percentages. Normal distribution was tested using the Shapiro-Wilk test. To compare continuous variables, the Wilcoxon rank sum test was used while Pearson's chi-square test or Fisher's exact test were used to compare categorical variables. The diagnostic value of vital parameters was evaluated using receiver operating characteristic (ROC) analysis and compared using the area under the curve (AUC). Buderer's formula was employed for post-hoc power analysis of our sample size [15].

The diagnostic accuracy of CT was calculated through a generalized mixed logistic regression model, adapted from the model proposed by Coughlin et al. [16], which was extended by a random intercept to allow adjustment for different scanner types and clinical sites (formula 1).

$$\mathit{logitPr}\left(Y_{\mathit{ij}}=1|X\right)={\beta }_0+{\beta }_1{X}_{\mathit{ij}}+\sum _{k=1}^K{\beta }_k{\gamma }_{\mathit{kij}}+{\delta }_i+{\epsilon }_{\mathit{ij}}$$ (1)

Here, Y_ij is defined as the binary result of chest CT in each patient. Y_ij = 1 is suspected COVID-19 and Y_ij = 0 is absence of COVID-19 in CT. X represents a matrix of covariates including the result of the RT-PCR test (X_ij = 1 confirmed COVID-19, X_ij = 0 absence of COVID-19 as defined by RT-PCR). β₀ is the intercept, β₁ represents the dichotomous result of RT-PCR, and β_k corresponds to Y_k for (k = 1, …, k) patient characteristics for patient i on site (or scanner) j. The random variation in the intercept is denoted by δ_j, while ε_ij indicates the error. 95% confidence intervals were calculated trough parametric bootstrapping with 1000 iterations. Pre- and posttest probabilities of COVID-19 were calculated using likelihood ratios. A p-value of p < 0.05 was considered statistically significant.

3. Results

A total of 269 patients was included in this retrospective study (155 males (58%) with a median age of 71 years (IQR: 24.4)). Assessment of initial vital parameters showed a median respiratory rate of 18/min (IQR = 7/min), O₂-saturation of 97% (IQR: 4%), and body temperature of 37 °C (IQR: 1.4 °C). Typical CT findings of COVID-19 were observed in 79 patients, of whom 44 were male (56%) and 35 were female (44%). In the 269 patients analyzed, COVID-19 was confirmed by RT-PCR in 34 patients (12%), with males (n = 20) being slightly more often affected than females (n = 14). However, this difference was not significant. Of the 34 RT-PCR positive patients, 28 were also CT positive.

Patient characteristics are summarized in Table 1 . It can be seen, that vital parameters different between patients without COVID-19 and patients with COVID-19, with COVID-19 patients showing higher values for CRP, higher body temperature, higher respiratory rate and lower oxygen saturation.

Table 1.

Study population and vital parameters.

	Absence of COVID-19	Presence of COVID-19
Patients (number)	235	34
Women (number, %)	100 (42.6)	14 (41.2)
Age	72, IQR: 25	64.79, SD: 13.47
Vital parameters
Body temperature (°C)	36.9 IQR: 1.4	38.01 SD: 1.12
Respiratory rate (breaths/min)	18 IQR: 5	23.32 SD: 6.99
Oxygen saturation (%)	98 IQR: 4	95 IQR: 9

This table gives an overview of patient characteristics of the overall study population subdivided by absence and presence of COVID-19. It becomes apparent, that the vital parameters body temperature, respiratory rate and oxygen saturation differ between the groups (absence of COVID-19 versus COVID-19), with COVID-19 patients showing higher values for CRP, higher body temperature, higher respiratory rate and lower oxygen saturation. Abbreviations: IQR: Interquartile range (in case of non-normal distribution of data); SD: standard deviation (normal distribution of data).

3.1. Diagnostic accuracy of a clinical prediction model

The diagnostic value of respiratory rate, body temperature, and blood oxygen saturation was investigated to determine whether these vital parameters alone could be used for the diagnosis of COVID-19, thus reducing the need for CT.

We found that the median respiratory rate and blood oxygen saturation differed significantly between patients with confirmed COVID-19 and patients with a negative RT-PCR result. The median respiratory rate of patients with confirmed COVID-19 pneumonia was 23/min (IQR: 7/min) and significantly higher than in patients without infection (18/min, IQR 5/min, Wilcoxon test (W) = 2637.5, p = 0.011). Similarly, patients with COVID-19 showed a significantly higher body temperature of 38.0 °C (IQR: 1.1 °C) and lower blood oxygen saturation of 95% (IQR 9%) compared to patients without COVID-19 (36.9 °C IQR: 1.4 °C, W = 4384, p < 0.001 and 97% IQR: 4%, W = 2029, p = 0.009). Taken together, these data indicate a potential diagnostic value of vital parameters in patients with suspected COVID-19. To compare the diagnostic accuracy of clinical parameters and CT findings, ROC curves were analyzed. The area under the curve (AUC) for respiratory rate was 0.67 (95% CI = 0.53–0.80), 0.75 (95% CI = 0.61–0.84) for body temperature, and 0.65 (95% CI = 0.60–0.75) for blood oxygen saturation. We found the largest AUC for combined clinical parameters (0.80, 95% CI = 0.68–0.90). Nevertheless, the diagnostic accuracy of the combined clinical parameters did not outperform the diagnostic accuracy of CT (see Fig. 1 ).

Fig. 1.

Predictive value of clinical parameters compared to CT.

This figure shows the diagnostic value of vital parameters alone for the diagnosis of COVID-19 and compares its accuracy with that of CT (results obtained with the baseline model). The point estimate of CT accuracy remains above the curve in all subfigures, indicating that vital parameters alone are not more accurate than a CT scan.

3.2. Diagnostic accuracy of CT

Two different models for estimating diagnostic accuracy of CT were constructed. The first, a baseline model, was only corrected for the scanner type, while the second model was also corrected for age, sex, breathing frequency, body temperature, and blood oxygen saturation. We found a sensitivity of 0.82 (95% CI 0.67–0.98) and specificity of 0.78 (95% CI 0.67–0.90) with the baseline model and a sensitivity of 0.86 (95% CI 0.71–1.00) and specificity of 0.78 (95% CI 0.57–0.98) with the models corrected for patient characteristics and initial vital parameters.

3.3. Accuracy of CT as a function of COVID-19 pretest probability

Using patient age, sex, and initial vital parameters, we propose a clinical risk score for COVID-19, which can be calculated as follows: One point is given each for a respiratory rate of ≥20/min, a body temperature ≥ 38 °C, a blood oxygen saturation ≤ 95%, patient age ≥ 65 years, and male sex. Based on this risk score, we defined three risk groups for the presence of COVID-19: low risk (0–1 points), intermediate risk (2–3 points), and high risk (4–5 points).

The pretest probability of COVID-19 was <10% in patients in the low-risk group, 10–25% in patients in the intermediate-risk group, and >25% in the high-risk group.

Based on the CT model, corrected for age, sex and vital parameters, for patients in the low-risk group, the positive predictive value (PPV) and negative predictive value (NPV) were 0.33 (95% CI 0.21–0.46) and 0.97 (0.95–1). In the intermediate-risk group, the PPV decreased to 0.29 (95% CI 0.15–0.44) and the NPV increased by 0.01 to 0.98 (95% CI 0.94–1). In the high-risk group, the PPV increased to 0.54 (95% CI 0.40–0.69), with the NPV only decreasing marginally to 0.94 (95% CI 0.88–1). Table 2 provides an overview of CT accuracy as a function of COVID-19 pretest probability, and Fig. 2 illustrates posttest probabilities for COVID-19 before and after a negative/positive CT result.

Table 2.

Diagnostic accuracy models of CT and CT combined with vital parameters.

Model	Sensitivity	Specificity	PPV	NPV
	(95% CI)	(95% CI)	(95% CI)	(95% CI)
CT (baseline)	0.82 (0.67–0.98)	0.78 (0.67–0.90)	0.35 (0.15–0.53)	0.97 (0.96–1.00)
CT (adjusted)	0.86 (0.71–1.00)	0.78 (0.57–0.98)	0.36 (0.16–0.55)	0.97 (0.95–1.00)
Low-risk	0.76 (0.61–0.91)	0.83 (0.77–0.90)	0.33 (0.21–0.46)	0.97 (0.95–0.99)
Intermediate-risk	0.84 (0.63–1.00)	0.76 (0.65–0.87)	0.29 (0.15–0.44)	0.98 (0.94–1.01)
High-risk	0.89 (0.76–1.00)	0.68 (0.54–0.83)	0.54 (0.40–0.69)	0.94 (0.88–1.00)

This tables shows the results for different models of diagnostic accuracy (baseline CT and CT adjusted for age, sex, and vital parameters) for all patients regardless of risk group, as well as the accuracy when the adjusted CT model was stratified by the pre-test probability of COVID-19.

Fig. 2.

Probability of COVID-19 after CT.

This figure shows the probability of COVID-19 before CT on the x-axis and the calculated and extrapolated posttest probabilities after a positive (blue line) or negative (red line) CT scan. It becomes apparent that for patients in the low-risk group, even after a positive scan, the probability of COVID-19 is only 50%, casting doubt on the appropriateness of using CT in this patient subset. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

4. Discussion

To date, RT-PCR remains the reference standard for the diagnosis of COVID-19, while CT has the advantage of detecting patients with lung infiltrates fast and early. Although the role of chest CT as a screening tool for COVID-19 or other respiratory tract infections is being avidly discussed, it is currently not recommended by the Fleischner Society and the American College of Radiology [12,20].

We found that CT accuracy is influenced by the prevalence/pretest probability of COVID-19 and was highest in patients with a high pretest probability/prevalence estimate of COVID-19. Based on our results, we therefore believe that CT may have added value in the management of COVID-19, not as a screening tool, but for triaging patients in the emergency department to the best treatment path depending on the severity of the disease. In patients presenting with fever and shortness of breath, a positive CT scan might shorten the time to diagnosis and facilitate fast hospitalization and focused treatment. On the other hand, a negative CT result does not completely rule out COVID-19 and especially symptomatic patients should therefore initially continue to be isolated, with further testing to identify possible other causes that explain their symptoms, such as abdominal infections, malignancies or cardiovascular diseases.

Nevertheless, not all patients with some form of respiratory symptoms should undergo CT imaging. Especially in patients with few symptoms (e.g. young patients with normal body temperature), CT examinations are most likely not effective with a very low positive predictive value. As positive CT results may only have added value if the positive predictive value is high [11,16], we advocate our approach of calculating a fast-and-easy to use score to estimate the pre-test probability of COVID-19. Another important aspect is, that maintaining hygiene can be difficult when a larger number of patients need to have a CT scan in a short time, and CT might thus become a hub for new infections [17]. Finally, the results of the present study might be helpful to institutions without rapid access to Covid-19 tests, especially when a low radiation dose protocol for minimization of radiation burden is used.

Luo et al. previously investigated the benefit of a scoring system based on the most common imaging findings to improve CT accuracy [5,18]. They reported score-dependent specificities ranging from 0.23 to 0.95. However, the disadvantage of their approach is that their scoring system requires the patient to undergo a CT first. In our approach, we chose variables that are among the first parameters to be assessed when the patient arrives in the emergency department/hospital. This allows immediate calculation of the pretest probability of COVID-19 and subsequent further triage of patients depending on their pretest probability. As a limitation, we have not included other patient information such as medical history, other diseases (especially lung diseases and obesity), laboratory data and medication, as they were not reliably recorded on initial assessment of patients [[18], 19., [20]]. However, the inclusion of some of these information might have been beneficial, as it could also have allowed for an assessment of the potential severity of the disease course. Leung et al. have shown that chronic obstructive lung disease (COPD) is a risk factor for COVID-19, as the patients affected by both had a more severe course of the disease [19,20]. Furthermore, several laboratory examinations have were demonstrated to be associated with more severe (e.g. elevated procalcitonin) or even less severe (elevated white blood cell count) disease (21).

Some studies, which advocated CT as a screening tool, reported surprisingly high sensitivities [5,18]. Ai et al. found a very high sensitivity of 0.97 for the detection of COVID-19 (higher than our reported values), but a relatively low specificity of 0.25 [5]. In their meta-analysis Xu et al. reported similar values for sensitivity (0.92) and specificity (0.25 and 0.33) [18]. While our sensitivity was lower, our specificity was substantially higher. A possible explanation for this may be the low prevalence of confirmed COVID-19 cases in our patient population (12% versus 59% in Ai et al.) and therefore a high proportion of true negative results [5]. However, it has to be noted that the above-quoted estimates of diagnostic accuracy should be interpreted with caution, as they were obtained in patients with already suspected COVID-19 in a hospital setting and are therefore prone to substantial selection bias [11,19]. The accuracy, especially the negative predictive value of RT-PCR has been criticized in recent studies. Kucirka et al. conducted a meta-analysis and estimated the false negative rate to be 38% by the day of onset of symptoms, which decreased to 20% the third day after onset of symptoms, but then slowly started to increase again [20]. False negative results could therefore also have affected the accuracy of CT in our study, resulting in a high estimate of sensitivity and a low estimate of specificity.

Our study has several more limitations. Similar to other retrospective studies with COVID-19 data, it was performed in a hospital setting, which probably led to some selection bias. Patients presenting in the emergency department have a higher a priori probability of disease, so the accuracy of the CT is not transferable to an outpatient setting. Selection bias could be further enhanced by the fact that the parameters used for our risk score might also have been used by the physician before referring a patient for CT. The lack of patient information regarding their medical history, laboratory data and ongoing limitation has to be acknowledged as another important limitation of this study In addition, the absolute number of COVID-19 cases confirmed by RT-PCR was relatively low. This small sample size may have affected the pre-test probabilities calculated using our risk score, and estimates of diagnostic accuracy of CT might differ to some degree if repeated in larger cohorts. However, we tried to at least partially correct for this uncertainty by calculating 95% confidence intervals for all estimates of diagnostic accuracy. Furthermore, we observed an overlap for clinical data between the infected and non-infected groups, as shown by the IQR, which likely has an impact on the diagnostic performance of the model. Also, the inclusion of the pulse rate might have had a positive effect on our calculation of the pre-test probability. However, since it was not reliably documented during initial patient assessment, it could not be included in our analysis.

5. Conclusions

A simple score to estimate the pre-test probability of COVID-19 can be calculated using age, sex, O₂ saturation, respiratory rate, and body temperature. Provided that only patients with a high pre-test probability undergo the examination, the accuracy of CT for the detection of COVID-19 might be increased.

Declaration of competing interest

There are no competing interests to declare.