How did radiologists’ diagnostic performance has changed in COVID‐19 pneumonia: A single‐centre retrospective study

Abstract

Aims

Delay and false positivity in PCR test results have necessitated accurate chest CT reporting for the management of patients with COVID‐19–suspected symptoms. Pandemic related workload and level of experience on covid‐dedicated chest CT scans might have affected the diagnostic performance of on‐call radiologists. The aim of this study was to reveal the interpretation errors (IEs) in chest CT reports of COVID‐19–suspected patients admitted to the Emergency Room (ER).

Methods

Chest CT scans between March and June 2020 were re‐evaluated and compared with the former reports and PCR test results. CT scan results were classified into four groups. Parenchymal involvement ratios, radiology departments’ workload, COVID‐19–related educational activities have been examined.

Results

Out of 5721 Chest CT scans, 783 CTs belonging to 664 patients (340 female, 324 male) were included in this study. PCR test was positive in 398; negative in 385 cases. PCR positivity was found to be highest in “normal” and “typical for covid” groups whereas lowest in “atypical for covid” and “not covid” groups. 5%‐25% parenchymal involvement ratio was found in 84.2% of the cases. Regarding the number of chest CT scans performed, radiologists’ workload has found to be increased six‐folds. With the re‐evaluation, a total of 145 IEs (18.5%) have been found. IEs were mostly precipitated in the first two months (88.3%) and mostly in the “not covid” class (60%) regardless of PCR positivity. COVID‐19 and radiology entitled educational activities along with the ER admission rates within the first two months of the pandemic have seemed to be related to the decline of IEs within time.

Conclusion

COVID‐19 pandemic made a great impact on radiology departments with an inevitable burden of daily chest CT reporting. This workload and concomitant factors have effects on diagnostic challenges in COVID‐19 pneumonia.

What’s known

• The radiologists role is crucial in interpretation of chest CT scans in cases of atypical pneumonia.

What’s new

• Pandemic related workload and level of experience on covid‐dedicated chest CT scans have affected thediagnostic performance of on‐call radiologists resulting in the possibility of interpretation errors.

1. INTRODUCTION

COVID‐19 pneumonia was detected for the first time in the world in Wuhan, China in 2019 and has spread all over the world over time. Since the virus, SARS‐CoV‐2, can lead to a wide spectrum of findings, additional testing is a must to make the correct diagnosis. The sensitivity of the real‐time reverse transcription‐polymerase chain reaction (RT‐PCR) tests used for diagnosis is variable and the use of Computed Tomography (CT) is very important for diagnosis in cases with pneumonia. ¹ , ² , ³ As the number of cases worldwide increased and reached 62 million by October 2020, CT findings of COVID 19 pneumonia have become better recognised and the scope of findings has expanded. ⁴ , ⁵ The world has encountered similar important outbreaks up to now; that is, 1918‐influenza, 2003‐SARS, 2009‐H1N1, 2012‐MERS and 2015‐Zika virus. Since these viral infections created public health emergencies, lots of effort has been spent to make a quick and accurate diagnosis. This, in turn, made necessary contributions to the knowledge of novel imaging findings of these diseases. ⁶ , ⁷ , ⁸ Since March 11, 2020; when the first case was seen in our country, CT examination was routinely performed on patients with suspicion of covid pneumonia together with the PCR testing. Regarding examples of increasing experience in the interpretation of the imaging findings in past crises, we think that our experience of CT findings in COVID‐19 pneumonia might have increased with the number of cases in this timeframe. Besides, there may have been differences in the evaluation of CT examinations due to the increased workload in the peak admission days. In this study, it was aimed to find out the possible changes in the diagnostic performance level of the radiology department by retrospectively re‐evaluating the reports of the chest computed tomography images of the patients who were admitted to the emergency room (ER) with the suspicion of COVID‐19.

2. METHODS AND MATERIALS

2.1. Patient selection

Chest computed tomography of patients who were admitted to our hospital's ER with suspicion of COVID‐19 between March and June 2020 were enrolled. Among these, patients who have RT‐PCR tests performed via nasopharyngeal and oropharyngeal swab specimens were selected. PCR tests had been finalised within 24 to 96 hours in the first days of the epidemic in our country. Patients whose CT scans were performed with a time gap longer than 72 hours with PCR testing were excluded from the study.

Ethics committee approval was received for this study both from the institutional Ethical Committee and The Ministry of Health.

2.2. CT acquisition technique

Radiological assessment of patients included unenhanced Chest CT imaging with covid‐dedicated scanning protocols in two scanners (128‐MDCT Siemens Somatom Definition; 16‐MDCT Toshiba Alexion): supine, end‐inspiration acquisition; slice thickness, 1.0‐1.5 mm; tube voltage, 120 kV; tube current, 200‐300 mAs; pitch factor, 1.5; multiplanar reformations with mediastinal and lung parenchymal window settings.

2.3. Radiological evaluation

CT images were retrospectively re‐evaluated by three radiologists. Typical and atypical chest CT findings related to COVID‐19 pneumonia were evaluated separately by each radiologist blinded to the previously written reports. Multifocal ground‐glass opacities (GGO), consolidation, GGO with superimposed consolidation, consolidation predominant pattern, crazy paving pattern, and melted sugar sign were considered as typical; pleural and/or pericardial effusion, cavity, pulmonary nodule, nodular pattern, lymphadenopathy, peribronchovascular distribution, halo and/or reverse halo sign, three‐in‐bud sign, bronchiectasis, airway secretions, pulmonary emphysema, pulmonary fibrosis, isolated pleural thickening and pneumothorax were considered as atypical findings for covid pneumonia. ⁹ Patients were categorised as “normal,” “typical for covid,” “atypical for covid” and “not covid” similarly with the previous structured reports. The “Not covid” condition was considered in cases where there are pathologies that are not listed under the aforementioned typical and atypical covid pneumonia findings; that is, mass, lobar consolidation. Afterward, parenchymal involvement ratios were visually defined for “typical for covid” and “atypical for covid” groups; <25%, 26%‐50%, 51%‐75% and 76%‐100%. In case of a conflict between evaluations, the decision was made with the consensus of three radiologists. In addition to this re‐evaluation, a fourth radiologist searched for preliminary reports, final reports and radiology consultation notes of these patients. The point of this thoughtful research is to find out the very first on‐call radiologist's comment on CT images.

Results of the re‐evaluation were compared with previously written reports and also with PCR test results. In this way, the diagnostic performance of CT reports was aimed to be determined throughout the whole process from the first stage of the epidemic to the present.

2.4. Workload and educational activities

The number of Thoracic CTs acquired for each 24‐hour working period was recorded to point out the possible diagnostic differences on the high‐intensity workdays. Also, the total number of reported COVID‐19 pneumonia–suspected CTs were cumulatively calculated to find out the breakpoint of the departmental learning curve regarding accurate diagnosis. Schedules of in‐department teaching activities, face‐to‐face meetings, and online meetings, conferences, webinars organised by the National Radiology Association were also noted. The possible positive effects levering the quality of CT reporting for covid‐pneumonia were investigated.

2.5. Statistical analysis

Statistical analyses were performed using the SPSS software version 22. Distribution of clinical characteristics across groups of CT and RT‐PCR results were presented with frequency tables. The chi‐square test was used to compare these proportions in different groups. The chi‐square test or Fisher's exact test was performed to test the significance of pairwise differences using Bonferroni correction to adjust for multiple comparisons. An overall %5 type‐I error level was used to infer statistical significance.

3. RESULTS

In this cohort; a total of 5721 chest CT scans were found to have acquired at our institution during March, April and May 2020. Chest CT’s were mostly requested in patients with symptoms of persistent cough, fever, history of a pulmonary nodule, mass, chest operation, trauma etc1478 patients were removed from the study due to the long latency of PCR results. A total of 3460 patients who did not have a PCR test at our institution were also removed. The remaining 783 Chest CT scans of 664 patients who have PCR test results were retrospectively included (Figure 1). In total, 82 patients had multiple CT scans during their hospitalisation period. In total, 340 patients were female (51.2%) and 324 patients were male (48.8%). RT‐PCR test was positive in 279; negative in 385 patients.

FIGURE 1.

Study flow‐chart

3.1. Radiological evaluation

A total of 783 chest scans were re‐evaluated with three separate radiologists blinded to the former CT scan reports and PCR test results. Parenchymal involvement analysis was executed via visual semi‐quantitative analysis of the three radiologists. A significant amount of CT scans resulted in only mild (<25%) parenchymal involvement (n = 405, 84.2%). A total of 57 patients showed 25%‐50% parenchymal involvement (11.85%). Only 19 patients showed a higher percentage (>50%) of parenchymal involvement (3.95%) (Table 1).

TABLE 1.

Parenchymal involvement ratios among COVID‐suspected chest‐CT scans

Chest‐CT	5%‐25%	26%‐50%	51%‐75%	76%‐100%	Total
Typical for covid	289	55	13	6	363
Atypical for covid	116	2	0	0	118
Total	405	57	13	6	481
%	84.20	11.85	2.70	1.25	100.00

A total of 132 CT scans were classified as “normal” whereas 170 CT scans as “not covid.” PCR test results were positive in nearly all of the “normal” group (n = 121, 91.7%); whereas negative in most of the “not‐covid” group (n = 143, 84.1%). The remaining 481 CT scans were suspicious for COVID‐19 pneumonia. PCR test results were positive in 59.2% of “typical for covid” classified scans, while negative in 70.3% of “atypical for covid” ones. The distribution of PCR test results among Chest CT subgroups has shown a statistically significant difference (P < .001). (Table 2).

TABLE 2.

Intergroup comparison of chest CT images according to interpretation error status and PCR test results

Chest‐CT	IE−	IE+	P
Normal	130 (98.5)	2 (1.5)	<.001
Typical for covid	351 (96.7)	12 (3.3)
Atypical for covid	74 (62.7)	44 (37.3)
Not covid	83 (48.8)	87 (51.2)

	PCR−	PCR+	P
Normal	11 (8.3)	121 (91.7)	<.001
Typical for covid	148 (40.8)	215 (59.2)
Atypical for covid	83 (70.3)	35 (29.7)
Not covid	143 (84.1)	27 (15.9)

Comparison of re‐evaluation with former documents related to the CT scans has shown that radiological evaluations of a total of 145 CT scans (18.5%) have changed over time. After this point, these diagnostic differences will be referred to as “interpretation errors” (IE). The number of CT scans, interpretation errors and their monthly distribution had calculated (Figure 2). Afterward, results of CT scans according to the aforementioned four classifications (normal, covid, typical for covid, atypical for covid and not covid) were correlated with the number of interpretation errors and PCR test results. The distribution of IEs among results of CT scans has shown a statistically significant difference (P < .001). (Table 2) Furthermore, it has been found that IEs were mostly precipitated in the “not covid” group regardless of PCR positivity (n = 87, 60%), (Table 3). The “atypical for covid” group was found to be the second most common one that IEs have determined (n = 44, 30.34%). The ratio of IEs in “normal” and “typical for covid” groups constitute less than ten per cent of all (9.66%).

FIGURE 2.

Monthly distribution of chest CT scans and interpretation errors

TABLE 3.

Distribution of interpretation errors according to chest‐CT, PCR test results and number of chest‐CTs

Chest‐CT/IE	PCR−	PCR+	IE/chest‐CT (%)
Normal	0	2	1.52
Typical for covid	1	11	3.31
Atypical for covid	29	15	37.29
Not covid	65	22	51.18
Total	95	50	18.52

3.2. Workload volume and educational activities

Radiologists’ workload volume especially in on‐call hours varies mostly depending on the ER admission rates. At our institution, primarily one radiologist at night shifts and one radiologist at working hours were responsible for emergency cases. The total number of chest CT scans reported has markedly increased up to 140 cases/day.

Educational activities dedicated to the novel‐coronavirus infection that took place within the radiology department and also within institutional panels, symposiums organised by the national radiological society, ministry of health and online meetings have been investigated with their dates (Appendix 1). It has been found that local/national educational activities mostly aggregated at the end of March and the first half of April.

Results of re‐evaluation of chest CT scans were correlated with former documents and gold‐standard PCR test results. IEs were correlated with the workload and educational activities. In this way, the learning curve in radiological diagnosis of COVID‐19 pneumonia was tried to be established. It has been found that the number of chest CT scans increase to the top level at the beginning of April then gradually decreases. On the other hand, the distribution of the ratio of IEs to the number of chest CT scans among these three months has pointed out that the ratio of IEs gradually decreases more than half after April, from 20.2% to 11.18% (Figure 2).

4. DISCUSSION

In our study, we aimed to expose the radiologists’ increasing experience on the evaluation of chest CT findings of COVID‐19 pneumonia. We have focused on some unobserved challenges on radiologists’ in the COVID‐19 pandemic to better understand the discrepancy in diagnostic performances.

Correlation of re‐evaluated chest CT results with PCR tests has demonstrated that the PCR test was negative in 84.1% of the scans in the “not covid” group. 91.7% PCR‐positivity was calculated in the cases labelled for “normal.” This finding can be explained with the vigorous use of CT in the ER setting, even for the patients who have COVID‐19 infection but lack pneumonia. Calculations with “typical for covid,” “atypical for covid” and a combination of these two groups have resulted in 62.8% sensitivity and 40% specificity of chest CT according to PCR test results.

We have identified diagnostic differences in 18.5% of the chest CT scans in our cohort. The distribution of this IEs have shown a dramatic accumulation within the first 60 days of the pandemic in our country. After this point, the IEs decrease down to 11.18% (Figure 2). Some reasons can explain the possible causes of these relatively high IE ratios and their course among the first three months of the pandemic. First of all, both national and global total number of cases and new confirmed cases prominently increased within the first two months. The number of total cases significantly increase from ~116.000 to ~3.2 million with an increase in the new confirmed case from 4600 to 86.000/day worldwide. Similarly, a total of ~120.000 cases have been identified in our country between 11th March and 1 May 2020 with 2615/day new confirmed cases on 1 May. ⁵ Our institution as well as most of the healthcare centres worldwide have faced striking hospital admission rates up to ~1000/day with the spread of coronavirus. Between the 1st of May and the 1st of June, new confirmed daily cases decrease by 67.92%. As of the 8th of January 2021, the cumulative number of covid cases is nearly 1.5 million in our country. ⁵ We have re‐evaluated the study cohort with a remarkable experience of ~17.000 chest CT scans reported in the past 9 months at our institution.

During the pandemic, the workload of radiologists has expeditiously increased parallel to the admissions with the suspicion of COVID‐19. According to the COVID‐19 guidelines published by the ministry of health (Appendix 2), chest CT scanning has become the secondary diagnostic tool after the gold standard PCR testing. To overcome the delay in diagnosis due to the PCR testing, clinicians were encouraged to use the CT scan as the quickest way to isolate/hospitalise the COVID‐19 infected patients. ¹⁰ Moreover, chest CT gained an important role in the clarification of PCR‐false negative but clinically highly suspected cases. ¹¹ Also as Saket D has stated, CT orders from ER have prominently risen due to the medical training focusing heavily on imaging. ¹² These all have seemed to be contributed to the acquisition of 5721 chest CTs within these three months. For comparison, ~900 chest CTs were performed at our institution between the same months of the last year (March to June 2019). Radiologists on shifts were directly affected by this six‐fold increase; up to a total of 140 covid‐suspected chest CT scans were reported in a day, along with the other non‐covid emergencies. Mossa‐Basha M et al have emphasised the importance of education, discussion and communication with ER physicians which significantly decreased the daily number of CT requests from 9‐10 to 2‐3 at their institution. ¹³ On the other hand, Ohana et al have demonstrated a peak number of 64 CT scans/day during the first wave of the pandemic at their institution in North‐Eastern France. Despite their strikingly high numbers, the highest number of daily CT scans in our study was more than twice their report. Another difficulty was that the covid dedicated CT scanning protocol included ~320 axial slices with a 1 mm slice thickness which makes it more time consuming than the standard chest CT scanning protocol with 3‐5 mm slice thickness. These factors have influenced the average reporting time, which was ~10‐30 minutes after the acquisition. All in all exponentially increased admission rates along with the increased number of covid dedicated, thin slice chest CT scans have become grounds for possible interpretation errors.

Another point worth mentioning is that; although most typical signs of covid pneumonia on chest CT had first been identified in Wuhan‐China, numerous atypical signs had evolved with the spread of the infection to a different profile of patients. As Falaschi et al stated, incorporation of China experience along with the other previously contaminated countries into daily clinical practice seems to have caused an improvement in our diagnostic performance. ⁴ The recognition of the relationship between atypical findings for COVID‐19 pneumonia on Chest CT may have contributed to the decline of IE rate. Earlier reports in the first wave of the pandemic have mentioned the need for further characterisation of the imaging features of COVID‐19. ¹⁵ Similarly, Tsou et al have emphasised the importance of defining the typical & atypical imaging findings and educational activities for an accurate diagnosis in SARS outbreak. ⁷ Another possible contributor is the aggregation of educational activities focused on radiological findings of covid pneumonia in the first two months. Our results have pointed out a learning curve that has improved after the first two months. A single centre study covering 2278 chest CTs similarly draws attention to the possible effects of education and level of exposure on the diagnostic performance of chest CTs. ¹⁴

Further analysis has shown a marked uneven distribution of IEs among the four CT result groups. More than half of the errors have been made in the “not covid,” followed by the “atypical for covid” group. IEs in the re‐evaluated “not covid” group has mostly consisted of incorrect diagnosis for “atypical for covid.” Similarly, analysis of re‐evaluated “atypical covid” group have shown a marked incorrect diagnosis for “typical for covid.” These findings showed that the radiological hallmarks of covid pneumonia were successfully identified but there was a lack of experience and knowledge, specifically about atypical radiological findings. This situation had probably created a tendency not to exclude the covid diagnosis, yet point out a suspicion with the statement of “atypical for covid” in CT reports. Another issue to mention is that there are subspecialised academic and staff radiologists in our department. The COVID‐19 pandemic has necessitated rapid responses from radiology departments to overcome the increased demand for diagnosis and management of covid pneumonia. Similar to most radiology departments’ management procedures, nearly all radiologists were integrated into a team dedicated to COVID‐19 diagnosis along with the partial cancellation of elective diagnostic/interventional procedures and outpatient diagnostic services in our department. ¹⁶ , ¹⁷ , ¹⁸ The subspecialised radiologists have faced an unexpected challenge in this crisis to maintain their versatility in reporting chest CT scans. ¹⁹ Similarly, Shi et al and Cavallo et al have also mentioned imperative outside‐role definition of radiology employees. ²⁰ , ²¹ A second look by chest imaging specialised radiologists might improve the diagnostic performance of CT scans that were previously reported by diverse sub‐speciality radiologists. ¹⁴ Hereby, this situation might also have contributed to the escalation of interpretation errors. Considering together with the increased workload and physical and psychological stress of the pandemic, it seems nearly inevitable to make IEs in the COVID‐19 outbreak.

There are some limitations to this study. First, we could have included only 13.6% of the Chest CT scans due to lack or delay of PCR test results. Second, we have ignored the decline in non‐COVID CT cases per/day and focused on COVID dedicated CT reporting volume. Retrospective design and the failure to control which trainings radiologists have attended are the other limitations.

There is no doubt that the COVID‐19 pandemic made a huge impact on radiology practices worldwide. An increasing level of experience over time promotes the performance of the radiology departments in the diagnosis of chest CT findings in COVID‐19 patients. COVID‐19–related admission rates and relevant changes in the daily workload, rotations of subspecialised personnel for urgent health care management are other factors affecting the diagnostic performance.

CONFLICTS OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

AMK and ZHA contributed to conceptualisation. AMK and ZHA contributed to methodology. AMK, TA and LA contributed to investigation. AMK, TA, LA, AA and ZHA contributed to data curation. AMK, TA and AA contributed to writing & editing. ZHA contributed to supervision.

Supporting information

 

 

Koc AM, Altin L, Acar T, Ari A, Adibelli ZH. How did radiologists’ diagnostic performance has changed in COVID‐19 pneumonia: A single‐centre retrospective study. Int J Clin Pract. 2021;75:e14693. 10.1111/ijcp.14693