Radiological Society of North America Chest CT Classification System for Reporting COVID-19 Pneumonia: Interobserver Variability and Correlation with Reverse-Transcription Polymerase Chain Reaction

Abstract

Purpose

To evaluate the Radiological Society of North America (RSNA) chest CT classification system for reporting coronavirus disease 2019 (COVID-19) pneumonia.

Materials and Methods

Chest CT scans of consecutive patients suspected of having COVID-19 were retrospectively and independently evaluated by two chest radiologists and a 5th-year radiology resident using the RSNA chest CT classification system for reporting COVID-19 pneumonia. Interobserver agreement was evaluated by calculating weighted κ coefficients. The proportion of patients with real-time reverse-transcription polymerase chain reaction (RT-PCR)–confirmed COVID-19 in each of the four chest CT categories (typical, indeterminate, atypical, and negative features for COVID-19) was calculated.

Results

In total, 96 patients (61 men; median age, 70 years [range, 29–94]) were included, of whom 45 had RT-PCR–confirmed COVID-19. The number of patients assigned to chest CT categories typical, indeterminate, atypical, and negative by the three readers ranged from 18 to 29, 26 to 43, 19 to 31, and 5 to 8, respectively. The κ coefficient among the chest radiologists was 0.663 (95% confidence interval [CI]: 0.565, 0.761). κ coefficients among the chest radiologists and the 5th-year radiology resident were 0.570 (95% CI: 0.443, 0.696) and 0.564 (95% CI: 0.451, 0.678), respectively. The proportion of patients with RT-PCR–confirmed COVID-19 in the chest CT categories typical, indeterminate, atypical, and negative for the three readers ranged from 76.9% to 96.6%, 51.2% to 64.1%, 2.8% to 5.3%, and 20% to 25%, respectively.

Conclusion

The RSNA chest CT classification system for reporting COVID-19 pneumonia has moderate-to-substantial interobserver agreement. However, the proportion of RT-PCR–confirmed COVID-19 cases in the categories atypical appearance and negative for pneumonia is nonnegligible.

Supplemental material is available for this article.

© RSNA, 2020

Summary

The RSNA chest CT classification system for reporting COVID-19 pneumonia has moderate-to-substantial interobserver agreement, but the proportion of RT-PCR–confirmed COVID-19 cases in the categories atypical and negative is nonnegligible.

Key Points

• ■ The Radiological Society of North America chest CT classification system for reporting COVID-19 pneumonia has moderate-to-substantial interobserver agreement.

• ■ The proportion of reverse-transcription polymerase chain reaction–confirmed COVID-19 cases in the categories atypical appearance and negative for pneumonia is nonnegligible.

Introduction

Coronavirus disease 2019 (COVID-19) is currently a pandemic and poses a major public health danger (1–5). Mortality rates among symptomatic patients may be as high as 5.6% for China and 15.2% outside of China (6). COVID-19 spreads easily from person to person (7–9). Hospitals should screen patients suspected of having COVID-19 to keep infected patients strictly isolated from noninfected patients and health care workers. The real-time reverse-transcription polymerase chain reaction (RT-PCR) assay is currently the standard of reference for diagnosing COVID-19. However, RT-PCR test is suboptimal for rapid triaging: it takes several hours before results become available, and sensitivity of the test may be insufficient to reliably exclude COVID-19 (10–14). Accordingly, RT-PCR testing should be repeated in patients with a negative initial result and persistent clinical suspicion of COVID-19 (10–14). Chest CT may be an attractive alternative or adjunct to RT-PCR testing because it can be performed rapidly. A study among more than 1000 Chinese patients reported that chest CT has a high sensitivity for the diagnosis of COVID-19 and that it may currently be considered as a primary tool for COVID-19 detection in epidemic areas (15). The promising results of the study by Ai et al (15) and the clinical need for a fast screening tool have led to the introduction of chest CT for patients suspected of having COVID-19 in our hospital in mid-March 2020. From the initial experience, we have learned that the interpretation of chest CT in patients suspected of having COVID-19 in frontline clinical practice is not always straightforward. This can be attributed to the relative lack of experience in interpreting chest CT in suspected COVID-19, the lack of clear and uniform diagnostic criteria in the literature, and CT imaging findings that may overlap with other lung diseases. As disagreement among CT interpreters can result in dissimilar diagnoses and subsequent patient management recommendations, high interobserver agreement is crucial before chest CT can be routinely used in practice. A chest CT classification scale may reduce differences in accuracy by reader experience and improve diagnostic performance. Recently, the Radiological Society of North America (RSNA) Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19 was published (16). Four categories for standardized CT reporting of COVID-19 were proposed based on current literature and expert consensus (16). However, this proposed system has not been evaluated yet to our knowledge. Therefore, the purpose of our study was to evaluate the RSNA chest CT classification system for reporting COVID-19 pneumonia.

Materials and Methods

This retrospective study was approved by the institutional review board of our hospital (Zuyderland Medical Center, Heerlen/Sittard/Geleen, the Netherlands; IRB number Z2020061), and patient consent was waived.

Patients and CT Protocol

Consecutive patients who presented with clinical suspicion of COVID-19 (ie, fever, cough, and/or shortness of breath [17]) in our hospital between March 12 and March 23, 2020, were potentially eligible for inclusion. Most patients had severe clinical symptoms and were being considered for hospitalization. Patients with known COVID-19 (proven by RT-PCR testing) before CT scanning were excluded. Cases that did not comply with the standard of reference (see paragraph below) were also excluded. The first 60 patients were already reported in our pilot study that examined the feasibility of chest CT for screening (submitted manuscript under review). Chest CT was performed with either a 64-slice CT scanner (Philips Incisive) or a 64-slice dual source scanner (Siemens Somatom Definition Flash). Scanning parameters were as follows: collimation 64 × 0.625 or 0.6 mm, 120 kVp, 667 maximum mA or 404 maximum mA, pitch 1.0 or 1.2, and matrix size 512 × 512. CT images were reconstructed in the transverse plane with 1.0-mm slice thickness and 1.0-mm increment. Images were also reconstructed in axial, coronal, and sagittal planes with 3.0-mm slice thickness.

RSNA Chest CT Classification System for Reporting COVID-19 Pneumonia

Four categories for standardized reporting of chest CT findings related to COVID-19 were proposed by the RSNA Expert Consensus Statement (16), that is, typical, indeterminate, atypical, and negative (Table 1). Examples of typical and indeterminate CT imaging features for COVID-19 are shown in Figures 1 and 2, respectively.

Table 1:

RSNA Chest CT Classification System for Reporting COVID-19 Pneumonia

Figure 1:

Example of typical CT imaging features for COVID-19 in a 55-year-old male patient. Chest CT image shows bilateral multifocal ground-glass opacities (arrows), which showed a posterior part/lower lobe predilection and mainly peripheral/subpleural distribution.

Figure 2:

Example of indeterminate CT imaging features for COVID-19 in a 36-year-old female patient. Chest CT image shows bilateral multifocal ground-glass opacities (arrows), which were mainly located in the right upper lobe. There was no posterior part/lower lobe predilection, and there was also no peripheral/subpleural distribution of lung abnormalities.

CT Analysis

CT scans were retrospectively and independently read by two chest radiologists (J.K. and T.M.H.d.J.) with 5 and 22 years of experience in chest CT interpretation and by a 5th-year radiology resident (B.A.C.M.F.) using the RSNA chest CT classification system (16) as mentioned in the previous paragraph. Before chest CT interpretation, the readers studied the literature with regard to the typical chest CT imaging features of COVID-19 pneumonia (18–22). At the time of chest CT interpretation, the readers were only aware of age, sex, and the clinical information as provided by the referring physician.

COVID-19 Reporting and Data System

In an additional analysis, all chest CT scans were also analyzed according to the recently published COVID-19 Reporting and Data System (CO-RADS) (23). CO-RADS uses a five-point scale of suspicion for pulmonary involvement of COVID-19 at chest CT (CO-RADS 5: very high level of suspicion; CO-RADS 4: high level of suspicion; CO-RADS 3: equivocal findings; CO-RADS 2: low level of suspicion; and CO-RADS 1: very low level of suspicion) (23).

Reference Standard

Nasal and pharyngeal swabs were collected for RT-PCR testing directly after chest CT. Patients with a negative initial RT-PCR result and persistent clinical suspicion (note that results of the first RT-PCR were available after 4 hours) were retested. Patients with any positive RT-PCR result were considered to be infected with COVID-19, whereas patients with (persistent) negative RT-PCR results were considered not to be infected with COVID-19.

Statistical Analysis

The degree of interobserver agreement was evaluated by calculating weighted κ coefficients. κ coefficients of 0–0.20, 0.21–0.40, 0.41–0.60, 0.61–0.80, and 0.81–1.00 were considered to indicate none to slight, fair, moderate, substantial, and almost perfect agreement, respectively (24). The proportion of patients with RT-PCR–confirmed COVID-19 in each of the chest CT categories was calculated for each of the readers. A one-way analysis of variance was conducted to assess whether there were differences in patient’s duration of symptoms between the four categories of the RSNA chest CT classification system in patients with RT-PCR–confirmed COVID-19. Analyses were executed using Microsoft Excel 2010 (Redmond, Wash) and MedCalc statistical software version 12.6.0 (MedCalc Software, Ostend, Belgium).

Results

The patient selection flow diagram is displayed in Figure 3. In total, 109 consecutive patients were potentially eligible for inclusion, and 13 of the 109 patients were excluded because they did not comply with the reference standard (12 patients did not undergo RT-PCR testing, whereas one patient with a negative initial RT-PCR result and persistent clinical suspicion did not undergo repeated RT-PCR testing).

Figure 3:

Flow diagram of patient selection.

Eventually, 96 patients (61 men; median age, 70 years [range, 29–94]) were included, of whom 45 (47%) had RT-PCR–confirmed COVID-19. The duration of symptoms before chest CT was reported in 36 of 45 patients (80%) with COVID-19, with a median of 7 days (range, 2–21 days).

The number of patients assigned to categories typical, indeterminate, atypical, and negative of the RSNA chest CT classification system (16) by the three readers ranged from 18–29, 26–43, 19–31, and 5–8, respectively (Table 2). κ coefficients between pairs of each of the three readers are displayed in Table 3. Using the RSNA chest CT classification system, there was substantial interobserver agreement between the chest radiologists (κ coefficient of 0.663) and moderate interobserver agreement between the chest radiologists and the 5th-year radiology resident (κ coefficients of 0.570 and 0.564, respectively). The proportion of patients with RT-PCR–confirmed COVID-19 in each of the categories of the RSNA chest CT classification system (16), as assigned by the readers, varied as follows: typical: 76.9%–96.6%; indeterminate: 51.2%–64.1%; atypical: 2.8%–5.3%; and negative: 20%–25% (Fig 4). Of all 45 patients with RT-PCR–confirmed COVID-19, 62.2% (28/45), 37.8% (17/45), and 44.4% (20/45) were called typical by chest radiologist 1, chest radiologist 2, and the 5th-year radiology resident, respectively. There were no significant differences in the patient’s duration of symptoms between the four categories of the RSNA chest CT classification system as assigned by the readers (for chest radiologist 1: F_3,32 = 0.971; P = .418); for chest radiologist 2: F_3,32 = 1.581; P = .213; and for the 5th-year radiology resident: F_3,32 = 1.542; P = .223).

Table 2:

Number of Patients Per Category of the RSNA Chest CT Classification System (16) as Assigned by Each of the Three Readers

Table 3:

Weighted κ Coefficients between Pairs of Readers Using the RSNA Chest CT Classification System

Figure 4:

Proportion of reverse-transcription polymerase chain reaction–confirmed COVID-19 cases in each of the four categories according to the RSNA chest CT classification system (16) for each of the three readers.

Using CO-RADS, there was substantial interobserver agreement between the chest radiologists (κ coefficient of 0.773) and between the chest radiologists and the 5th-year radiology resident (κ coefficients of 0.658 and 0.648, respectively) (Table E1 [supplement]). The proportion of patients with RT-PCR–confirmed COVID-19 in each of the categories of CO-RADS (23) varied as follows: CO-RADS 5: 81.8%–96.7%; CO-RADS 4: 33.3%–76.9%; CO-RADS 3: 34.6%–50%; CO-RADS 2: 3.1%–5.6%; and CO-RADS 1: 11.1%–22.2% (Fig E1 [supplement]).

Discussion

At present, the role of chest CT in diagnosing COVID-19 is not completely clear. According to the seventh edition of the Chinese Novel Coronavirus Pneumonia Diagnosis and Treatment Plan, COVID-19 may be suspected based on epidemiologic history and clinical presentation, which includes chest imaging findings (including chest radiography and CT) (25). However, chest CT is not described as a diagnostic criterion for COVID-19 (25). The American College of Radiology recommends that chest CT should not be used to screen for or as a first-line test to diagnose COVID-19 (26). The Royal College of Radiologists stated that, based on current evidence, there is no role for CT in the diagnostic assessment of patients suspected of having COVID-19 in the United Kingdom (27). Many other national radiologic societies have not made (clear) recommendations or statements yet with regard to the role of chest CT in diagnosing COVID-19. In practice, frontline physicians may request chest CT in patients suspected of having COVID-19 for faster triaging or as an extra diagnostic tool, which is also the case in our hospital. In addition, chest CT may be performed for other reasons in patients with COVID-19 who are still not diagnosed with COVID-19. Radiologists who interpret chest CT studies should be vigilant for possible COVID-19 infection, especially in endemic areas. COVID-19 is a new disease, and chest CT interpretation in patients with possible COVID-19 may not always be straightforward. The recently published Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19 by the RSNA may provide radiologists and referring clinicians guidance and confidence in reporting these findings and a more consistent framework to improve clarity (16).

The four-category RSNA chest CT classification system for reporting COVID-19 pneumonia was based on current literature and expert consensus (16). We found substantial interobserver agreement between chest radiologists and moderate interobserver agreement between chest radiologists and a 5th-year radiology resident when using this system in patients suspected of having COVID-19. It should be noted, however, that the proportion of RT-PCR–confirmed COVID-19 cases in the categories atypical appearance and negative for pneumonia was nonnegligible. Interestingly, the proportion of RT-PCR–confirmed COVID-19 cases was lower in the atypical appearance category (2.8%–5.3%) than in the negative for pneumonia category (20%–25%). This can be explained by the fact that this study included symptomatic patients (ie, fever, cough, and/or shortness of breath) and that the atypical appearance category also included abnormalities consistent with another lung disease (not COVID-19). Therefore, the prevalence of diseases other than COVID-19 (eg, bacterial lobar pneumonia, bronchial and bronchiolar infections, and typical cardiogenic pulmonary edema) was considerably higher in the atypical appearance category than in the negative for pneumonia category, whereas the opposite was true for the prevalence of COVID-19 between these two categories. On the other hand, as expected, the proportion of RT-PCR–confirmed COVID-19 cases increased from categories indeterminate to typical for all readers.

At the time of conducting this study, another chest CT classification scale for diagnosing COVID-19 pneumonia was circulating in the Netherlands, CO-RADS, which has recently been published (23). The RSNA chest CT classification system for reporting COVID-19 pneumonia (16) and CO-RADS are very similar (categories typical, indeterminate, atypical, and negative for pneumonia of the RSNA chest CT classification system (16) are essentially equal to CO-RADS categories 5, 4–3, 2, and 1 (23), respectively). Not surprisingly, there were no real differences when using CO-RADS (23).

Our study had some potential limitations. First, because of the limited availability of RT-PCR kits in our hospital, it was not feasible to retest all patients with a negative initial RT-PCR result. Accordingly, only 19 of 52 patients (37%) with an initial negative RT-PCR result underwent repeated RT-PCR testing. However, according to our reference standard, all patients with persistent clinical suspicion were retested. Second, each of the three readers assigned relatively few patients (5 up to 8) to the negative for pneumonia category of the RSNA chest CT classification system (16). The relatively limited sample size in this category can be explained because most patients had severe clinical symptoms and were being considered for hospitalization. Third, the prevalence of COVID-19 was relatively high (47%) in our study population. The proportion of RT-PCR–confirmed COVID-19 cases in each of the categories may be different in areas with different COVID-19 prevalence. Fourth, there was a wide variation in the duration of symptoms before chest CT (median of 7 days [range, 2–21 days]), whereas it is known that chest CT appearance of COVID-19 can dramatically change over time (28). However, this variation reflects clinical practice, as some patients present earlier in the course of the disease while other patients present later in the course of the disease. In addition, there were no significant differences in the patient’s duration of symptoms between the four chest CT categories. Fifth, our study had a retrospective design. A prospective study is needed to validate our findings in an independent and larger sample of patients.

In conclusion, the RSNA chest CT classification system for reporting COVID-19 pneumonia has moderate-to-substantial interobserver agreement. However, radiologists and clinicians should take into account that the proportion of RT-PCR–confirmed COVID-19 cases in the categories atypical appearance and negative for pneumonia is nonnegligible.