Diagnostic Performance of CO-RADS and the RSNA Classification System in Evaluating COVID-19 at Chest CT: A Meta-Analysis

Abstract

Purpose

To determine the diagnostic performance of the COVID-19 Reporting and Data System (CO-RADS) and the Radiological Society of North America (RSNA) categorizations in patients with clinically suspected coronavirus disease 2019 (COVID-19) infection.

Materials and Methods

In this meta-analysis, studies from 2020, up to August 24, 2020 were assessed for inclusion criteria of studies that used CO-RADS or the RSNA categories for scoring chest CT in patients with suspected COVID-19. A total of 186 studies were identified. After review of abstracts and text, a total of nine studies were included in this study. Patient information (n¸ age, sex), CO-RADS and RSNA scoring categories, and other study characteristics were extracted. Study quality was assessed with the QUADAS-2 tool. Meta-analysis was performed with a random effects model.

Results

Nine studies (3283 patients) were included. Overall study quality was good, except for risk of non-performance of repeated reverse transcriptase polymerase chain reaction (RT-PCR) after negative initial RT-PCR and persistent clinical suspicion in four studies. Pooled COVID-19 frequencies in CO-RADS categories were: 1, 8.8%; 2, 11.1%; 3, 24.6%; 4, 61.9%; and 5, 89.6%. Pooled COVID-19 frequencies in RSNA classification categories were: negative 14.4%; atypical, 5.7%; indeterminate, 44.9%; and typical, 92.5%. Pooled pairs of sensitivity and specificity using CO-RADS thresholds were the following: at least 3, 92.5% (95% CI: 87.1, 95.7) and 69.2% (95%: CI: 60.8, 76.4); at least 4, 85.8% (95% CI: 78.7, 90.9) and 84.6% (95% CI: 79.5, 88.5); and 5, 70.4% (95% CI: 60.2, 78.9) and 93.1% (95% CI: 87.7, 96.2). Pooled pairs of sensitivity and specificity using RSNA classification thresholds for indeterminate were 90.2% (95% CI: 87.5, 92.3) and 75.1% (95% CI: 68.9, 80.4) and for typical were 65.2% (95% CI: 37.0, 85.7) and 94.9% (95% CI: 86.4, 98.2).

Conclusion

COVID-19 infection frequency was higher in patients categorized with higher CORADS and RSNA classification categories.

Summary

The frequency of coronavirus disease 2019 infection was higher in patients with higher CO-RADS and RSNA classification categories, which supports the order of grading used by both systems.

Key Points

• ■ Using the lowest clinically meaningful thresholds of CO-RADS of at least 3 and indeterminate according to the RSNA classification, sensitivity values were 92.5% and 90.2%, which implies that CO-RADS 1 and 2 and RSNA classification categories negative and atypical certainly do not exclude COVID-19.

• ■ Using the highest thresholds of CORADS 5 and typical according to the RSNA classification, specificity values increased up to 93.1% and 94.9% at the cost of sensitivity, with values of 70.4% and 65.2, respectively.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has caused a major global crisis. On December 2, 2020, there were 64 million confirmed cases and almost 1.5 million confirmed deaths due to COVID-19 worldwide (1). Although most countries have already experienced the first surge of rising COVID-19 cases, second surges have started in late 2020. Chest imaging has an important role in the evaluation of patients with COVID-19 (2). The chest imaging findings of COVID-19 were first reported in January 2020 and included bilateral lung involvement and ground-glass opacities in the majority of hospitalized patients (3). Since this first report (3), several studies on the diagnostic value of chest CT in COVID-19 have been published. However, as most initial studies did not use uniform diagnostic criteria (4), their results cannot directly be translated to clinical practice.

Two major chest CT classification scales for standardized CT reporting of COVID-19 have been developed, namely the COVID-19 Reporting and Data System (CO-RADS) (5) and the Radiological Society of North America (RSNA) classification system for reporting COVID-19 pneumonia (6, 7). CO-RADS basically consists of five categories (CO-RADS 1 to 5; Table E1 and Figures E1-5 [supplement]), whereas the RSNA classification system consists of four categories (negative, atypical, indeterminate, and typical; Table E2 and Figures E1-5 [supplement]). CO-RADS and the RSNA chest CT classification system are very similar. CO-RADS categories 1, 2, 3-4, and 5 are essentially equal to categories negative, atypical, indeterminate, and typical of the RSNA classification system, respectively (5, 8). The use of these standardized diagnostic classification systems may reduce observer variation, enhance clinical communication, and improve generalizability. However, the diagnostic yields of both the CO-RADS and RSNA categorizations are not completely clear yet. Original studies on this topic may suffer from small sample sizes and potential methodological quality concerns. Aggregated data are necessary to understand the clinical interpretability of these chest CT classification systems for the diagnosis of COVID-19. Although there have already been meta-analyses published on the diagnostic performance of chest CT in detecting COVID (4, 9), the initial studies included within these meta-analyses suffered from methodological quality issues and did not use uniform diagnostic criteria such as the CO-RADS and RSNA categorizations. These shortcomings limit translation of diagnostic performance values to clinical practice. Therefore, our objective was to determine, in a meta-analysis, the diagnostic performance of the CO-RADS and the RSNA classification system in patients with clinically suspected COVID-19 infection.

Materials and Methods

The study adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guideline (10).

Data Sources

A search in MEDLINE and Embase was conducted to find original publications on the diagnostic performance of the CO-RADS and the RSNA classification systems in evaluating symptomatic with clinically suspected COVID-19 infection. The following search term was used: (CO-RADS OR CORADS OR Radiological Society of North America OR RSNA) AND (Corona OR Coronavirus OR Covid-19 OR SARS-Cov-2 OR 2019nCoV OR Wuhan-virus) AND (Computed tomography OR Computerized tomography OR Computed tomographic OR CT OR CAT OR HRCT). In addition, the journal Radiology: Cardiothoracic Imaging was manually searched for potentially relevant publications. Publications which cited the original CO-RADS (5) and RSNA classification system for reporting COVID-19 pneumonia (6, 7) were also searched using the cited reference function in Web of Science and MEDLINE. The search was updated until August 24, 2020.

Study Selection

Original studies which provided data on the diagnostic performance of the CO-RADS or RSNA classification system in evaluating patients with clinically suspected COVID-19 infection, and in which reverse transcription polymerase chain reaction (RT-PCR) was the reference standard, were eligible for inclusion. Reviews, abstracts, and studies were excluded for the following reasons: (a) included fewer than 10 patients, (b) reported insufficient data to compose a 2×2 contingency table to calculate sensitivity and specificity on per-patient level for any CO-RADS or RSNA classification system threshold, and (c) only provided data on the performance of artificial intelligence-based analyses. When overlapping data were presented in more than one study, the study with the largest number of patients was selected. Titles and abstracts of retrieved studies were reviewed using aforementioned selection criteria. The full-text version of each potentially eligible study was then reviewed to definitively determine if the study fulfilled the selection criteria.

Study Data Extraction

For each included study, the main characteristics (country of origin, patient inclusion period, number of patients, age, and sex of patients, clinical characteristics of included patients, CT protocol, CT interpreters, reference standard, and COVID-19 frequency) were extracted by two independent reviewers (R.M.K., radiologist, and H.J.A., thirdyear resident in radiology). If data from multiple readers were reported, only data from the first reader were extracted and used for the analyses. The number of patients with and without COVID-19 according to the different CO-RADS and the RSNA classification categories was also extracted. Data on interobserver or intraobserver agreement using the CO-RADS and the RSNA classification system were also extracted. Any discrepancies were solved by consensus with a third reviewer (T.C.K., radiologist).

Study Quality Assessment

The quality of included studies was assessed by two independent reviewers (R.M.K. and H.J.A.) using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool, which comprises four key items: patient selection, index test, reference standard, and flow and timing (11). Any discrepancies were solved by consensus with a third reviewer (T.C.K.).

Statistical Analyses

Frequency of COVID-19 in each of the categories of the CO-RADS and the RSNA classification system were calculated for each individual study and pooled with a random effects model. Sensitivity and specificity of the CO-RADS and RSNA classification systems at specific diagnostic thresholds in detecting COVID-19 (ie, CO-RADS thresholds of at least 3, at least 4, 5, and RSNA classification thresholds indeterminate and typical) were pooled using a bivariate random-effects model (12). The numbers were pooled in each CORADS and in each RSNA classification category separately. The same random effects model was used per each study, across different categories. Cochran's Q and Chi-squared tests were performed to test for heterogeneity between studies, which was defined as P < .10. Statistical analyses were performed using the Open Meta-Analyst software package (13) and Meta-analysis of Diagnostic Accuracy Studies package in R software (14, 15).

Results

Literature Search

Figure 1 displays the study selection process. A total of 182 studies were eligible for inclusion after searching databases. After screening titles and abstracts, 168 studies were excluded, leaving 14 studies that were potentially eligible for inclusion. After reading the full text of the 14 studies, three studies (16-18) were excluded because the diagnostic performance of either CO-RADS or the RSNA classification system was not investigated, one study (5) was excluded because no data on a per-patient level were reported, and another study (19) was excluded because there were overlapping data with another study (8) which comprised a larger number of patients. Nine studies were eventually included (8, 20-27).

Figure 1:

Flow diagram of study selection. The asterisk indicates that there were duplicate studies.

The main study characteristics are shown in Table 1 and Table E3 (supplement). All assessed studies were performed between January and June 2020. The median number of patients per study was 312 (range, 71-859), and the total number of patients of all studies combined was 3283. All nine studies included patients with a clinical suspicion of COVID-19. The mean frequency of COVID-19 was 48.7% (range, 41.7–59.8%). Of all patients included in the nine studies, 1979 patients were evaluated with CO-RADS and 1400 patients were evaluated with the RSNA classification system.

Table 1:

Main Characteristics of the Included Studies

Study quality

Figure 2 provides a summary of the QUADAS-2 quality assessments. In one study (20), it was unclear whether patients were enrolled consecutively or randomly. There was no risk of bias with regard to patient selection in the other studies or with regard to index test. Risk of bias with respect to reference test was rated high in three studies (23, 26, 27) because repeated RT-PCR testing was not used in all patients with a negative initial RT-PCR result and persistent clinical suspicion of COVID-19. Risk of bias with respect to reference test was rated unclear in one study (21), because it was not clear whether all patients with an initial negative RT-PCR result and a persistent clinical suspicion of COVID-19 underwent repeated RT-PCR testing. In one study (20), there was potential risk of bias with regard to flow and timing, because the time interval between CT and RT-PCR testing was not reported. There was no risk of bias with regard to flow and timing in the other studies, because the maximum time interval between chest CT and RT-PCR did not exceed seven days (22). There were no applicability concerns.

Figure 2:

Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) quality assessments of included studies.

Diagnostic Performance of CO-RADS

The frequency of COVID-19 in each of the categories of CO-RADS is displayed in Table 2. With higher CO-RADS classification, the frequency of COVID-19 increased. Pooled frequency of COVID-19 in CO-RADS categories 1, 2, 3, 4, and 5 were 8.8%, 11.1%, 24.6%, 61.9%, and 89.6%. Pooled sensitivity and specificity of the CO-RADS and the RSNA classification system at specific thresholds are displayed in Table 3. Pooled pairs of sensitivity and specificity using CO-RADS thresholds were the following: at least 3, 92.5% (95% CI: 87.1, 95.7) and 69.2% (95% CI: 60.8, 76.4); at least 4, 85.8% (95% CI: 78.7, 90.9) and 84.6% (95% CI: 79.5, 88.5); and 5, 70.4% (95% CI: 60.2, 78.9) and 93.1% (95% CI: 87.7, 96.2).

Table 2:

Frequency of COVID-19 in each of the Categories of CO-RADS

Table 3:

Pooled Sensitivity and Specificity at Specific Thresholds According to CO-RADS

Diagnostic Performance of the RSNA Classification System

The frequency of COVID-19 in each of the categories of the RSNA classification systems is displayed in Table 4. With higher RSNA classification, the frequency of COVID-19 increased. Pooled frequencies of COVID-19 in RSNA classification categories negative, atypical, indeterminate, and typical were 14.4%, 5.7%, 44.9%, and 92.5%. Pooled sensitivity and specificity of the RSNA classification system at specific thresholds are displayed in Table 5. Pooled pairs of sensitivity and specificity using RSNA classification thresholds were the following: indeterminate, 90.2% (95% CI: 87.5, 92.3) and 75.1% (95% CI: 68.9, 80.4) and typical, 65.2% (95% CI: 37.0, 85.7) and 94.9% (95% CI: 86.4, 98.2).

Table 4:

Frequency of COVID-19 in Each of the Categories of the RSNA Classification System

Table 5:

Pooled Sensitivity and Specificity at Specific Thresholds According to the RSNA Classification System.

Interobserver and intraobserver agreement

For the CO-RADS, substantial to almost perfect interobserver agreement has been reported, with ĸ values of 0.648 to 0.773 (8) and intraclass correlation coefficients of 0.800 to 0.874 (20). For the RSNA classification system, moderate to substantial interobserver agreement has been reported, with ĸ values of 0.500 (23) and of 0.570 to 0.663 (8). None of the included studies reported data on intraobserver agreement.

Discussion

This meta-analysis provides pooled data with regard to the frequency of patients with COVID-19 for each category of CO-RADS and the RSNA classification system in patients with clinically suspected with having a COVID-19 infection. With higher CO-RADS and RSNA classification category, the frequency of patients with COVID-19 increased. This supports the order of grading that is used by both systems. In CO-RADS 5, the prevalence of COVID-19 was 89.6%. In the RSNA category typical, the frequency of COVID-19 was 92.5%. We also provided sensitivity and specificity values for specific diagnostic thresholds. Using the lowest clinically meaningful thresholds of CO-RADS of at least 3 and indeterminate according to the RSNA classification, sensitivity values were 92.5% (95% CI: 87.1, 95.7%) and 90.2% (95% CI: 87.5, 92.3%), respectively. These findings imply that CO-RADS 1 and 2 and RSNA classification categories negative and atypical do not exclude COVID-19. Furthermore, when using these low diagnostic thresholds, specificity is only moderate with values of 69.2% (95% CI: 60.8, 76.4) for CO-RADS of at least 3 and 75.1% (95% CI: 68.9, 80.4%) for RSNA indeterminate. If higher diagnostic thresholds are applied, specificity naturally increases at the cost of sensitivity. Using CO-RADS of at least 5 and the RSNA classification typical as diagnostic thresholds, specificity values increased up to 93.1% (95% CI: 87.7, 96.2) and 94.9% (95% CI: 86.4, 98.2). However, when using these high diagnostic thresholds, sensitivity is only moderate with values of 70.4% (95% CI: 60.2, 78.9) and 65.2% (95% CI: 37.0, 85.7).

Methodological quality of the studies included in the current meta-analysis generally appears to have higher quality that studies included within prior meta-analyses (4, 9). In two prior meta-analyses, high risk of bias was present in all six included studies (100%) (4) and in ten of thirteen included studies (77%) (9). In our current meta-analysis, the "reference standard" was the only QUADAS-2 item which was deemed to be of high risk of bias. This item applied to three of the nine included studies (33%) because repeated RT-PCR testing was not used in all patients with a negative initial RT-PCR result and persistent clinical suspicion of COVID-19 (23, 26, 27).

Importantly, we provide a meta-analysis that specifically focused on the diagnostic performance of chest CT in COVID-19 by selecting studies that used standardized diagnostic criteria. Therefore, our study results are more generalizable and useful to clinical practice compared to other prior meta-analyses on CT for COVID-19 assessment. Our finding that CO-RADS 1 and 2 and RSNA classification categories negative and atypical do not exclude COVID-19 are in line with the results of a meta-analysis in nearly 3500 patients, which reported an estimated frequency of 10.6% for normal chest CT findings in symptomatic patients with COVID-19 (28). In a prior meta-analysis of six studies which did not use uniform diagnostic criteria, pooled sensitivity and specificity were 94.6% (95% CI: 91.9, 96.4) and 46.0% (95% CI: 31.9, 60.7), respectively (4). Using CO-RADS of at least 3 and RSNA classification indeterminate as diagnostic thresholds, similar sensitivity values of 92.5% (95% CI: 87.1, 95.7) and 90.2% (95% CI: 87.5, 92.3) can be achieved, while relatively higher specificity values of 69.2% (95% CI: 60.8, 76.4) and 75.1% (95% CI: 68.9, 80.4) are obtained. Thus, when using CO-RADS or the RSNA classification system instead of non-standardized criteria, it appears that specificity may be improved without sacrificing sensitivity.

If a low threshold is being used (eg, any lung abnormality on chest CT is considered positive for COVID-19), virtually all COVID-19 cases with lung abnormalities will be correctly classified, but all non-COVID-19 cases with any lung abnormality at chest CT will be incorrectly classified as having COVID-19 (29). By applying standardized diagnostic criteria such as CO-RADS or the RSNA classification system, a higher proportion of non-COVID-19 cases with lung abnormalities due to other lung diseases will be correctly classified as not having COVID-19 but an alternative lung disease. It should be noted that the studies in our meta-analysis included patients between January and June 2020, a period with a high COVID-19 frequency (mean of 48.7%; range, 41.7–59.8%). Specificity is likely to decrease with lower COVID-19 frequency and increasing frequency of other viral lung infections such as influenza (30).

Our study has some limitations. First, the included studies used RT-PCR, which is an imperfect reference standard with a reported sensitivity of 89% (95% CI: 81, 94) (31). Sensitivity of RT-PCR appears to be lower in elderly patients (31), which may be due to sampling error in these patients who are more likely to have poorer performance status (26). Furthermore, vendor-specific effects and differences in the quality assurance process may affect the performance of RT-PCR (31). However, RT-PCR is still the recommended method to confirm current COVID-19 infection (32-34). Second, because of the relatively low number of included studies, we did not perform subgroup or meta-regression analyses to explain statistical heterogeneity between studies. Geographical differences, non-reported prevalence of other lung diseases, interobserver variability in chest CT assessment, RT-PCR performance, and some methodological quality issues may have been potential sources of heterogeneity. Note that interobserver agreement varies from substantial to almost perfect for the CO-RADS (8, 20) and from moderate to substantial for the RSNA classification system (8, 23).

In conclusion, COVID-19 infection frequency was higher in patients categorized with higher CO-RADS and RSNA classification categories. Our data may be useful for deciding on the probability of COVID-19 based on chest CT (along with clinical information and RT-PCR).