Abstract
Background
Despite its high sensitivity in diagnosing COVID-19 in a screening population, chest CT appearances of COVID 19 pneumonia are thought to be non-specific.
Purpose
To assess the performance of United States (U.S.) and Chinese radiologists in differentiating COVID-19 from viral pneumonia on chest CT.
Methods
A total of 219 patients with both positive COVID-19 by RT-PCR and abnormal chest CT findings were retrospectively identified from 7 Chinese hospitals in Hunan Providence, China from January 6 to February 20, 2020. A total of 205 patients with positive Respiratory Pathogen Panel for viral pneumonia and CT findings consistent with or highly suspicious for pneumonia by original radiology interpretation within 7 days of each other were identified from Rhode Island Hospital in Providence, RI. Three Chinese radiologists blindly reviewed all chest CTs (n=424) to differentiate COVID-19 from viral pneumonia. A sample of 58 age-matched cases was randomly selected and evaluated by 4 U.S. radiologists in a similar fashion. Different CT features were recorded and compared between the two groups.
Results
For all chest CTs, three Chinese radiologists correctly differentiated COVID-19 from non-COVID-19 pneumonia 83% (350/424), 80% (338/424), and 60% (255/424) of the time, respectively. The seven radiologists had sensitivities of 80%, 67%, 97%, 93%, 83%, 73% and 70% and specificities of 100%, 93%, 7%, 100%, 93%, 93%, 100%. Compared to non-COVID-19 pneumonia, COVID-19 pneumonia was more likely to have a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001), fine reticular opacity (56% vs. 22%, p<0.001), and vascular thickening (59% vs. 22%, p<0.001), but less likely to have a central+peripheral distribution (14.% vs. 35%, p<0.001), pleural effusion (4.1 vs. 39%, p<0.001) and lymphadenopathy (2.7% vs. 10.2%, p<0.001).
Conclusion
Radiologists in China and the United States distinguished COVID-19 from viral pneumonia on chest CT with high specificity but moderate sensitivity.
Summary
Radiologists had high specificity but moderate sensitivity in differentiating COVID-19 from viral pneumonia on chest CT.
Key Results
■ Three Chinese radiologists had sensitivities of 72%, 72% and 94% and specificities of 94%, 88% and 24% in differentiating 219 COVID-19 from 205 non-COVID-19 pneumonia.
■ Four United States radiologists had sensitivities of 93%, 83%, 73% and 73% and specificities of 100%, 93%, 93% and 100%.
■ The most discriminating features for COVID-19 pneumonia included a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001) and vascular thickening (58% vs. 22%, p<0.001).
Introduction
Since the initial outbreak of Coronavirus disease-19 (COVID-19) from Wuhan, China in late December 2019 (1), there have been 87,137 confirmed cases and 2,873 reported deaths distributed across 60 countries as of March 1st 2020 (2, 3). China has had the majority of COVID 19 cases (92%) (3). Patients infected with COVID-19 typically present with fever, cough, dyspnea, and muscle aches while imaging frequently reveals bilateral pneumonia (5).
The standard diagnostic method being used is real-time polymerase chain reaction (RT-PCR) to detect viral nucleotides from specimens obtained by oropharyngeal swab, nasopharyngeal swab, bronchoalveolar lavage, or tracheal aspirate (6). However, recent reports have revealed that RT-PCR has a sensitivity as low as 60-71% for detecting COVID-19 (5, 7, 8), which can possibly be attributed to low viral load present in test specimens or laboratory error (7, 9). These false negatives hinder quarantine efforts, necessitate repeat testing and has the potential to overload the current supply of testing kits and related infrastructure (8). By contrast, chest CT has demonstrated about 56-98% sensitivity in detecting COVID-19 at initial presentation and can be helpful in rectifying false negatives obtained from RT-PCR during early stages of disease development (7, 8). Chest CT of COVID-19 patients reveals areas of consolidation and ground-glass opacity (GGO) with bilateral peripheral involvement in multiple lobes progressing to “crazy-paving” patterns and consolidation. CT signs gradually improve beginning approximately 14 days post-symptom onset (10, 11, 12).
Despite its high sensitivity in diagnosing COVID-19 in a screening population, chest CT had low specificity (25%) in a recent report of 1014 patients with COVID-19 (5). Prior studies have not directly compared chest CT patterns of COVID-19 from viral pneumonia on chest CT. The purpose of this study was to assess the performance of United States (U.S.) and Chinese radiologists in differentiating COVID-19 from viral pneumonia on chest CT.
Materials and Methods
Patient cohort
The institutional review board of all seven hospitals in Hunan Providence, China and Rhode Island Hospital from the U.S. approved this retrospective study and written informed consent was waived. A total of 256 patients with both positive COVID-19 by RT-PCR and chest CT imaging within two weeks were retrospectively identified from 7 Chinese hospitals in Hunan Providence, China from January 6 to February 20, 2020. The RT-PCR results were extracted from the patients’ electronic medical records in the hospital information system (HIS). The RT-PCR assays were performed by using TaqMan One-Step RT-PCR Kits from Shanghai Huirui Biotechnology Co., Ltd or Shanghai BioGerm Medical Biotechnology Co., Ltd, both of which have approved use by the China Food and Drug Administration (CFDA). For patients with multiple RT-PCR assays, positive result on the last performed test was adopted as confirmation of diagnosis. The number of cases included from each hospital is shown in Supplementary Table 1. Among these patients, 37 with negative chest CT were excluded, resulting in a final cohort of 219 patients. The chest CT protocols from the 7 hospitals are shown in Supplementary Table 2.
The radiology search engine MONTAGE (Nuance Communications, Burlington, MA) at Rhode Island Hospital in Providence, RI was used to identify cases that contain the word “pneumonia” in the impression section of the radiology CT reports from 2017 to 2019. The identified CT scans were directly downloaded from the hospital Picture Archiving and Communications Systems (PACS) and non-chest CTs were excluded. Positive results from Respiratory Pathogen Panel (RPP) were used to locate patients of possible viral pneumonia from 2017 to 2019. The tests of ePlex Respiratory Pathogen panel (GenMark Diagnostics, Carlsbad, CA) were performed in the Microbiology Lab of Rhode Island Hospital Pathology Department according to it manufacture protocol (13). A diagram illustrating the initial breakdown of RPP search results is shown in Supplementary Figure 1. The two lists were cross-matched to generate a final list that contained CT chest scans with the word “pneumonia” in the final impression and positive RPP test within 7 days of each other. Then, the impression sections of these CT reports were reviewed by a research assistant (BH) and a radiologist (HXB) board-certified in general diagnostic radiology and interventional radiology with one year of practice experience to identify 205 cases with final CT impression being “consistent with” or “highly suspicious for” pneumonia. Our final cohort consisted of 424 patients. A diagram illustrating patient inclusion and exclusion is shown in Figure 1. A diagram illustrating the final breakdown of RPP results is shown in Figure 2.
Figure 1:
Illustration of study flow RIH: Rhode Island Hospital
Figure 2:
Distribution of viral pathogens based on RPP test in the final cohort. It is an FDA approved assay that simultaneous detects 19 viruses (influenza A virus; influenza A H1 virus; influenza A 2009 H1 virus; influenza A H3 virus; influenza B virus; adenovirus; coronaviruses [HKU1, OC43, NL63, and 229E]; human rhinovirus/enterovirus; human metapneumovirus; parainfluenza viruses 1, 2, 3, and 4; and respiratory syncytial virus [RSV] [RSV subtype A and RSV subtype B]) and 2 bacteria (Mycoplasma pneumoniae and Chlamydia pneumoniae). The ePlex panel (16) has been proven to be a highly sensitive and specific multiplex assay for respiratory pathogen detection. In a multicentric study, the positive percent agreement values (equivalent to sensitivity when a perfect reference method is unavailable) ranged from 85.1% to 95.1% and the negative percent agreement values (equivalent to specificity) ranged from 99.5% to 99.8% when compared to another well-established RP panel from BioFire (Salt Lack City, UT) (17).
Radiologist interpretation
Three Chinese radiologists, who were blinded to RT-PCR results, reviewed all chest CT images and scored each case as COVID-19, pneumonia of other etiology or neither. Fifty-eight cases age-matched between the COVID-19 and non-COVID-19 pneumonia groups were randomly selected from the entire cohort and evaluated in the same way by four U.S. radiologists. All identifying information was removed from the CT studies, which were shuffled and uploaded to 3D slicer for interpretation. Information on the radiologists regarding location of practice, years in practice, cardiothoracic imaging fellowship and COVID-10 specific training experience is shown in Table 1.
Table 1:
Backgrounds of radiologists who reviewed the cases
CT features
Different CT features on the entire cohort were recorded by two Chinese radiologists in consensus. If consensus could not be reached, it was resolved by a senior radiologist (X.Z.) with more than 10 years of chest CT experience.
Statistical analysis
Continuous variables were expressed as medians and ranges, while categorical variables as counts and percentages. Metrics such as sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated to evaluate the diagnostic performance of the radiologists. For the calculations, COVID-19 was considered positive result, while pneumonia of other etiology and neither were considered negative result. Exact binomial 95% confidence intervals were calculated for sensitivity, specificity, PPV, NPV, and accuracy using the epiR package in the R statistical computing language (R version 3.4.2, The R Foundation for Statistical Computing, Vienna, Austria; https://www.r-project.org). P values of <0.05 were considered statistically significant.
Results
Our final cohort consisted of 424 patients, including 205 patients with non-COVID-19 pneumonia from the U.S. and 219 patients with COVID-19 from China. The average number of days between CT and positive RT-PCR test result was 4.1±4.4 days for the COVID-19 group; the average number of days between CT and positive RPP test result was 1.0±1.7 days. Patients with COVID-19 were younger (mean age of 45 vs. 65 years, p<0.001) and less likely to have elevated white blood cell count (29% vs. 55%, p<0.001), but more likely to have reduced lymphocyte count (16% vs. 44%, p<0.001). Clinical characteristics including comorbidities comparing the two groups are shown in Table 2.
Table 2:
Clinical characteristics of the US non-COVID-19 and Chinese COVID-19 patient cohorts
For the entire cohort, the accuracy of the three Chinese radiologists to identify COVID-19 from non-COVID-19 pneumonia was 83% (95 CI: 79-86%), 80% (95% CI: 76-83%), and 60% (95% CI: 55-65%), respectively (Table 3). Sensitivity ranged from 72 to 94%, and specificity showed great variation (24 to 94%) (Table 3).
Table 3:
Performance results of the three Chinese radiologists who reviewed all 424 cases
In a randomly selected, age-matched cohort of 58 patients, the accuracy of four U.S. radiologists to differentiate COVID-19 from non-COVID-19 pneumonia was 97% (95% CI: 88-100%), 88% (95 CI: 77-95%), 83% (95% CI: 71-91%), and 84% (95% CI: 73-93%), respectively (Table 4). Sensitivity ranged from 70 to 93%, and specificity ranged from 93 to 100% (Table 4).
Table 4:
The results of 3 Chinese radiologists and 4 U.S. radiologists who reviewed 58 age-matched cases.
Compared to non-COVID-19 pneumonia, COVID-19 pneumonia was more likely to have a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001), fine reticular opacity (56% vs. 22%, p<0.001), vascular thickening (59% vs. 22%, p<0.001) and reverse halo sign (11% vs. 1%, p=0.005), but less likely to have a central+peripheral distribution (14% vs. 35%, p<0.001), air bronchogram (14% vs. 23%, p=0.014), pleural thickening (15 vs. 33%, p<0.001), pleural effusion (4 vs. 39%, p<0.001) and lymphadenopathy (2.7% vs. 10.2%, p<0.001) (Table 5).
Table 5:
CT features comparing COVID-19 and viral pneumonia
Figure 3 demonstrates example cases in which the majority of the radiologists mistook non-COVID-19 pneumonia for COVID-19 infection (A-B) or vice versa (C-H).
Figure 3a:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Discussion
COVID-19 is spreading rapidly worldwide while present diagnostic methods for identifying the virus have limitations. RT-PCR testing has low sensitivity early on in the disease course. Although chest CT has high sensitivity, it has low specificity (7, 11). This low specificity may stem from the fact that it is difficult to distinguish COVID-19 findings from findings of other disease on chest CT (5). To optimize patient management, medical care, and disease control it is important to determine the efficacy of chest CT in distinguishing COVID-19 from pneumonia of other etiologies by radiologists. The study conducted herein revealed that radiologists are capable of distinguishing COVID-19 from viral pneumonia on chest CT with high specificity but moderate sensitivity.
Healthcare providers being able to reliably differentiate COVID-19 from other causes of pneumonia on chest CT would benefit diagnostic workup of the disease by compensating for the poor sensitivity of RT-PCR, particularly during early disease stages. Although chest CT has demonstrated high sensitivity relative to RT-PCR testing for COVID-19 diagnosis, it may not reveal distinct patterns for COVID-19 in all cases. This can make it hard to distinguish COVID-19 from other causes of viral pneumonia. For example, influenza and COVID-19 both demonstrate GGO and consolidation on chest CT (14). Introducing the possibility of pathologies with similar chest CT findings to those of COVID-19 ultimately complicates the differential diagnosis. Our study is significant because it demonstrates that radiologists are capable of distinguishing COVID-19 from other etiology of pneumonia on chest CT with high specificity. This suggests that there is potential that if the differential diagnosis is between COVID-19 and non-COVID-19 pneumonia, a negative diagnosis of COVID-19 by radiologists on chest CT may be good enough to exclude patients from having the disease with fairly good certainty. Our analysis of specific cases where radiologists were wrong reveals that the mistakes were made when the COVID-19 chest CT findings are either subtle (likely reflecting early time in the disease process) (Figures 3C and 3D) or when COVID-19 has atypical chest CT findings (Figures 3E, 3F, 3G, and 3H). It is worth noting that non-COVID-19 pneumonia can also have typical appearance of COVID-19 (Figures 3A and 3B). This poses a dilemma because mandated quarantine for all suspected cases can put significant strain on medical infrastructure, healthcare providers, and the lives of patients, but may need to be followed as a necessary precaution due to variation in presentation with timing of disease and atypical findings on chest CT. Future direction includes development of an artificial intelligence classifier that can further augment radiologist performance in combination with clinical information.
Figure 3c:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Figure 3d:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Figure 3e:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Figure 3f:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Figure 3g:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Figure 3h:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Figure 3b:
A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)
Our study has several limitations. First, the cohort size was small, especially when it comes to cases reviewed by U.S. radiologists. There is selection bias associated with our screening strategy as well. It remains unclear if diagnostic outcomes would improve in a more well-balanced and larger-scale prospective study of similar design. It is also noteworthy that the U.S. radiologists in this study had minimal training specific to diagnosing COVID-19 and that Chinese radiologists practiced in an area with relatively low prevalence of the disease. It is possible that Chinese radiologists working near the epicenter of the disease with a higher degree of experience specific to COVID-19 would have performed significantly better than either group in the present study. In addition, although RPP test and chest CT findings within 7 days of symptom presentation were used to enrich our “pneumonia of other etiology” cohort with viral pneumonia cases, the cause and effect relationship is not 100%. Thus, it is possible that some of the selected patients had mixed viral and bacterial pneumonia or other diseases entirely. Finally, the radiologists were not given clinical information during the evaluation, which could have further improved their performance.
As more research is done, information may be gathered by providers to make this differential more facile to navigate. Until that point, it is recommended that individuals with signs of pneumonia on chest CT be quarantined while RT-PCR testing is performed in conjunction with a thorough medical evaluation including travel history and disease contacts in order to make an accurate COVID-19 diagnosis and prevent disease spread. In conclusion, radiologists had high specificity but moderate sensitivity in distinguishing COVID-19 from viral pneumonia on chest CT.
APPENDIX
H.X.B and B.H. contributed equally to this work.
Abbreviations:
- COVID-19
- Coronavirus Disease 2019
- RT-PCR
- Reverse Transcription Polymerase Chain Reaction
- FDA
- U.S. Food and Drug Administration
- RPP
- respiratory pathogen panel
References
- 1.Zhu N, Zhang D, Wang W, Li X, Yang B, Song J. et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med 2020; 382:727-733. Doi: 10.1056/NEJMoa2001017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Centers for Disease Control and Prevention . Coronavirus Disease 2019 (COVID-19) Locations with Confirmed COVID-19 Cases. https://www.cdc.gov/coronavirus/2019-ncov/locations-confirmed-cases.html. Published February 29, 2020. Accessed March 1, 2020.
- 3.World Health Organization . Coronavirus Disease 2019 (COVID-19) Situation Reports – 41. https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports. Published March 1, 2020. Accessed March 1, 2020.
- 4.World Health Organization . Statement on the Second Meeting of the International Health Regulations (2005) Emergency Committee Regarding the Outbreak of Novel Coronavirus (2019-nCoV). 2020. https://www.who.int/news-room/detail/30-01-2020-statement-on-the-second-meeting-of-the-international-health-regulations-(2005)-emergency-committee-regarding-the-outbreak-of-novel-coronavirus-(2019-ncov). Published January 31, 2020. Accessed February 29, 2020.
- 5.Ai T, Yang Z, Hou H, Zhan C, Chen C, Lv W, Tao Q, Sun Z, Xia L. Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology 2020. Doi: 10.1148/radiol.2020200642. Published February 26, 2020. Accessed March 1, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Center for Disease Control and Prevention . Interim Guidelines for Collecting, Handling, and Testing Clinical Specimens from Persons Under Investigation (PUIs) for Coronavirus Disease 2019 (COVID-19). 2020. https://www.cdc.gov/coronavirus/2019-ncov/lab/guidelines-clinical-specimens.html. Published February 14, 2020. Accessed February 28, 2020. [Google Scholar]
- 7.Fang Y, Zhang H, Xie J, Lin M, Ying L, Pang P, Ji W. Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR. Radiology 2020. Doi: 10.1148/radiol.2020200432. Published February 19, 2020. Accessed February 29, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kanne JP, Little BP, Chung JH, Elicker BM, Ketail LH. Essentials for Radiologists on COVID-19: An Update—Radiology Scientific Expert Panel. Radiology 2020. Doi: 10.1148/radiol.2020200527. Published February 27, 2020. Accessed February 27, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Xie X, Zhong Z, Zhao W, Zheng C, Wang F, Liu J. Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing. Radiology 2020. Doi: 10.1148/radiol.2020200343. Published February 12, 2020. Accessed March 1, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bernheim A, Mei X, Huang M, Yang Y, Fayad ZA, Zhang N, Diao K, Zhu X, Li K, Li S, Shan H, Jacobi A, Chung M. Chest CT Findings in Coronavirus Disease-19 (COVID-10): Relationship to Duration of Infection. Radiology 2020. Doi: 10.1148/radiol.2020200463. Published February 20, 2020. Accessed February 25, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Pan F, Ye T, Sun P, Gui S, Liang B, Li L, Zheng D, Wang J, Hesketh RL, Yang L, Zheng C. Time Course of Lung Changes of Chest CT During Recovery From 2019 Novel Coronavirus (COVID-19) Pneumonia. Radiology 2020. Doi: 10.1148/radiol.2020200370. Published February 13, 2020. Accessed February 29, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Chung M, Bernheim A, Mei X, Zhang N, Huang M, Zeng X, Cui J, Xu W, Zang Y, Fayad ZA, Jacobi A, Li K, Li S, Shan H. CT Imaging Features of 2019 Novel Coronavirus (2019-nCoV). Radiology 2020. Doi: 10.1148/radiol.2020200230. Published February 4, 2020. Accessed February 24, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.GenMark Diagnostics, Inc. ePlex Pipeline. GenMarkDx. https://genmarkdx.com/solutions/panels/eplex-panels/eplex-pipeline/. Published 2020. Accessed February 19, 2020.
- 14.Zu ZY, Jiang MD, Xu PP, Chen W, Ni QQ, Lu GM, Zhang LJ. Coronavirus Disease 2019 (COVID-19): A Perspective from China. Radiology 2020. Doi: 10.1148/radiol.2020200490. Published February 21, 2020. Accessed February 28, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Du Qingyang. Diagnostic Criteria. In: COVID-19 Pneumonia Diagnosis and Treatment Plan, trial version 6. Beijing, CN: National Health Commission of the PRC, February; 19, 2020. 4-6. [Google Scholar]
- 16.Babady NE, England MR, Jurcic Smith KL, He T, Wijetunge DS, Tang YW, Chamberland RR, Menegus M, Swierkosz EM, Jerris RC, Greene W. Multicenter Evaluation of the ePlex Respiratory Pathogen Panel for the Detection of Viral and Bacterial Respiratory Tract Pathogens in Nasopharyngeal Swabs. J Clin Microbiol 2018; 56 (2) e01658-17. Doi: 10.1128/JCM.01658-17. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.