The challenge of COVID-19 low disease prevalence for artificial intelligence models: report of 1,610 patients

Introduction

It has been shown that artificial intelligence (AI) may yield rapid and improved detection of coronavirus disease 2019 (COVID-19) by integrating chest CT findings with clinical symptoms, exposure history and laboratory testing (1).

Early detection and reduction of workload for healthcare workers have been proposed as main applications of AI in COVID-19 pandemic (2), despite limitations, constraints and pitfalls (3,4).

Methods

In Italy, the first western country to be hit by the COVID-19 pandemic, our institution implemented a multi-site collaborative effort aimed to centralize the reading of chest CT scans using an AI solution (InferRead™ CT Lung, Infervision, DE, USA) as a rapid diagnostic tool to: (I) label patients with suspected COVID-19 infection, (II) alert the radiologists for images review and, eventually, (III) prioritize the final report of suspected cases and RT-PCR tests for the definite diagnosis. Institutional Review Board (IRB) approval was obtained for this study.

The core algorithm applied in this study was based on a deep convolutional neural network structure. The algorithm includes automatic segmentation of COVID-19 lung lesions and its quantitative output also provides a risk probability for the diagnosis of COVID-19 pneumonia.

During the whole timeframe, we a priori defined a risk of 30% as cut-off value to stratify the AI probability of COVID-19 into “Low Risk” (<30%) and “High Risk” (≥30%). Indeed, a preliminary study conducted in our hospital on the first 100 patients showed that, among the suspected cases identified by AI, patients with COVID-19 probability <30% received a confirmed diagnosis by RT-PCR with a low percentage of 4.8%.

Results

Between February 17^th and May 24^th 2020, the AI dedicated server received and consecutively assessed 1,610 chest CT scans. Of these, 37.8% (608/1,610) were labeled as suspected COVID-19 cases by AI, reaching the zenith of new cases in the week of March 23^rd–29^th. Noteworthy, despite inter-regional differences, the prevalence of new cases in the AI database mirrored what was observed at national level as registered by the Istituto Superiore di Sanità (the zenith of daily new cases of COVID-19 in Italy was on March 23^rd) (5) (Figure 1A). During the following weeks, despite a decreased prevalence of COVID-19 in Italy (down to 11% of the number of cases at the zenith on May 24^th), the number of suspected cases identified by AI remained relatively high (41% of the number of cases at the zenith in the population assessed by AI).

Figure 1.

Weekly new suspected cases identified by AI between February 17^th and May 24^th in Italy. (A) Comparison of weekly new suspected cases identified by AI (red bars) with national data (blue bars). Data are normalized to the number at the zenith for both distributions (number of cases at the zenith is 38,551 for national data and 101 for AI server data); (B) risk stratification of the number of weekly new suspected cases identified by AI into “High Risk” (red bars) and “Low Risk” (orange bars). AI, artificial intelligence.

The risk stratification of suspected cases into “High Risk” and “Low Risk” showed that, despite the number of patients with “High Risk” scores dropped during the month of May, the number of patients labeled as “Low Risk” was high and, with the decreased prevalence of the disease, represented the majority of suspected cases identified by AI (Figure 1B).

Comment

The AI model applied in this study was based on chest CT data and did not integrate clinical symptoms, exposure history or laboratory testing. Nevertheless, our findings support AI as a tool to label suspected patients with COVID-19 infection on chest CT scans and prioritize management. Also, our results highlight the disease prevalence as the main challenge for AI solutions. As Mei et al. (1) pointed out, the implementation of AI trained with either chest CT datasets only or integrating epidemiological, clinical, laboratory with radiological information, shows high diagnostic performance where and when the prevalence of disease is high. However, it is unlikely to perform just as well where and when the prevalence of disease is low.

Although our arguments require further investigations, the analysis on an Italian testing dataset proves that AI predicting models for COVID-19 can be applied for binary classification in the high prevalence settings but should take into account risk stratification in low prevalence settings, in order to improve the diagnostic performance and decrease the false positive rates in the detection of COVID-19.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.