Table 2.
Characteristics of the external validation dataset and accuracy of AI-aided triage
| External validation set (n=2120) | Tianyou Hospital dataset*(n=1097) | Xianning Central Hospital dataset†(n=820) | The Second Xiangya Hospital dataset‡(n=203) | ||
|---|---|---|---|---|---|
| Sex | |||||
| Male | 1079 (51%) | 506 (46%) | 463 (56%) | 110 (54%) | |
| Female | 1041 (49%) | 591 (54%) | 357 (44%) | 93 (46%) | |
| Age, years | 43 (31–56) | 48 (36–58) | 34 (27–49) | 45 (34–63) | |
| CT scans, n | 2120 | 1097 | 820 | 203 | |
| CT manufacturers | |||||
| GE Medical System (Chicago, IL, USA) | 1730 (82%) | 978 (90%) | 752 (92%) | 0 | |
| Siemens (Munich, Germany) | 271 (13%) | 0 | 68 (8%) | 203 (100%) | |
| United Imaging Healthcare (Shanghai, China) | 119 (6%) | 119 (11%) | 0 | 0 | |
| CT findings | |||||
| Positive | 802 (38%)§ | 547 (50%) | 180 (22%) | 75 (37%) | |
| Negative | 1318 (62%)¶ | 550 (50%) | 640 (78%) | 128 (63%) | |
| AI-aided triage performance‖ | |||||
| Sensitivity (95% CI) | 0·923 (0·914–0·932) | 0·934 (0·925–0·944) | 0·900 (0·880–0·924) | 0·893 (0·862–0·932) | |
| Specificity (95% CI) | 0·851 (0·842–0·860) | 0·855 (0·840–0·868) | 0·859 (0·846–0·874) | 0·789 (0·752–0·828) | |
| Positive predictive value (95% CI) | 0·790 (0·777–0·803) | 0·865 (0·851–0·878) | 0·643 (0·613–0·673) | 0·713 (0·662–0·764) | |
| Negative predictive value (95% CI) | 0·948 (0·941–0·954) | 0·929 (0·919–0·940) | 0·968 (0·962–0·976) | 0·927 (0·905–0·953) | |
| AUC (95% CI) | 0·953 (0·949–0·959) | 0·966 (0·961–0·971) | 0·931 (0·921–0·945) | 0·908 (0·888–0·929) | |
| RT-PCR testing | |||||
| Positive | 217/910 (24%) | 118/411 (29%) | 87/369 (24%) | 12/130 (9%) | |
| Negative | 693/910 (76%) | 293/411 (71%) | 282/369 (76%) | 118/130 (91%) | |
| AI-aided triage performance** | |||||
| Sensitivity (95% CI) | 0·876 (0·854–0·898) | 0·907 (0·883–0·935) | 0·839 (0·803–0·880) | 0·833 (0·750–1·000) | |
| Specificity (95% CI) | 0·519 (0·501–0·539) | 0·386 (0·358–0·401) | 0·660 (0·633–0·686) | 0·517 (0·473–0·562) | |
| Positive predictive value (95% CI) | 0·363 (0·343–0·384) | 0·373 (0·345–0·401) | 0·432 (0·392–0·469) | 0·149 (0·109–0·189) | |
| Negative predictive value (95% CI) | 0·930 (0·918–0·944) | 0·911 (0·889–0·939) | 0·930 (0·913–0·949) | 0·968 (0·957–1·000) | |
| AUC (95% CI) | 0·774 (0·757–0·791) | 0·725 (0·699–0·751) | 0·837 (0·810–0·866) | 0·679 (0·608–0·781) | |
| RT-PCR positive and CT positive cases | 191/217 (88%) | 109/118 (92%) | 72/87 (83%) | 10/12 (83%) | |
| Sensitivity of AI-aided triage on RT-PCR positive and CT positive cases (95% CI) | 0·974 (0·966–0·987) | 0·972 (0·964–0·989) | 0·972 (0·963–1·000) | 1·000 (1·000–1·000) | |
Data are n (%), median (IQR), or n/N (n%), unless otherwise stated. AUC=area under the receiver operating curve.
Dataset from Wuhan (Hubei province): 2018 population, 8 837 300 (according to National Bureau of Statistics of China), 50 006 confirmed COVID-19 cases (calculated up to March 25, 2020, according to the National Health Commission of the People's Republic of China), and a disease prevalence of 0·566% (ie, number of confirmed cases in the total population).
Dataset from Xianning (Hubei province): 2018 population, 2 485 000 (according to National Bureau of Statistics of China), 836 confirmed COVID-19 cases (calculated up to March 25, 2020, according to the National Health Commission of the People's Republic of China), and and a disease prevalence of 0·034% disease prevalence.
Dataset from Changsha (Hunan province): 2018 population, 7 288 600 (according to 2018 National Bureau of Statistics of China data), 242 confirmed COVID-19 cases (calculated up to March 25, 2020, according to the National Health Commission of the People's Republic of China), and a disease prevalence of 0·003% disease prevalence.
Of 802 positive CT scans, 772 clearly mentioned COVID-19 signs in the radiological impression section and 30 had an ambiguous radiological impression description but described COVID-19 signs in the radiological findings section (11 in the Tianyou Hopsital dataset, nine in the Xianning Central hospital dataset, and ten in the Second Xiangya Hospital dataset).
Of 1318 negative CT studies, 593 had negative CT findings and 725 had positive findings not associated with COVID-19 (324 in the Tianyou Hopsital dataset, 291 in the Xianning Central Hospital dataset, and 110 in the Second Xiangya Hopsital dataset).
Radiological CT findings were used as the reference standard.
RT-PCR was used as the reference standard.