Table 1.
Characteristics of the included studies.
| Author, year | Study design | Sample characteristics | Hospital (country) | Evaluation outcomes |
| Abràmoff et al, 2018 [25] | Observational study (prospective) | 819 patients | 10 primary care clinics (United States) | APa (sensitivity, specificity, imageability rate) |
| Aoki et al, 2020 [26] | Experimental study (cross-over design) | 6 physicians | The University of Tokyo Hospital (Japan) | COb (reading time, mucosal break detection rate) |
| Arbabshirani et al, 2018 [27] | Observational study (prospective) | 347 routine head CTc scans of patients | Geisinger Health System (United States) | AP (AUCd, accuracy, sensitivity, specificity) CO (time to diagnosis) |
| Bailey et al, 2013 [28] | Crossover RCTe | 20,031 patients | Barnes-Jewish Hospital (United States) | POf (ICUg transfer, hospital mortality, hospital LOSh) |
| Barinov et al, 2019 [29] | Experiment (within subjects) | 3 radiologists | NRi | AP (AUC) CO (diagnostic accuracy) |
| Beaudoin et al, 2016 [30] | Observational study (prospective) | 350 patients (515 prescriptions) | Centre hospitalier universitaire de Sherbrooke (Canada) | AP (number of triggered recommendations, precision, recall, accuracy) |
| Bien et al, 2018 [31] | Experimental study (within subjects) | 9 clinical experts | Stanford University Medical Center (United States) | AP (AUC) CO (specificity, sensitivity, accuracy) |
| Brennan et al, 2019 [32] | Nonrandomized trial | 20 physicians | An academic quaternary care institution (United States) | AP (AUC) CO (risk assessment changes, AUC, usability) |
| Chen et al, 2020 [33] | RCT | 437 patients | Renmin Hospital, Wuhan University (China) | CO (blind spot rate) |
| Connell et al, 2019 [34] | Before-after study | 2642 patients | Royal Free Hospital, Barnet General Hospital (United Kingdom) | PO (renal recovery rate, other clinical outcomes, care process) |
| Eshel et al, 2017 [35] | Observational study (prospective) | 6 expert microscopists | Apollo Hospital, Chennai (India); Aga Khan University Hospital (Kenya) | AP (sensitivity, specificity, species identification accuracy, device parasite count) |
| Giannini et al, 2019 [36] | Before-after study | 22,280 patients in the silent period, 32,184 patients in the alert period | 3 urban acute hospitals under University of Pennsylvania Health System (United States) | AP (sensitivity, specificity) PO (mortality, discharge disposition, ICU transfer, time to ICU transfer, clinical process measures) |
| Ginestra et al, 2019 [37] | Survey | 43 nurses and 44 health care providers | A tertiary teaching hospital in Philadelphia (United States) | CO (nurse and provider perceptions) |
| Gómez-Vallejo et al, 2016 [38] | Observational study (retrospective) | 1800 patients (2569 samples) | A Spanish National Health System hospital (Spain) | AP (accuracy) CO (system perceptions) |
| Grunwald et al, 2016 [39] | Observational study (retrospective) | 15 patients, 3 neuroradiologists | A comprehensive stroke center (Germany) | AP (e-ASPECTS performance) |
| Kanagasingam et al, 2018 [40] | Observational study (prospective) | 193 patients, 4 physicians | A primary care practice in Midland (Australia) | AP (sensitivity, specificity, PPVj, NPVk) |
| Keel et al, 2018 [41] | Survey | 96 patients | St Vincent’s Hospital, University Hospital Geelong (Australia) | AP (sensitivity and specificity, assessment time) PO (patient acceptability) |
| Kiani et al, 2020 [42] | Experimental study (within subjects) | 11 pathologists | Stanford University Medical Center (United Kingdom) | AP (accuracy) CO (accuracy) |
| Lagani et al, 2015 [43] | Observational study (prospective) | 2 health care providers | Chorleywood Health Centre (United Kingdom) | AP (system performance) CO (usability) |
| Lin et al, 2019 [44] | RCT | 350 patients | 5 ophthalmic clinics (China) | AP (accuracy, PPV, NPV) CO (time to diagnosis) PO (patient satisfaction) |
| Lindsey et al, 2018 [45] | Experimental study (within subjects) | 40 practicing emergency clinicians | Hospital for Special Surgery (United States) | AP (AUC) CO (sensitivity, specificity, misinterpretation rate) |
| Liu et al, 2020 [46] | RCT | 1026 patients | No. 988 Hospital of Joint Logistic Support Force of PLA (China) | CO (ADRl, PDRm, number of detected adenomas and polyps) |
| Mango et al, 2020 [47] | Experimental study (within subjects) | 15 physicians | 13 different medical centers (United States) | AP (AUC, sensitivity, specificity) CO (AUC, interreliability, intrareliability) |
| Martin et al, 2012 [48] | Observational study (prospective) | 214 patients | 13 different medical centers (United States) | AP (sensitivity, PPV) PO (ACSCn, care-supported activities) |
| McCoy and Das, 2017 [49] | Before-after study | 1328 patients | Cape Regional Medical Center (United States) | PO (hospital mortality, hospital LOS, readmission rate) |
| McNamara et al, 2019 [50] | Observational study (prospective) | 3 breast cancer experts | John Theurer Cancer Center (United States) | CO (decision making) |
| Mori et al, 2018 [51] | Observational study (prospective) | 791 patients, 23 endoscopists | Showa University Northern Yokohama Hospital (Japan) | AP (NPV) CO (time to diagnosis) |
| Nagaratnam et al, 2020 [52] | Observational study (retrospective) | 1 patient | Royal Berkshire Hospital (United Kingdom) | PO (patient care and clinical outcomes) |
| Natarajan et al, 2019 [53] | Observational study (prospective) | 213 patients | Dispensaries under Municipal Corporation of Greater Mumbai (India) | AP (sensitivity, specificity) |
| Nicolae et al, 2020 [54] | RCT | 41 patients | Sunnybrook Odette Cancer Centre (Canada) | AP (day 30 dosimetry) CO (planning time) |
| Park et al, 2019 [55] | Experimental study (within subjects) | 8 clinicians | Stanford University Medical Center (United States) | CO (specificity, sensitivity, accuracy interrater agreement, time to diagnosis) |
| Romero-Brufau et al, 2020 [56] | Pre-post survey | 81 clinical staff | 3 primary-care clinics in Southwest Wisconsin (United States) | CO (attitudes about AIo in the workplace) |
| Rostill et al, 2018 [57] | RCT | 204 patients, 204 caregivers | NHS, Surrey and Hampshire (United Kingdom) | CO (system evaluations) PO (early clinical interventions, patient evaluations) |
| Segal et al, 2014 [58] | Observational study (prospective) | 16 pediatric neurologists | Boston Children’s Hospital (United States) | CO (diagnostic errors, diagnosis relevance, number of workup items) |
| Segal et al, 2016 [59] | Observational study (prospective) | 26 clinicians | Boston Children’s Hospital (United States) | CO (diagnostic errors) |
| Segal et al, 2017 [60] | Structured interviews | 10 medical specialists | Geisinger Health System and Intermountain Healthcare (United States) | CO (system perceptions) |
| Segal et al, 2019 [61] | Observational study (prospective) | 3160 patients (315 prescription alerts) | Sheba Medical Center (Israel) | AP (accuracy, clinical validity, and usefulness) PO (changes in medical orders) |
| Shimabukuro et al, 2017 [62] | RCT | 142 patients | University of California San Francisco Medical Center (United States) | PO (LOS, in-hospital mortality) |
| Sim et al, 2020 [63] | Observational study (prospective) | 12 radiologists | 4 medical centers (United States and South Korea) | AP (sensitivity, FPPIp) CO (sensitivity, FPPI, decision change) |
| Steiner et al, 2018 [64] | Experimental study (within subjects) | 6 anatomic pathologists | NR | CO (sensitivity, average review per image, interpretation difficulty) |
| Su et al, 2020 [65] | RCT | 623 patients, 6 endoscopists | Qilu Hospital of Shandong University (China) | CO (ADR, PDR, number of adenomas and polyps, withdrawal time, adequate bowel preparation rate) |
| Titano et al, 2018 [66] | RCT | 2 radiologists | NR | CO (time to diagnosis, queue of urgent cases) |
| Vandenberghe et al, 2017 [67] | Observational study (prospective) | 1 pathologist and 2 HER2 raters | NR | CO (decision concordance, decision modification) |
| Voerman et al, 2019 [68] | Before-after study | NR | Five Rivers Medical Center, Pocahontas (United States) | CEq (average total costs per patient) PO (numbers of patients with clostridium difficile and antibiotic-resistant infections, LOS, antibiotic use) |
| Wang et al, 2019 [69] | RCT | 1058 patients, 8 physicians | Sichuan Provincial People’s Hospital (China) | CO (ADR, PDR, number of adenomas per patient) |
| Wang et al, 2019 [70] | RCT | 75 patients | 4 primary care clinics affiliated with Brigham and Women’s Hospital (United States) | CO (anticoagulation prescriptions) |
| Wang et al, 2020 [71] | RCT | 962 patients | Caotang branch hospital of Sichuan Provincial People’s Hospital (China) | CO (ADR, PDR, number of adenomas and polyps per colonoscopy) |
| Wijnberge et al, 2020 [72] | RCT | 68 patients | Amsterdam UMC (Netherlands) | PO (median time-weighted average of hypotension, median time of hypotension, treatment, time to intervention, adverse events) |
| Wu et al, 2019 [73] | Observational study (prospective) | 3600 residents | 3 ophthalmologists, community healthcare centers (China) | AP (AUC) CO (ophthalmologist-to-population service ratio) |
| Wu et al, 2019 [74] | RCT | 303 patients, 6 endoscopists | Renmin hospital of Wuhan University (China) | AP (accuracy, completeness of photo documentation) CO (blind spot rate, number of ignored patients, inspection time) PO (adverse events) |
| Yoo et al, 2018 [75] | Observational study (prospective) | 50 patients, 1 radiologist | NR (Korea) | AP (sensitivity, specificity, PPV, NPV, accuracy) CO (sensitivity, specificity, PPV, NPV, accuracy) |
aAP: application performance.
bCO: clinician outcomes.
cCT: computed tomography.
dAUC: area under the curve.
eRCT: randomized controlled trial.
fPO: patient outcomes.
gICU: intensive care unit.
hLOS: length of stay.
iNR: not reported.
jPPV: positive-predictive value.
kNPV: negative-predictive value.
lADR: adenoma detection rate.
mPDR: polyp detection rate.
nACSC: ambulatory care sensitive admissions.
oAI: artificial intelligence.
pFFPI: false-positive per image.
qCE: cost-effectiveness.