Table 1.
Summary of Included Studies
| Source | Study Design | Population | Sample Size | Major Findings Pertinent to This Scoping Review | Sources of Funding; Conflicts of Interest |
|---|---|---|---|---|---|
| Adhikari et al9 | Retrospective | Patients undergoing inpatient surgery | 2911 | A machine learning algorithm accurately predicted postoperative acute kidney injury using preoperative and intraoperative data | NIGMS, University of Florida CTSI; NCATS; I Heerman Anesthesia Foundation; SCCM Vision Grant |
| Artis et al10 | Observational | Trainee ICU presentations | 157 | Potentially important data were omitted from 157 of157 presentations; missing an average 42% of all data elements | AHRQ |
| Bagnall et al11 | Retrospective | Patients who had colorectal surgery | 1380 | Six traditional risk models used to predict postoperative morbidity and mortality had weak accuracy with AUC, 0.46–0.61 | St Mark’s Hospital Foundation |
| Bechara et al12 | Observational | Healthy volunteers and participants with prefrontal cortex damage | 16 | Participants began to decide advantageously before they could consciously explain what they were doing or whythey were doing it | National Institute of Neurological Diseases and Stroke |
| Bertrand et al13 | Prospective | ICU patients and attending surgeons | 419 | Clinicians thought 45% of all patients had decision-making capacity; a minimental status examination found that 17% had decision-making capacity | Pfizer, Fisher & Paykel; Pfizer; Alexion; Gilead; Jazz Pharma; Baxter; Astellas |
| Bertsimas et al14 | Retrospective | Emergency surgery patients | 382 960 | A app-based machine learning model accurately predicted mortality and 18 postoperative complications (AUC, 0.92) | None reported |
| Bihorac et al15 | Retrospective | Patients undergoing major surgery | 51 457 | A machine learning algorithm using automated EHR data predicted 8 postoperative complications (AUC, 0.82–0.94)and predicted mortality at 1,3, 6,12, and 24 mo (AUC, 0.77–0.83) | NIGMS; NCATS |
| Blumenthal-Barby et al16 | Review | Articles about heuristics in medical decision-making | 213 | Among studies investigating bias and heuristics among medical personnel, 80% identified evidence of bias and heuristics | Greenwall Foundation; Pfizer |
| Brennan et al17 | Prospective | Physicians | 20 | A machine learning algorithm was significantly more accurate than physicians in predicting postoperative complications (AUC, 0.73–0.85 vsAUC, 0.47–0.69) | NIGMS, University of Florida CTSI; NCATS |
| Che et al18 | Retrospective | Pediatric ICU patients | 398 | A gradient boosting trees method allowed for quantification of the relative importance of deep model inputs in determining model outputs | NSF; Coulter Translational Research Program |
| Chen-Ying et al19 | Retrospective | Clinic patients | 840 487 | A deep model predicted 5-y stroke occurrence with greater sensitivity (0.85 vs 0.82) and specificity (0.87 vs 0.86) than logistic regression | None reported |
| Christie et al20 | Retrospective | Trauma patients | 28212 | A machine learning ensemble accurately predicted mortality among trauma patients in the United States, South Africa, and Cameroon with AUC ≥0.90 inall settings | NIH |
| Clark et al4 | Retrospective | Surgical patients | 885 502 | The ACS Surgical Risk Calculator accurately predicted mortality (AUC, 0.94) and morbidity (AUC, 0.83) | None reported |
| Cohen et al6 | Retrospective | Studies assessing the ACS Surgical Risk Calculator | 3 | Externalvalidation studies assessing ACS Surgical Risk Calculator performance may have been compromised by small sample size, case-mix heterogeneity, and use of data from a small number of institutions | None reported |
| Delahanty et al21 | Retrospective | ICU patients | 237 173 | A machine learning algorithm accurately predicted inpatient death (AUC, 0.94) | Alesky Belcher; Intensix, Advanced ICU |
| Dybowski et al22 | Retrospective | ICU patients | 258 | An artificial neural network predicted in-hospital mortality more accurately than logistic regression (AUC 0.86 vs 0.75) | Special Trustees for St Thomas’ Hospital |
| Ellis et al23 | Prospective | Volunteers | 1948 | Induction of fear and anger had unique and significant influences on decisions to take hypothetical medications | NCI |
| Gage et al24 | Prospective | Patients with atrial fibrillation | 2580 | Models commonly used to predict risk of stroke were moderately accurate with AUC ranging from 0.58–0.70 | AHA, NIH, Danish and Netherlands Heart Foundations, Zorg Onderzoek Nederland Prevention Fund, Bayer, UK StrokeAssociation |
| Gijsberts et al25 | Retrospective | Patients with no baseline cardiovascular disease | 60 211 | Associations between risk factors and development of atherosclerotic cardiovascular disease were different across racial and ethnic groups | Netherlands Organization for Health Research and Development, NIH |
| Hao et al26 | Retrospective | ICU patients | 15 647 | Deep learning models predicted 28-d mortalitywith 84%−86%accuracy | None reported |
| Healey et al1 | Retrospective | Surgical inpatients | 4658 | Behind technical errors, diagnostic and judgment errors were the second most common cause of preventable harm | None reported |
| Henry et al27 | Retrospective | ICU patients | 16234 | A machine learning early warning score accurately predicted the onset of septic shock (AUC 0.83), identifying approximately two-thirds of all cases prior to the onset of organ dysfunction | NSF; Google Research, Gordon and Betty Moore Foundation |
| Hubbard et al28 | Prospective | Trauma patients | 980 | A machine learning ensemble predicted mortalitymore accurately than logistic regression (5% gain) | US Army Medical Research and Materiel Command, NIH |
| Hyde et al7 | Prospective | Patients undergoing colorectal resections | 288 | The likelihood of a serious complication was underestimated bythe ACS Surgical Risk Calculator (AUC, 0.69), but the calculator accurately predicted postoperative mortality (AUC, 0.97) | None reported |
| Kim et al29 | Retrospective | ICU admissions | 38 474 | A decision tree model predicted in-hospital mortalitymore accurately than APACHE III (AUC, 0.89 vs 0.87) | NCRR |
| Knops et al30 | Systematic review | Studies about decision aids in surgery | 17 | Decision aid use was associated with more knowledge regarding treatment options and preference for less invasive treatment options with no observable differences in anxiety, quality of life, or complications | None reported |
| Komorowski et al31 | Retrospective | Septic ICU patients | 96 156 | A reinforcement learning model recommending intravenous fluid and vasopressor strategies outperformed human clinicians; mortality was lowest when decisions made by clinicians matched recommendations from the reinforcement learning model | Orion Pharma, Amomed Pharma, Ferring Pharma, Tenax Therapeutics; Baxter Healthcare; Bristol-Myers Squibb; GSK; HCA International |
| Koyner et al32 | Retrospective | Hospital admissions | 121 158 | A machine learning algorithm accurately predicted development of acute kidney injury within 24 h (AUC, 0.90) and 48 h (AUC, 0.87) | Satellite Healthcare; Philips Healthcare; EarlySense; Quant HC |
| Leeds et al8 | Observational | Surgery residents | 124 | Residents reported that lack of electronic and clinical workflow integration were major barriers to routine use of risk communication frameworks | NCI; ASCRS; AHRQ |
| Legare et al33 | Systematic review | Studies about shared decision-making | 38 | Time constraints impair the shared decision-making process among providers, patients, and caregivers | Tier 2 Canada Research Chair |
| Loftus et al34 | Retrospective | Patients with lower intestinal bleeding | 147 | An artificial neural network predicted severe lower intestinal bleeding more accurately than a traditional clinical prediction rule (AUC, 0.98 vs 0.66) | NIGMS, NCATS |
| Lubitz et al5 | Retrospective | Patients undergoing colorectal surgery | 150 | The ACS Surgical Risk Calculator accurately predicted morbidityand mortalityfor elective surgery but underestimated risk for emergent surgery | None reported |
| Ludolph et al35 | Systematic review | Articles about debiasing in health care | 68 | Many debiasing strategies targeting health care clinicians effectively decrease the effect of bias on decision-making | University of Lugano Institute of Communication and Health |
| Lundgren-Laine et al36 | Observational | Academic intensivists | 8 | Academic intensivists made approximately 56 ad hoc patient care and resource use decisions per day | Finnish Funding Agency for Technology and Innovation; Tekes; Finnish Cultural Foundation |
| Morris et al37 | Interviews | Academic surgeons | 20 | Younger surgeons felt uncomfortable defining futility and felt pressured to perform operations that were likely futile | AHRQ |
| Pirracchio et al38 | Retrospective | ICU patients | 24508 | A machine learning ensemble predicted in-hospital mortality (AUC, 0.85) more accurately than SAPS-II (AUC, 0.78) and SOFA (AUC, 0.71) | Fulbright Foundation; Doris Duke Clinical Scientist Development Award; NIH |
| Pirracchio et al39 | Observational | Simulated data sets | 1000 | A machine learning ensemble predicted propensity scores more accurately than logistic regression and individual machine learning algorithms | Fulbright Foundation; Assistance Publique-Hopitaux de Paris; NIH |
| Raymond et al3 | Prospective | Preoperative clinic patients | 150 | After reviewing ACS Surgical Risk Calculator results, 70% would participate in prehabilitation and 40% would delay surgery for prehabilitation | GE Foundation; Edwards Lifesciences; Cheetah Medical |
| Sacks et al40 | Observational | Surgeons | 767 | Facing clinical vignettes for urgent and emergent surgical diseases; surgeons exhibited wide variability in the decision to operate (49%−85%) | Robert Wood Johnson/Veterans Affairs Clinical Scholars program |
| Schuetz et al41 | Retrospective | Clinical encounters in an EHR | 32 787 | A deep model predicted the onset of heart failure more accurately than logistic regression (AUC, 0.78vsAUC, 0.75) | NSF; NHLBI |
| Shanafelt et al2 | Observational | Members of the ACS | 7905 | Nine percent of all surgeons reported making a major medical error in the last 3 mo, and lapses in judgment were the most common cause (32%) | None reported |
| Shickel et al42 | Retrospective | ICU admissions | 36216 | A deep model using SOFA variables predicted in-hospital mortality with greater accuracy than the traditional SOFAscore (AUC 0.90 vs 0.85) | NIGMS; NSF, University of Florida CTSI; NCATS; J Crayton Pruitt Family Department of Biomedical Engineering; Nvidia |
| Singh et al43 | Systematic review | Articles about CRP to predict leak after colorectal surgery | 7 | The positive predictive value of serum C-reactive protein 3–5 d after surgerywas 21%−23% | Auckland Medical Research Foundation, New Zealand Health Research Council |
| Stacey et al44 | Systematic review | Randomized trials about decision aids | 105 | Participants exposed to decision aids felt that they were more knowledgeable, informed, and clear about their values and played a more active role in the shared decision-making process | Foundation for Informed Medical Decision Making, Healthwise |
| Strate et al45 | Prospective | Patients with acute lower intestinal bleeding | 275 | A bedside clinical prediction rule using simple cutoff values predicted severe lower intestinal bleeding (AUC, 0.75) | American College of Gastroenterology, National Research Service Award, American Society for Gastrointestinal Endoscopy |
| Sun et al46 | Observational | Simulated type 1 diabetics | 100 | A reinforcement learning model performed as well as standard intermittent self-monitoring and continuous glucose monitoring methods, but with fewer episodes of hypoglycemia | Swiss Commission of Technology and Innovation |
| Van den Bruel et al47 | Retrospective | Primary care patients | 3890 | Clinician intuition identified patients with illness severity that was underrepresented bytraditional clinical parameters | Research Foundation Flanders, Eurogenerics, NIHR |
| Van den Bruel et al48 | Systematic review | Articles about clinical parameters for serious infections | 30 | Traditional clinical parameters associated with serious infection were often absent among patients with serious infections | Health Technology Assessment, NIHR |
| Vohs et al49 | Observational | Undergraduate students |
34 | Higher decision-making volume was associated with decreased physical stamina, persistence, qualityand quantity of mathematic calculations, and more procrastination | NIH, Social Sciences and Humanities Research Council, Canada Research Chair Council, McKnight Land-Grant |
Abbreviations: ACS, American College of Surgeons; AHA, American Heart Association; AHRQ, Agency for Healthcare Research and Quality; ASCRS, American Society of Colon and Rectal Surgeons; AUC, area under the curve; CRP, C-reactive protein; CTSI, Clinical and Translational Sciences Institute; EHR, electronic health record; GSK, Glaxo Smith Kline; ICU, intensive care unit; NCATS, National Center for Advancing Translational Sciences; NCI, National Cancer Institute; NCRR, National Center for Research Resources, Acute Physiology, and Chronic Health Evaluation; NHLBI, National Heart, Lung, and Blood Institute; NIGMS, National Institute of General Medical Sciences; NIH, National Institutes of Health; NIHR, National Institute for Health Research; NSF, National Science Foundation; SAPS, Simplified Acute Physiology Score; SCCM, Society of Critical Care Medicine; SOFA, Sequential Organ Failure Assessment.