Table 1. Checklist for ideal algorithms in healthcare.
Desiderata | Criteria | Yes | Location | No | N/A |
---|---|---|---|---|---|
Explainable Yesa Partiallyb Noc N/Ad |
Feature importance: conveys the relative importance of features in determining algorithm outputs | ||||
Descriptive accuracy: describes what the algorithm has learned (e.g., weights in a neural network) | |||||
Simulatability: clinicians can understand and mentally simulate the model’s process for generating predictions | |||||
Relevance: describes relevancy as judged by the algorithm’s target human audience | |||||
Dynamic Yesa Partiallyb Noc N/Ad |
Temporality: captures temporal changes in physiologic signals and clinical events | ||||
Continuous monitoring: performance is reassessed at several time points, including the point at which performance is expected to plateau | |||||
Precise Yesa Partiallyb Noc N/Ad |
Data frequency: rate of data collection matches the rate of physiologic changes | ||||
Complexity: algorithm complexity matches the complexity of the prediction or classification task | |||||
Autonomous Yesa Partiallyb Noc N/Ad |
Efficiency: the algorithm executes without the need for time-consuming, manual data entry by the end user (i.e., patient, provider, or investigator) | ||||
Fair Yesa Partiallyb Noc N/Ad |
Generalizability: algorithm is developed and validated across diverse patient demographics and practice settings | ||||
Selectivity: excludes features that lack pathophysiologic or linguistic association with outcomes, but may introduce bias | |||||
Objectivity: includes variables that are minimally influenced by clinician judgments (e.g., vital signs) | |||||
Reproducible Yesa Partiallyb Noc N/Ad |
Generalizability: validated externally, prospectively | ||||
Collaboration: algorithm is shared with the research community | |||||
Compliance: fulfills SPIRIT-AI extension guidelines (if trial) and fulfills CONSORT-AI guidelines |
aOverall adjudication is “Yes” when all criteria are either met or not applicable.
bOverall adjudication is “Partially” when some but not all criteria are either met or not applicable.
cOverall adjudication is “No” when no criteria are met.
dOverall adjudication is “N/A” when all criteria are not applicable.
CONSORT-AI, Consolidated Standards of Reporting Trials-Artificial Intelligence; N/A, not applicable; SPIRIT-AI, Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence.