Figure 3.

Clinical validation strategy for medical machine learning models. Data retrieved from the ICU, from the icuCDA and hospital services (eg, icuLab) can be used for model development and validation offline and the learned models can be tested in the data stream in a test environment (Stage 1). All internally validated (Stage 1 tested) models can be run in production and displayed on the UI in the MD’s office (silent period, Stage 2), but only the models validated successfully on prospective data streams (Stage 2) can have model outputs displayed on the UI in the ICU (Stage 3), as in a randomized clinical trial for the model. Purely observational clinical outcome studies and post-implementation evaluation and retraining or fine tuning of algorithms can be performed offline using data from the ResearchDB.