Making ICU prognostication patient centered: is there a role for dynamic information?

The advent of modern severity of illness models nearly three decades ago, such as the Acute Physiology Assessment and Chronic Health Evaluation (APACHE) model, transformed our understanding of the outcomes of critically ill patients(1). Scientists and hospital administrators have used these models to benchmark ICU performance, adjust for patient differences in nonrandomized studies of critical illness, examine secular trends in mortality, and compare severity of illness among participants across randomized trials. Despite their ability to accurately estimate the risk of death in populations of critically ill patients, these models have important limitations that hinder their ability to predict outcomes of individual patients. For example, most severity of illness models fail to predict outcomes with the certainty necessary to influence life and death decisions, and are therefore less useful at the bedside. Additional shortcomings include known variation in an individual's predicted risk of death across different models, and the failure of such models to include an individual's response to treatment, instead relying on data from the time of ICU admission(2).

In this issue of Critical Care Medicine, Mayaud et al. seek to improve upon limitations imposed by the static nature of models such as APACHE by developing a mortality prediction model in 1500 patients with sepsis and hypotension who were captured in an open access database (MIMIC II) of critically ill patients(3). This database draws minute-to-minute monitoring data, test results, physician orders, demographic, and administrative data into one source(4). The temporal granularity of the database allowed the authors to use physiologic measures before, during, and after a hypotensive episode, as well as the treatments initiated in response to hypotension, in the creation of their model. Upon testing the model on a smaller validation sample of patients from the same source, the model including dynamic information significantly outperformed all other models. Area under the receiver operating characteristic curve (AUC), a measure of the models ability to discriminate patients who live from those who die, was 0.82 for the dynamic model compared to 0.70 for APACHE and 0.54 for SAPS-I.

Mayaud et al. are not the first investigators to determine that change in clinical data over time improves clinical prediction in the ICU. Investigators developed the Sequential Organ Failure Assessment (SOFA) score(5), and the Multiple Organ Dysfunction Score (MODS)(6) in part to improve upon the static nature of traditional severity measures. Several studies demonstrate that decreases in SOFA score are highly predictive of lower mortality, and often out perform static measures such as the APACHE score measured at the time of admission(7, 8). Serial measurements of APACHE, on the other hand, also improve upon a single measurement(9). Nevertheless, the model developed by Mayaud and colleagues is an excellent reminder of the ability for temporally nuanced data to inform prognosis.

Although the inclusion of dynamic data in a predictive model is not unique, there are several more novel features of Mayaud et al.'s approach to model development of importance. Their utilization of highly granular clinical data from MIMIC II provides some insight into the promise of deidentified open source critical care datasets for clinical and health services research. As hospitals rapidly adopt electronic health records, some of the barriers to the creation of such research datasets diminish. Such datasets, drawn from dozens or even hundreds of hospitals, have great potential to improve our understanding of the epidemiology of common critical illness syndromes and play a role in comparative effectiveness research. In addition, rather than a priori variable selection, Mayaud et al. rely upon a genetic algorithm for variable selection. Some iterative approach is usually needed to select variables for model inclusion when a large number of candidate variables are available, and while a genetic algorithm appears to be a capable addition to model developer's armamentarium that includes artificial neural networks, support vector machines, and random forests, these newer ‘black box’ techniques have not been shown to outperform the standard technique of logistic regression when building prediction models in the ICU(10, 11).

It is important to keep in mind that while the model of Mayaud and colleagues appears promising, it should be validated prior to being deployed in practice. This would require comparison of its metrics (AUC, c-statistic, and net reclassification index) against APACHE IV, SAPS II, or other prediction models in a dataset truly separate from that in which the model was created. The dynamic model will likely not perform as well on an external source given unmeasured hospital and geographic factors, differences in patient populations, and variations in ways that clinical data is gathered between data sources(12). Given the aforementioned limitations, will this model really get us any closer to the goal of bedside mortality prediction for individual patients?

Perhaps the more important question is whether this application of mortality prediction is a realistic goal at all. Imprecision of risk estimates is only one reason why prognostic information has surprisingly little influence on end-of-life decision-making(13). No predictive model, no matter how much data goes into its creation or which methods are employed for variable selection and model building, will perfectly identify which individual patients will survive to leave the hospital. Even the best possible predictive model will also not address the optimistic bias that may limit patients' and family members' ability to incorporate poor prognostic information into decision making(14). As we think about moving the science of risk prediction in the ICU forward, instead of focusing on how more temporal information may improve our mortality prediction models, perhaps an alterative paradigm to make predictive models more dynamic might be to focus their development on longer-term outcomes. Given our evolving understanding of the burdens of critical care survivorship, particularly the impact of cognitive and functional dysfunction on quality of life(15, 16), and the importance of long-term outcomes to patients(17), future ICU prognostic models will be strengthened if they incorporate predictions of patient-centered outcomes beyond death, allowing individual patient decisions to be influenced by a forecast of outcomes most important to them.