POSTCARDS from a SIESTA: Crossing the translational and generalizability gap for predictive models of ARDS-related mortality

Villar et al’s study tests a new mortality prediction score in SIESTA (ALIEN, STANDARDS, STANDARDS-2) and externally validated in PANDORA data (1). This novel dataset in SIESTA contains three trials with 1000 patients in moderate to severe acute respiratory distress syndrome (ARDS). Although the prediction of mortality in ARDS using either machine learning or scoring methods is not new, this has been confirmed in a new dataset. The generation of this dataset is itself novel, as there was an attempt at standardized oxygenation support using fixed PEEP (10 cm H2O) and FiO2 (50%) if possible. This makes P/F ratios more comparable across patients. (2)

It is commendable that the authors have used rigorous methods to train and validate these models. By using 100 bootstrapping folds, along with five-fold cross validation (using 80% train/20% test splits), the authors have created a relatively robust training pipeline. This is limited by potential information leakage, with hyperparameter tuning conducted on the full dataset as opposed to a left-out set. The benefit is that the hyperparameter tuning continues to be effective in another prospective observational cohort environment for moderate-severe ARDS.

The strongest predictors for this model included plateau pressures at t0, t24, and the number of organ failures. As high plateau pressures are associated with decreased compliance, often with more severe ARDS, this associates higher mortality with more severe ARDS. This is a reasonable finding. (3) Furthermore, as more organ failures in the sequential organ failure assessment (SOFA) is itself associated with higher mortality, it is logical that increased organ failure plays a strong role in this model. (4) However, these are not new findings.

In order to address the classic problem of high dimensionality to sample size, the authors utilize a feature selection method called the Genetic Algorithm (GA). This method is a class of search algorithms that are inspired by evolutionary biology and natural selection, and outperform random search algorithms due to their ability to utilize historical data to optimize their search space. (5,6) Within the context of feature selection, GAs utilize the concept of ‘genes’ representing the input feature domain, and an ‘organism’ that represents a potential set of optimal features. They start by randomly initializing the population, which consists of all probable solutions to a given objective. In second stage, a ‘fitness score’ is assigned to each individual candidate in the population, such that the higher the fitness score the more probable for being chosen for reproduction. Then selection occurs, in which pair-wise candidates are selected for reproduction, where variation operators are applied to either perform ‘crossover’ or ‘mutation’ tasks; the former representing where randomly selected information is used to generate a child of equal length, and the latter where new information is generated in the child. This process is followed by replacement wherein the new ‘child’ populations replace the parents, until the overall fitness score improves, indicating a more optimal solution. This process is repeated until a stopping criteria is met, usually where a threshold for a fitness score has been reached.

This heuristic based adaptation of the random search algorithm has been utilized across multiple domains, and has resulted in robust selection of highly enriched features.(7,8) Yet, there are important disadvantages of the algorithm that are pertinent to critical care medicine. Importantly, the initialization parameters often drive convergence, thus the subset of patients chosen for initialization may influence subsequent candidate populations. In the case of this particular approach, the authors sought to overcome this limitation by performing 100 iterations of bootstrapped search; thus optimizing the likelihood that the proposed set can represent generalizable features. Yet, the the conventional GA is prone to premature convergence from the lack of global search ability due to the loss of population diversity during evolution.(9,10) Sometimes, this loss of diversity occurs due to the presence of highly associated features, i.e. those that are dominant in terms of their correlation with the outcome in interest, in this case mortality. Indeed, the final set of variables proposed by the authors, including plateau pressure are highly associated with changing lung compliance. However, the use of such a dimensional reduction strategy may have obfuscated other potentially salient candidate features that may have proven to be superior predictive candidates. The lack of an alternative feature selection algorithm indeed leaves the reader with open-ended questions about what may have emerged within this highly unique dataset.

Among several active challenges of translating novel clinical decision support tools (CDSTs) to the bedside has been associated with the poor generalizability. Indeed such challenges have been reported across multiple studies, including those derived from clinical trials. A variety of contributors exist, however a key aspect arises from the occurrence of missingness of measures that are often programmatically captured during the trial. Additionally, participants are often monitored at predefined intervals, which may not be practically implemented in the real world setting. Collectively, these factors pose limitations when interpreting machine learning-based results that arise from trials data. However, several methods exist to reliably extract meaning, including in the use of robust stochastic control to approximate a probability distribution around occurrences to approximate variable-level uncertainty. These factors, along with other approaches to maximizing generalization may be best explored when such data is made available to the wider community for rigorous and reproducible experimentation.

As noted earlier, the dataset with PEEP 10 and FiO2 50% for standardizing the meaning and significance of P/F ratios is a novel exercise. Oxygenation is notoriously hard to model across two independent factors. This enforcement supports more robust comparisons.

The extension of this work using SIESTA data has generated a new mortality score for moderate-severe ARDS that is, although not novel, interesting for a first use dataset. It standardizes metrics for P/F measurement to improve generalizability. However, the new ARDS definition, including heated high-flow nasal cannula (HFNC, e.g., Optiflow, Airvo), may present complications, as there is no pressure or PEEP to adjust for HFNC. Consequently, plateau pressures and PEEP cannot be standardized. (11)

Overall, POSTCARDS, while confirming the more recent SPIRES score, remains an interesting and useful study. Confirmation of SPIRES is still a good exercise and effort by a talented team and network. This dataset opens the opportunity for further research, with additional robust insights to be drawn. We look forward to future developments from SIESTA and PANDORA data.