Unveiling the Future of Postoperative Outcomes Prediction: The Role of Machine Learning and Trust in Healthcare

Introduction

Machine learning (ML) has swiftly emerged as a transformative force in healthcare, particularly in predicting postoperative outcomes. Since the first paper in the anesthesiology literature by Lee et al. in 2018 [1] dozens of papers have been published leveraging electronic health record data to predict a variety of perioperative outcomes such as mortality [2, 3], acute kidney injury (AKI) [2, 4, 5, 6], readmission [7, 8, 9] and others. However, despite the volume of papers with predictive models, a paucity of these have been implemented into clinical care. In this issue, Barker et al. provides a model to predict unplanned care escalations after surgery. This manuscript provides an excellent example of the promise of predictive models as well as the gaps that need to be bridged to deploy the models into clinical care.

Understanding Machine Learning in Healthcare

At its core, machine learning is a subset of artificial intelligence (AI) that enables systems to learn from data and make predictions or decisions without being explicitly programmed. ML models are specifically designed to be applied to new data (i.e., patients) that have not yet been seen in order to predict the future. In contrast, traditional statistical models are developed to best describe the source dataset; they are not intended to be extrapolated to new situations, though they are often used “off-label” in that way.

Machine learning models generally operate through a series of steps: data collection, data preprocessing, model training, and evaluation. Initially, a large volume of data—often comprising electronic health records (EHRs), imaging data, or patient demographics—is collected. These data are then preprocessed to handle missing values, normalize features, and address any biases. Following preprocessing, the data are divided into training and testing subsets. During the training phase, the model learns to associate input features with outcomes through algorithms such as regression analysis, decision trees, or neural networks. Finally, the model’s performance is evaluated using metrics like accuracy, precision, recall, and area under the receiver operating characteristic curve (AUC-ROC) to assess its predictive power. These metrics examine how accurate the model will be when applied to new patients.

Previous ML Models in Postoperative Outcome Prediction

A growing body of literature has examined ML models for predicting postoperative outcomes. Overall researchers have tried a variety of methods including neural networks, logistic regression and tree-based models such as random forest and gradient boosted trees. By and large, the tree-based models seem to perform the best, and generally have ROC AUCs in the 0.75–0.90 range.

Less commonly reported, but equally important are measures of model calibration such as the area under the precision recall curve (AUC-PR) and threshold measures like precision (also called positive predictive value), negative predictive value and sensitivity. These latter scores allow readers to understand how the model will perform on individual patients as opposed to simply discussing the group as a whole.

SHAP Scores and Feature Importance

To interpret ML models effectively, it is crucial to understand which features drive the predictions. SHAP (SHapley Additive exPlanations) scores are a valuable tool for this purpose. Developed from cooperative game theory, SHAP values decompose a prediction into contributions from individual features, providing insight into their importance.

For instance, in a model predicting postoperative infection risk, SHAP scores might reveal that preoperative glucose levels and the patient’s age are the most influential factors. This transparency allows clinicians to focus on key variables and integrate ML insights into their decision-making processes. By highlighting feature importance, SHAP scores facilitate the development of more interpretable and actionable models, ultimately enhancing their clinical utility [10].

Challenges and the Need for External Validation

Despite promising results, ML models are not without limitations. One significant challenge is the need for external validation. Models developed on one dataset may not generalize well to different populations or clinical settings. For example, a model trained on data from a single hospital may perform poorly when applied to a different institution with different patient demographics or treatment protocols.

External validation involves testing the model on independent datasets to ensure its robustness and generalizability. Without such validation, there is a risk of overfitting—where the model performs well on training data but poorly on new data. To address this, researchers advocate for rigorous validation strategies, including cross-validation and independent cohort testing, to confirm the model’s predictive accuracy across diverse settings.

Trust and Acceptance of ML in Healthcare

The integration of ML into healthcare presents a double-edged sword. On one hand, ML models offer the potential to enhance clinical decision-making and improve patient outcomes. On the other hand, there is a pressing need to build trust among healthcare professionals and patients.

Trust in ML systems hinges on several factors: model transparency, interpretability, and the demonstrated value of predictions. Clinicians need to understand how a model arrives at its recommendations and ensure that it aligns with their clinical judgment. Transparent models, such as those employing SHAP scores, contribute to this trust by making the decision-making process more understandable.

Furthermore, the effectiveness of ML models must be demonstrated through real-world applications. The adoption of ML technologies is likely to increase if these models consistently deliver accurate predictions and translate into tangible improvements in patient care. Collaboration between data scientists, clinicians, and policymakers is essential to facilitate the development of trustworthy ML systems and address ethical and practical concerns related to their use [11].

A Proposed Model

The paper by Barker et al. in this edition of the journal is an excellent example of what is done well in ML research as well as the gaps necessary for implementation [12]. Firstly, the authors are to be commended for a paper that identifies a clear clinical problem (unplanned upgrades of care) and produces a solidly performing model to predict it. The authors are also commended for using SHAP scores to provide visibility into the features driving model performance. Of note, many of these factors are highly correlated with patient complexity/illness (i.e. anesthesia time, Aldrete score) or direct markers of illness (i.e. tachycardia and hypotension).

However, this model is also not ready for deployment. While the authors report an AUC-ROC of 0.752 they do not report the AUC-PR curve which is critical for understanding how the model performs on the individual, as opposed to global, level. Because most patients are not “average” this crucial metric will be important for evaluating performance before implementation. Similarly missing are values for sensitivity and precision at various thresholds. Additionally, this model lacks external validation or evaluation for different subgroups (surgical procedures, demographic groups, etc.) making it impossible to evaluate or reduce bias. Thus, the manuscript provides a good baseline, but work remains before implementing the model into care.

Another noteworthy aspect of their model is that three of the top influential factors were vital signs that were “missing data”. Thus, a patient without a blood pressure or pulse oximetry measurement recorded in the EHR would be stratified at higher risk of post-discharge deterioration. It does not matter to the model whether the blood pressure was unmeasurable, not measured, or measured but not recorded. However, to a human anesthesiologist supervising the recovery room, these are important distinctions and so a missing value would probably prompt an investigation prior to making a disposition decision. This raises the aforementioned issue of how much such a model should be trusted and relied upon for decision making/support.

Conclusion

Machine learning holds immense potential to revolutionize the prediction of postoperative outcomes, offering more precise and personalized risk assessments. While existing models have shown promising results, their successful integration into clinical practice requires careful consideration of feature importance, external validation, and the establishment of trust. By addressing these challenges and fostering collaboration, the healthcare community can harness the power of ML to enhance patient care and surgical outcomes.

Data Availability

No datasets were generated or analysed during the current study.