Risk scores in anaesthesia: the future is hard to predict

Summary

External validation helps to assess whether a given risk prediction model will perform well in a target population. Validation is an important step in maintaining the utility of risk prediction models, as their ability to provide reliable risk estimates will deteriorate over time (calibration drift).

The number of available risk prediction models has increased over the past decade, and their use in the surgical population has been encouraged. In 2011, the National Confidential Enquiry into Patient Outcome and Death report ‘Knowing the Risk’ recommended that ‘an assessment of mortality risk should be made explicit to the patient and recorded clearly on the consent form and in the medical record’.^1,2 Risk prediction models aim to calculate an individual's risk of having (diagnostic) or developing (prognostic) an event.^3,4 There are several examples of clinical risk prediction models in anaesthesia, including the Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM), the National Emergency Laparotomy Audit, and Surgical Outcome Risk Tool (SORT).5, 6, 7, 8 Whilst such clinical risk prediction models are designed to be used routinely in clinical practice, they are often applied sporadically and without validation in their target population, which means we do not know if they will perform well for their intended purpose.²

In this issue of BJA Open, Torlot and colleagues⁹ externally validated the following four prediction models: SORT, nzRISK, and POSSUM and its Portsmouth variant, by assessing their performance in 44 031 adult Australian patients from the non-indigenous population. Their well-conducted validation study was reported using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist, which is the recognised reporting standard for multivariable prognostic or diagnostic prediction models. As the number of available prediction models grows, adherence to reporting guidelines, such as the TRIPOD statement, is essential to ensure the overall scientific integrity of such models.¹⁰

In their study, Torlot and colleagues⁹ uncovered the commonly encountered problem of ‘calibration drift’, which means calibration (how trustworthy the risk estimate is) has become worse over time.¹¹ In the study of Torlot and colleagues,⁹ calibration drift resulted in the 30 day mortality rate being over-predicted when applied to the target population. There are several reasons for calibration drift for any given prediction model, including changes in population demographics, clinical practice patterns, the introduction of new guidelines, or even a change in data recording practices or systems. Although frequently encountered, this is an underappreciated issue and highlights the importance of ensuring that risk prediction models are accurate for each intended target population before assessing their clinical value. Furthermore, prediction models are frequently developed as a one-time activity, but the reality is that model construction is just a small part of a larger ‘pipeline’, which would be more appropriately viewed as a cycle. Validation is a crucial step in these cycles.^12,13 One way to address this problem is by updating prediction models on a regular basis. The European System for Cardiac Operative Risk Evaluation is one example where an existing prediction model was updated with contemporaneous data and predictors.^13,14

Static prediction models are always at risk of being outdated. ^12,13 Therefore, approaches are needed to determine how to validate them and when they should be formally updated.¹² One solution is proposed by Davis and colleagues,¹⁵ who presented a model-agnostic calibration drift detection system. They used data-driven methods to monitor calibration metrics to detect when model performance deteriorates, providing a guide for when they should be updated. Updating prediction models requires time, money, and infrastructure that are not always readily available, and even prediction models that are periodically updated are at risk of calibration drift between updates. The ideal solution may lie in an approach called dynamic modelling (or online machine learning), in which new data are continuously monitored and the prediction model is continuously updated and validated. Although these methods are deployed in marketing and by social media companies,¹⁶ they are not yet well established in healthcare research and would require significantly more complex infrastructure and ongoing technical expertise, which is not the case for current ‘static’ prediction models.^13,15,17

An alternative to updating prediction models is to design bespoke risk prediction models for the target population in whom they will be deployed, which Torlot and colleagues⁹ discussed. Whilst this approach has merits, it risks contributing to ‘research waste’, where old, outdated prediction models are discarded in favour of newer models.¹³ In this situation, careful consideration is needed for what prediction task is required. For example, Grant and colleagues¹⁸ highlight that a model designed to benchmark multiple procedures accordingly to adjusted surgical risk will look very different to a model trying to predict a patient's particular outcome for a specific procedure.

As for the future, the increasing collection and availability of Electronic Health Records (EHRs) clearly offer a rich data source of broadly representative data often for large populations from which new prediction models can be derived, but consideration is still required for how the data were collected, and for what purpose, to understand the potential for selection bias.¹² Whilst the extent and sources of bias are yet to be fully described, validating prediction models developed in systems with inherent bias will be difficult.¹⁹

The precise future of risk prediction models in anaesthesia is hard to forecast. A plethora of policy statements and guidelines have made the case for systematic perioperative risk estimation, but it has not become embedded into routine clinical practice. Closing this translational gap requires funding for qualitative work on patient, clinician, and policy expectations of preoperative risk estimation and funding to support an implementation science-based approach to carefully integrate risk prediction models into the clinical workflow. This will require recognition of the state of healthcare technology relative to other industries and a considered approach to ensure risk prediction models help, not hinder, clinical practice.

Authors' contributions

Editorial conception: DJD, SDR, MS, MH.

Initial draft: DJD, MA; Critical revisions: DJD, MA, SDR, MS.

Funding

Dr Daniel James Drayton is an Academic Clinical Fellow in Anaesthesia funded by the National Institute for Health and Care Research (NIHR).

Declarations of interest

The authors declare that they have no conflicts of interest.