Real-time handling of missing predictors

This editorial refers to ‘Real-time imputation of missing predictor values in clinical practice’, by S.W.J. Nijman et al., on page 154.

The use of prediction models in clinical practice has become common over the past decades. Examples in the cardiovascular domain are SCORE,¹ estimating the 10-year risk of fatal cardiovascular (CV) disease (CVD), the GRACE models,² estimating the risk of CV mortality and morbidity in various populations of patients with coronary artery disease (CAD), and CHA₂DS₂-VASc for the prediction of stroke in patients with atrial fibrillation.³ The application of prediction models in clinical practice is strongly recommended by the European Society of Cardiology (ESC),⁴ in order to stimulate evidence-based cardiology and to harmonize treatment in the European region. The mentioned models, among others, have transparency and simplicity as a common feature. Risk estimates are based on a limited number of variables, and the end users have clear insight into the relation between the value of the variable in an individual patient and the level of estimated risk, which, obviously, promotes use of the models.

The creation of prediction models is an art. In an article worth reading,⁵ Steyerberg and Vergouwe described a systematic method for model development and validation. Without going into much detail, among others, they refer to the fact that many data sets used for model development are incomplete with respect to the values of potential relevant predictor variables. The solution to exclude patients with missing values from analysis is inefficient (i.e. a complete case analysis reduces statistical power), since available information of other predictors is then also lost. More worrisome, however, is the scenario that missingness is associated with the magnitude of the unobserved variable or with the presence of the outcome of interest. A complete case analysis might then result in biased estimates of the predictor-outcome relation, with obvious consequences for the quality of the resulting prediction model.⁶ Therefore, ‘imputation’ of missing data is recommended when developing prediction models, and several statistical approaches for this have been described.⁵ Best estimates of the missing values are thus obtained, based on relations between variables in the dataset.⁷

An appropriate application of published prediction models in routine clinical practice is an art as well. In particular, one needs to understand that, while outcome predictors to be used in prediction models might be population invariant, the actual level of risk might vary considerably between populations. Hence, prediction models developed in another population must be recalibrated for appropriate risk estimation and clinical decision making based thereon. We refer to the study by Harrison et al.⁸ for an example. Another relevant aspect is the (in)completeness of data on predictors that compose a published (and validated) prediction model. Indeed, data on certain predictors might be missing in a particular patient presenting at the outpatient clinic, so that model-based outcome estimates are inaccurate or even cannot be obtained. An obvious solution, which might work in most scenario’s, is to determine the value of the predictor in question at the moment of the patient visit. However, sometimes, this takes time, while risk assessment is needed on the spot. In such cases, more complex solutions are required.

Nijman et al.⁹ describe two methods for real-time handling of missing predictor values when using prediction models in practice and faced with missing values in individual patients presenting at the outpatient clinic. Both methods aim to impute missing values based on available data in other individuals. Utilizing the mean imputation (M-imp) method, missing values are estimated by the mean value in a representative sample of the target population. The method of joint modelling imputation (JMI) allows personalized imputation by adjusting for observed characteristics of the individual patient, which results from regression analyses on multiple (representative) datasets. In a simulation study, exploring the data of the Utrecht Cardiovascular Cohorts, JMI showed better performance and calibration than M-imp regarding models aiming to predict onset of CVD or coronary death. Also, the authors demonstrated that performance of JMI greatly improved when imputations were based on all observed patient data and not restricted to only the predictors that were in the prediction model. Moreover, real-time missing predictor imputations were most accurate when the imputation method relied on characteristics that were directly estimated from a sample from the target population, rather than from an external though related dataset.

Nijman et al.¹⁰ are to be complimented for putting the somewhat overlooked topic of ‘real-time handling of missing predictors’ on the research agenda. The poorer performance of the M-imp method will not have surprised the authors, as one of the co-authors already stated in 2006 that ‘… overall mean imputation … almost always results in biased estimates’. Further research of the potential of JMI is warranted, and the method might be extended to joint modelling with multiple imputation (JMMI), as such multiple imputation is known to result in better estimates of standard errors (and confidence intervals).¹⁰ The argument that this approach is ‘computationally expensive’⁹ is not convincing now that desktops are available with high computing power. For now, we concur with the lesson learned: if it is not possible to determine the value of a missing predictor on the spot, then use all observed data, including auxiliary variables, from representative samples of the target population to determine its value. To facilitate such an approach, patients, clinicians, researchers, and regulators should be encouraged to work together to foster secure data sharing.

Conflict of interest: none declared.

 

The opinions expressed in this article are not necessarily those of the Editors of the European Heart Journal – Digital Health or of the European Society of Cardiology.