Predictive averaging and Rubin’s rule-based model pooling to predict survival risk with imputations in the presence of missing patient data: methodology and verification using two case studies and simulations

Abstract

Background

Practical estimation of patient risk is often complicated by predictor values being unobserved or missing. In this paper we study the calibration and application of models for patient risk prediction when both the past patient data to which the prediction models are fit contains such missing values, while the future patient records may also be partially observed. Typical applications are often encountered in prognostic observational research or with registry data.

Methods

This paper presents a detailed study on the use of multiple imputation in prediction, to deal with missing values in both past (calibration data) as well as future patient predictor records. We present the two distinct approaches which may be considered to use imputations to account for the presence of missing values. One method directly averages survival predictions obtained from separate models which are fitted on distinct imputations of the data. The other method first pools the intermediate effect and baseline estimates of these models, before calculation of the predictions from the pooled model. Such pooled estimates could be obtained from Rubin’s rules, for example. Methods are introduced and demonstrated based on two motivating datasets. The application for Cox regression survival modeling is studied in detail. Method performance is verified through cross-validation, a separate validation set and an extensive simulation study.

Results

All methods are comparable with respect to bias and Brier score (accuracy) assessment. Major differences are however found between both methods when comparing predicted per-patient risks between repetitions of the fitting procedure with a different set of imputations and when the same number of imputations is used. Predictive averaging is preferable, because the difference between such replicate predictions can be reduced to zero by increasing the number of imputations, which is not the case with the pooled model.

Conclusions

Predictive averaging should be preferred to use of a single pooled model, when using multiple imputations for missing predictor data. Single imputation should not be used in prediction on missing data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-026-02888-z.

Background

Problem area and state-of-art

Missing predictor data poses a problem in the estimation of risk prediction rules, and especially in the application of such rules when future patient records have missing values on the required predictor values. Multiple imputation (MI) [7, 8, 29] is an attractive general approach to account for missing data, but methodological work on application in prediction is still underdeveloped. This applies particularly to prognostic studies and prediction of life-times.

Published literature on imputation in the survival context generally is limited. Carpenter and Kenward [8] dedicate a book section (8.1) to the topic, while White and Royston [40] discuss imputation for the Cox model specifically. The Cox model is also considered as an example by Bartlett et al. [2] in a paper on congenial imputation. Earlier contributions are due to van Buuren [37] and Lin [22]. A more recent paper has investigated imputation in the context of time-dependent covariates with Cox regression [21], while Bartlett [3] investigates imputation in competing risk settings. Work by Erler, N. et al. [11] investigates Bayesian forms of imputation.

With exception of the (second) update edition of the book by Carpenter and Kenward [8], the above publications are all concerned with model and effect estimation, but none investigates patient survival prediction application, which is the objective of this paper. A recent paper by Bonneville et al. [5] studies use of imputation for missing values in calibration of survival prediction, when future records to which the survival model is applied are fully observed. Aside from the fact that no prediction on future partially observed records is investigated, the analysis is also limited in that only three simulated fully observed future records are considered for a demonstration of predictive performance. Wood, A. et al. [41] study prediction problems with imputation, when future observations are also affected by missing predictor values, but do not consider either practical or methodological issues in survival application. More importantly, these authors did not investigate actual validatory assessment – neither through cross-validation or using a new validation set - which is a restriction pointed out in that paper’s discussion. Furthermore, the paper did not study the Monte Carlo variation induced by imputation. The recent second edition of the book by Carpenter and Kenward [8] proposes use of a single (Rubin’s rules) pooled model for prediction, but does not discuss survival prediction explicitly. The prediction problem with missing values in future records is also studied by Mercaldo and Blume [24] who strongly advocate using the pattern-submodel approach as an alternative to imputation. The paper investigates normal-outcome simulations as well as an actual data-based application with logistic regression, but not survival prediction. In this paper we study survival prediction, when imputations are used to account for missing values in the predictors which occur in both the calibration data as well as in future new observations to which the prediction rule is eventually applied.

Recent work by Mertens et al. [25] investigates the problem of using imputations in prediction of binary outcome, to account for the presence of missing observations in predictors. A general methodological template on the use of multiple imputation in the calibration of predictive rules is presented which identifies two basic methodological approaches. The first consists of averaging the patient-specific predictions which are obtained from separately fitted prediction models on each imputed data set within a sequence of multiple imputed datasets. We will refer to this approach as prediction-averaging (approach 1). The second and more traditional approach utilizes a single “concensus” model obtained from combining the effect estimates across the distinct regression models fitted on multiple imputations into a single estimate (using Rubin’s rules e.g.), which is then plugged into the regression equations. We will refer to this method as pooled regression (approach 2). The methodologies are evaluated for the prediction of early death. Authors conclude the first approach substantially reduces the prediction Monte-Carlo error induced by imputation.

Objectives and outline of paper

This paper generalizes and verifies application of the ideas and methods described in Mertens et al. [25] for survival outcome prognosis with Cox proportional hazards models fitted to censored time-to-event outcomes. Our work is motivated by three recent case-studies in clinical survival prediction [13, 31, 32].

Generalization to Cox regression induces additional problems. First, one needs to account for variation in estimated baseline hazards which may be induced by imputation. Secondly, multiple imputations need to take both censoring and the non-linear dependence of outcome into account.

The structure of the paper is as follows. We first introduce the two survival prediction examples. Methods section describes the methodological framework for prediction with imputation in the censored survival setting. Both the prediction-averaging based method as well as pooled estimation for Cox models are described (Implementation for Cox regression section). Two distinct versions of the pooling approach for survival analysis are presented, which differ in the manner in which the baseline hazard is handled with imputations. Results section describes the performance assessment methodology used to assess the respective prediction approaches as well as results from both simulations and application to the real data. We define the summary performance measures used, with special attention to imputation-induced Monte Carlo variation on the calibrated predictions. Application in both internal and external validation is studied. Results are presented from application of performance measures on predicted survival probabilities at 1 and 5 years of follow-up. A simulation exercise is used to verify results for method comparison and further assessment. Various combinations of missing value patterns and strength of association of predictors with outcome are investigated. We finish with conclusions and a discussion.

Data

This research arose from our regular clinical collaborations and was specifically motivated through the analysis of two real datasets in clinical survival analysis studies we encountered in that work. We therefore present a performance study of the proposed methodologies in a data analytic application in these two datasets. We first introduce the data.

CRT data

Calibration data

The CRT data consists of an observational cardiology cohort of 1053 patients [13] who underwent CRT (Cardiac Resynchronization Therapy) implantation between August 1999 to June 2013 at the Leiden University Medical Centre (LUMC). The study objective was to investigate patient prognosis using all-cause mortality as outcome (494 deaths out of 1053 patients, of which 438 are cardiovascular related). The median follow-up is 60 months. The study investigated 14 predictor variables: Age, glomerular filtration rate (Egfr), hemoglobin (Hb), left ventricular ejection fraction (Lvef), Gender, atrio-ventricular junction ablation (AtrAct), HIS bundle (His), ischemic etiology (ET), New York Heart Association class (NYHA), diabetes mellitus (Dm), mitral regurgitation (Mr), QRS duration (Qrs), left bundle branch block (Lbbb), left ventricular diastolic dysfunction (Lvdias). Four of these are continuous (italic script) with the remainder binary or polychotomous. There are 529 patients records containing missing information, of which the majority is concentrated in a single variable (Lvdias) which is missing in 524 cases (50%). In addition, information is absent for Egfr in 2 cases, for Hb in 7 cases, for Lvef in 20 cases and finally for Mr in 30 cases. The total number of missing values in the data file was 583, dispersed across 11 missing data patterns (Fig. 1).

Fig. 1.

Missing data patterns in the CRT and CLL data sets. In each picture, a row represents a missing data pattern. Blue denotes observed variables and red missing. The variable names are indicated at the top. The number of observations in each pattern is indicated at the left-hand side, while the total number of missing observations for each variable is shown at the bottom of each picture. The total number of missing variables for each pattern is shown at the right

 

In both the original clinical as well as this paper, Cox models are generated on this data adding all 14 predictors with linear effects and using dummy variables in the usual manner for categorical variable codings. Before analysis, administrative censoring was applied at 7 years from baseline.

Validation data

After publication of the original study [13] an external validation dataset was assembled. It included 216 new patients: 123 patients from the LUMC registry (enrolled from June 2013 to April 2018) and 93 patients from the Rennes University Hospital registry (enrolled from March 2012 to January 2017). All-cause mortality was recorded and predictor information obtained for the same set of 14 predictor variables for all patients with the same definitions.

For the validation data, “only” 20 patients having records containing missing values out of 216 (9% versus 50% in the calibration set). For all these 20 patients, the Lvdias measurement was missing. There was a single additional missing value on Hb, such that the total number of missing values was 21 with 3 missing values data patterns (versus 11 such patterns in the calibration data).

CLL data

The CLL data describe the risk factors and outcomes of a cohort of patients with Chronic Lymphocytic Leukemia (CLL) who received an allogeneic hematopoietic stem cell transplantation. Data have been extracted from the registry of the [10] (see References) and have been upgraded and updated through a data quality initiative. The impact on several outcomes of a large series of risk factors, including patient-, disease-, procedure- and center-related information, has been studied in two papers [31, 32]. In both papers, multiple imputation was used to account for missing values. The first paper calculates predicted survival and competing risks probabilities for reference patients by pooling coefficients of the Cox models and the baseline hazards. The second paper studies a frailty model for unmeasured center heterogeneity and gives pooled coefficients and p-values of significance tests of the frailty components.

Later in this paper, we use a simplified version of one of the analyses in Schetelig 2017a. The outcome of interest is overall survival (OS) up to five years after first allogeneic stem cell transplantation (data were artificially censored at 5 years). The dataset contains 694 patients for whom 314 deaths were observed. For the whole group, OS was 64% (SE 2%1) at 2 years and 47% (SE 2%) at 5 years. The predictor variables included in the model are age at transplantation (continuous), performance status indicated by the Karnofsky Index (four categories), remission status at transplantation (three categories), cytogenetic abnormalities (four categories), previous autologous transplantation (two categories), donor type (three categories), patient-donor sex match (four categories) and conditioning regimen (three categories). Information is missing for performance status (9%), remission status (6%), cytogenetic abnormalities (25%), patient-donor sex match (1%) and conditioning regimen (1%) (Fig. 1). No separate validation data was available for the CLL study.

Methods

We write for a calibration data-matrix of predictors, where contains missing values and is fully observed. We note that missing predictor data may be observed on distinct variables between different patients. Let be a corresponding vector of univariate survival time outcomes on n individuals. In most survival applications, survival outcome will only be partially known due to censoring, such that we only observe follow-up times subject to a corresponding vector of binary censoring indicators . The objective is to devise a prediction rule based on the data which can be applied to estimate survival probabilities at times t for future (validation) observations with predictors for which outcome is not yet observed. The notation distinguishes calibration data from future samples. As for the calibration data, samples to which the predictor rule is applied may themselves contain missing predictor values, such that the future predictor data also splits into observed and missing components as . Methodology to estimate survival predictions with imputation section describes the two methodological approaches to predictive model estimation with imputations for partially unobserved calibration data, in a situation where the future predictor data is fully observed (). Accounting for missing values in prediction section extends the methodology for the general case where the new (future) observations are also partially observed (). Implementation for Cox regression section discusses specific implementation issues for the Cox model.

Cox regression-based survival predictions with multiple imputations

We consider survival predictions calculated from a basic Cox model

with both and unknown parameters of the model. We write for the joint set of model parameters.

Methodology to estimate survival predictions with imputation

In both the predictive averaging and plug-in pooling approaches, we first apply multiple imputation to generate a set of K imputed datasets and calculate separate Cox models on each to obtain the corresponding set of within-model estimates , k=1,...,K.

The prediction-averaging method (Fig. 2) first calculates the within-model predictions

corresponding to the individual calibrated Cox models on each imputation within the sequence of multiple imputations, for k=1,...,K. We then directly aggregate the set of predictions to a combined estimate , which may be obtained using means, medians or another suitable statistic.

Fig. 2.

Diagram representing cross-validatory calculations for the prediction-averaging method (approach 1). A fold-partition is defined on the data, after which each fold is taken in turn and a single imputation generated on all data with outcomes removed in the left-out fold. The Cox model is generated on the complementary calibration data and applied to the imputed left-out fold data. This process is repeated K times for K imputations. The set of K predictions are combined within individuals. Note how each observation appears in only one left-out fold with this method, but the fold-definition can change between imputations

In contrast, the plug-in pooling approach (Fig. 3) first summarizes the set of separate model estimates to a combined estimate which is plugged into the Cox regression equation to obtain the model , with ‘suitably pooled’ log hazard ratio estimates based on multiple imputation and the baseline hazard. The required predictions are then generated from the usual corresponding equations

where with an estimate of the cumulative hazard at time t. Implementation for Cox regression section discusses the baseline hazard estimation for approach 2 in more detail.

Fig. 3.

Diagram representing cross-validatory calculations for the plug-in pooled method (approaches 2A and 2B). A fold-partition is defined on the data and then kept fixed for the remainder of calculations. K imputations are then directly calculated on the full data with outcomes removed from each left-out fold. Cox models are then fitted in the calibration part of the data and Rubin’s rules are used to generate a pooled model which is applied to the left-out fold imputations

The above two algorithms have their foundation in probability theory. Let us write for the imputations which will be used to estimate the unobserved .Then, from a Bayesian methodological standpoint, the plug-in pooled approach can be understood as approximating sampling from the conditional densities

This reveals MI as stochastic integration which obtains the estimates by (first) sampling from and then from (see Carpenter et al. [8], section 2.4 e.g.). One typically averages the samples to a pooled estimate of effects using the so-called Rubin’s rules. We will refer to this as the ‘classical’ approach, which is most appropriate for much described applications which target effect estimation (see Carpenter et al. [8] e.g. for extensive description of the argument). It gives unbiased estimates of effects (parameters within in this case) under the Missing At Random (MAR, Molenberghs et al. [26]) assumption.

In prediction application however, we should in principle use the so-called predictive density instead (see Seber, 1984, page 292; among others), which models the required future outcome as a function of the observed past calibration data together with the new observed predictor record . Prediction-averaging circumvents calculation of pooled parameter estimates by directly estimating the future outcome from the predictive density

As for ‘classical’ applications of MI for the estimation of model (effect) parameters, stochastic integration may again be used to approximate the relevant moments of this density by first sampling the and as described previously and then generating either samples or the expectations corresponding to the conditional densities for k=1,...,K. In this paper, we will pursue the latter option.

Accounting for missing values in prediction

For a new observation of which the outcome has not yet been observed and to which the prediction rule will be applied, the corresponding predictor vector may itself contain missing values . These must also be estimated. From a purely methodological point of view, this is easily achieved by extending the above (stochastic) integration to also include

When using MCMC in the Bayesian context, an often used simple method to implement the above prediction with stochastic integration, is to combine the existing calibration data with the new observed predictor data for which predictions are to be generated, by appending the new data record to the calibration data matrix consisting of the records and setting the future (unobserved) outcome to missing within the augmented data matrix. An MCMC chain is then run with this combined data-matrix, treating all missing observations as unknown parameters which are simulated from the corresponding conditional densities, including the . This is an established and tested approach within Bayesian statistics which is frequently re-discovered, such as by Janssen et al. [17] for clinical prediction application. The same approach is also used for prediction in the modern INLA approach [20] e.g. and can also be found in a more recent contribution from Nijman et al. [27].

This solution is also convenient with the semi-parametric partial-likelihood-based Cox model as computations can be carried out with standard software. With this implementation, we generate multiple imputations , k=1,...,K for both the unobserved calibration and prediction data on the augmented combined dataset , treating the as yet unknown and to be predicted future outcome as missing data. The resulting imputations , k=1,...,K are combined with the observed calibration data and used to fit the corresponding Cox models across all imputations, which can again be done with standard software.

As discussed above, we may now either pool the model parameters in some sense across the K models (using Rubin’s rules e.g.) to define a combined consensus Cox model and apply this to the K imputed predictor sets , k=1,...,K. Alternatively, each Cox model may be directly applied to its compatible imputed predictor set and the corresponding prediction calculated.

Irrespective of the approach taken and when we have missing values in the predictor set, there will in full generality be K predictions , k=1,...,K, although these will coincide for the plug-in pooled approach when predicting fully observed records. This set of predictions may either be reported as such or summarized using statistics.

Implementation for Cox regression

Implementation in survival analysis generally and Cox regression specifically introduces two complicating issues. The first is the precise definition of pooling of model parameters for the baseline hazards with Cox regression. The second concerns the manner of accounting for censored survival outcomes with imputation.

The special issue of model parameter pooling arises in prediction applications for the second approach (plug-in pooled method) when we combine the model parameters to generate the pooled model. Two basic approaches are available, denoted 2A and 2B respectively, which are distinguished by the calculation of the consensus baseline hazard estimate . The simplest solution (2A) straightforwardly averages (Rubin’s rules) the regression parameters and baseline hazard estimates, to define the consensus final model using the estimates

With this implementation, the within-imputations hazard estimates are calculated directly using the Breslow estimator on the corresponding imputations and then combined directly to give a consensus hazard estimate . In other words, Rubin’s rules are applied separately to hazards and effect estimates. The second approach (2B) first applies Rubin’s rule for the combination of the regression parameters only (the above second equation), and then subsequently uses this estimate to directly calculate the baseline hazard using the Breslow estimator on the “average” of the imputed datasets , in a single estimation step. Note we thus ignore the within-imputations estimates .

There are several options to combine the constituent predictions , k=1,...,K (whether using the first or the second approach): using Rubin’s rules means-averaging, applying medians or transforming the probabilities using a linearizing transform such as the logit and then average on the transformed scale. In this paper we used means-averaging directly on the calculated probabilities.

The second key issue which must be addressed for survival applications is to handle censoring. To address this problem, we used the method described by White and Royston [40] (see also Carpenter et al. [8] (section 8.1.3, page 173). This method substitutes the Nelson-Aalen estimates of the cumulative marginal hazard for the observed follow-up time in the calibration data. We chose this method primarily because it reduces the computational burden. Imputation models are then constructed using the Nelson-Aalen estimate and censoring indicator, in addition to the regression covariates. The imputations themselves were generated using the Fully Conditional Specification method ([26], chapter 13).

With the above choices, we have defined 3 implementations for Cox proportional hazards modeling of two fundamentally distinct methodological concepts on the use of multiple imputation for missing values in prediction, which we refer to in tables, graphs and the remainder of the paper as approaches 1 (prediction-averaging), 2A and 2B (plug-in pooled estimation).

Cross-validating a prognostic rule with multiple imputations

An advantage of the above methodology is that it is relatively straightforward to apply with internal validation, such as cross-validation. Once a fold-partition consisting of L folds has been defined on a dataset, we may remove each fold in turn and set its outcome data to missing, after which the above described methodology (Accounting for missing values in prediction section) is applied on the union of the calibration fold with the outcome-deleted left-out fold. Predictions may be generated with approaches 1, 2A or 2B (Methodology to estimate survival predictions with imputation and Implementation for Cox regression sections). Figures 2 and 3 highlight the structure of the methods. Note how for approach 1 we define the fold partition first, after which imputations are directly generated from the data with outcomes set to missing in the left-out fold. Once the Cox model is fitted on the calibration portion of the data, the predictions may be calculated directly for the left-out data. Once predictions have been obtained for all observations across all folds in this manner, we have the option of using a new fold definition and repeat the entire process up to a maximum K times. For approach 2, the fold definition are kept fixed and K imputations are directly generated on data with outcomes set missing for each left-out fold. The pooled model is then estimated across the imputations and subsequently applied to the left-out fold.

A fixed fold definition is used for the second approach, as this method involves pooling of regression coefficients first. For the first method, this restriction does not apply, and hence we have the option to change the fold definition from each imputation to the next. This is not essential however. In this paper we chose to change fold definitions. Irrespective of the approach, every observation will appear in one fold only for any imputation (see Figs. 2 and 3). The predictions which are then pooled for each observation are obtained from the respective folds of which it is a member, across the imputations.

It is worth pointing out that in such cross-validatory calculations, the fold definitions are completely at random and thus the outcome data which is set to missing in any left-out fold is hence completely at random also, by definition of the procedure. As explained by Little [23] page 1227, missing outcome data does not contribute to the likelihood equations of the regression of outcome on predictors. Regarding the Nelson-Aalen estimator calculations and to respect the cross-validatory logic, these are always computed from the data in the calibration set corresponding to each left-out fold only. The Cox models themselves are subsequently fitted using the original outcome data within the imputed calibration sets.

Software

We implemented the approaches shown in Figs. 2 and 3 in the statistical analysis software R Core Team [28]. Cox regression models were fit with the coxph function from the “Survival” package [35]. Ihmputation was carried out with te package ‘mice’ (see van Buuren and Groothuis-Oudshoorn [39] and van Buuren [38], page 276) with standard settings, which implements the so-called “chained equations approach”. The methods presented in this paper were implemented in a survival extension of the R package ‘mipred’, available from CRAN and Github (https://github.com/BartJAMertens).

Results

Application and assessment in the CRT and CLL data

We investigate the variability of survival predictions through Monte Carlo simulation, using the CRT and CLL datasets. In addition, we investigate prediction accuracies (Brier score) [14, 34]. First, we investigate MC variation across single-imputation-based replications of predictions, for 1000 replications r=1,...,1000 of the prediction procedure. Note how we change the notation to subscript “r” here, instead of the previously used “k”, to emphasize we are investigating variation between repetitions of the prediction procedure when K is held fixed equal to 1. We then study the between-replicate MC variation of the previously described combined predictions derived with methods 1, 2A and 2B (Implementation for Cox regression section) when multiple-imputation is used instead of single-imputation. This is done for increasing numbers of imputations K>1, specifically 10, 100 and finally 1000 imputations. Accuracy measures of prediction are subsequently studied for increasing numbers of imputations used (K=1, 10, 100 and 10000. Finally, we assess predictive variation for the CRT data in a new validation set. All cross-validation statistics are calculated with the L-fold cross-validatory approaches described above and in Figs. 2 and 3, with L=10.

Cross-validatory assessments for the CRT and CLL data

Monte Carlo variation of single-imputation based predictions

Approaches 1, 2A and 2B coincide when single-imputation (K=1) is used. Figure 4 plots cross-validated survival predictions at t=5 years versus the within-patient averages

across 1000 replicates r=1,...,1000 of single-imputation-based predictions, for the CRT and CLL data, where i=1,..,n denotes each individual with n the sample size. We distinguish between predictions corresponding to patients with partially missing predictor records (red) (which must also be imputed) and those on fully observed records (black) (no further imputation required on the predictor record). MC variation of single-imputation-based predictions is high. Observations containing missing values have higher variation of associated predictions as compared to complete records.

Fig. 4.

Replicated cross-validated survival predictions , with r=1,...,1000 (1000 replications) for each patient when using single-imputation (K=1) at t= five years follow-up in both the CRT (left plot) and CLL data (right plot) plotted versus the mean of these replicate predictions . Note that approaches 1, 2A and 2B coincide for single imputation. Red plotting symbols show predictions for individuals with missing covariates, while black denotes predictions based on fully observed records

 

Following Mertens et al. [25], we calculate a simple summary measure of this variation by calculating deviations around the mean. As shown in Fig. 3, the variation is heteroscedastic, but nevertheless of near-constant variance for patients with average survival rate between 0.2 and 0.8 (see also Cox, D. R. and Snell, E. J. [9], page 18). An ad hoc statistic may now be defined directly at the probability scale, by discarding all deviations corresponding to patients with or and computing the and percentiles and across the remaining deviations . Finally, we report as a measure of spread of predictive probabilities induced by imputation variation (expressed as percentage). Note that the interpretation of the R statistic is in essence that of a range statistic on within-individual deviations from their mean prediction (across the replicates). It is thus a non-parametric estimate of the variation in the predictions which is due to imputation. By restricting the calculation to those individuals with a mean prediction between 0.2 and 0.8, the deviations from the mean are of approximately constant variation and moreover approximately symmetric about the means. In addition, it captures variability where it is highest. The R statistic will equal approximately 2 times the distance from the mean to either the 10th or 90th percentile (Q) – which are themselves also approximately constant across the same range of mean-predictions.

For the CRT data and when K=1, this measure is as large as 15.3% at 5 years (11.8% at 1 year) when calculated using predictions from partially observed records only. These numbers are much smaller when predicting for individuals with fully observed records, respectively 6.7% and 6.1% at 5 and 1 years. We find similar results for the CLL data with K=1 as R=17.3% and 13.8% at 5 and 1 years respectively for partially observed records, reducing to 9.9% and 7.5% for fully observed records respectively. These numbers may also be found in the left-most column of Table 1 (for K=1). The lower levels of variation for prediction from fully observed records can be explained by imputations inflating variation of predicted probabilities in two ways. This happens first, due to the induced uncertainty in the fitted models. When predicting records which contain missing values, the imputed values will themselves add to the variation of the predicted probabilities.

Table 1.

Variance measures R(t) (percentage scale) fort=1 or 5 years calculated on cross-validated predictions (survival probability) for the CRT and CLL data

Year	Observation status	Variation of predictions when using K multiple imputations
		K=1	K=10			K=100			K=1000
		CRT DATA : Approaches 1 (averaging), or 2A or 2B (pooled-model based)
		ap1,2AB	ap1	ap2A	ap2B	ap1	ap2A	ap2B	ap1	ap2A	ap2B
1	missing	11.8	3.5	5.0	5.1	1.2	4.2	4.3	0.4	3.8	3.8
	observed	6.1	1.9	5.2	5.2	0.6	4.9	4.9	0.2	5.0	5.0
5	missing	15.3	4.6	6.5	6.4	1.5	5.0	5.0	0.5	4.7	4.7
	observed	6.7	2.1	5.2	5.2	0.7	4.9	4.9	0.2	4.9	4.9
		CLL DATA : Approaches 1 (averaging), or 2A or 2B (pooled-model based)
		ap1,2AB	ap1	ap2A	ap2B	ap1	ap2A	ap2B	ap1	ap2A	ap2B
1	missing	13.8	4.5	6.5	6.5	1.4	5.0	5.0	0.4	5.2	5.2
	observed	7.5	2.3	5.8	5.8	0.7	5.5	5.5	0.2	5.5	5.5
5	missing	17.3	5.5	8.3	8.3	1.7	6.4	6.4	0.5	6.4	6.5
	observed	9.9	3.0	7.5	7.6	1.0	7.2	7.2	0.3	7.3	7.3

The R(t) measure summarizes MC variation between predictions across 20 replicate analyses using the same prediction method for eitherK=1, 10, 100or 1000. Please refer to the paper for the precise definition of the measure. The shorthand notations “ap1”, “ap2A” and “ap2B” refer to approaches 1, 2A and 2B respectively. Note that all methods coincide whenK=1(single imputation)

Monte Carlo variation between replicates of multiple-imputation-based predictions

The above results suggest survival prediction with single-imputation should be treated with caution and might be ill-advised. This raises the question to what extent the Monte Carlo errors could be reduced by using multiple imputations K>1, where approaches 1, 2A and 2B will differ.

We therefore generated 20 repeat calibrations for each of the methods (replications of the analysis) using a new set of imputations for each calibration (new seeds for each replicated analysis). This generates 20 replicate estimates of the combined prediction for each patient and with each of the 3 methodologies, for K= 10, 100 and 1000, in addition to the above discussed results for K=1 (single imputation). This allows us to investigate the sensitivity of prediction at the patient-level to imputation variation when K>1.

Let be the predicted (combined) survival probability with any of the above methods at time t for the patient and r denotes the replicate (calibration) of the model (to simplify the notation, we omit the chosen number of imputations used in the calibration). In full analogy to the procedure for single-imputation, we now compute variation measures R(t) based on the deviations at t=1 and t=5 years and the mean across replicates.

Table 1 shows the calculated statistics R(t) (expressed as percentage) for K=10, 100 and 1000 and all 3 methods separately, in addition to the results for K=1 (already discussed Cross-validatory assessments for the CRT and CLL data section). These numbers represent ranges of the prediction variation expected at the individual (patient) level should we recalibrate any approach, due to Monte Carlo variation in multiple imputation. Multiple conclusions can be drawn. First, approach 1 has smaller between-replicates variation in prediction as compared to methods 2A or 2B for K>1. This is true, irrespective of the number of multiple imputations considered (10 to 1000), both for the prediction from fully observed and with (partially) missing records and at 1 and 5 years follow-up. Secondly, we notice an at least doubling of the measures R(t) when comparing approaches 2A and 2B with method 1, for K=10 and on fully observed observations, while numbers increase by a third on partially observed records. The differences are even larger at K=100 and particularly for 1000 imputations where the range statistics ‘R’ are separated by an order of magnitude in favor of approach 1 and irrespective of whether we are predicting on fully or partially observed records. Thirdly, we find that variation reduces as we increase the number of imputations on which the prediction combination is based (for all methods). Results indicate that 10 imputations are insufficient for predictive calibration in the presence of missing data in these examples and that substantial improvements can be made by increasing this number to 100 at least. However, for approach 1, reductions in predictive variation are still achieved when using 1000 instead of 100 imputations. Indeed, up to 1000 imputations may be required, at least for the data investigated in this paper, as only for K=1000 and approach 1 does the predictive variation drop to below 1%, which seems required for practical use in medical application. In contrast, variation does not reduce substantially for approaches 2A and 2B when increasing imputation numbers beyond 100. The predictive variation measure R(t) is stuck well above 4.5% for methods 2A and 2B and for both K=100 and 1000. Remarkably, this is also true for prediction based on the fully observed records. There is little difference in Monte Carlo variation for approaches 2A and 2B between prediction based on fully observed or partially missing records when K=1000. This suggests there may be a bound on the number of imputations beyond which the Rubin’s rules based methods no longer profit from increased numbers of imputations. Predictive variation is substantively and systematically lower for fully observed records, when comparing with results for partially missing records. The latter feature would be expected, as predicting from partially missing records needs to account for the additional uncertainty due to the required imputation. The above results are consistent across analyses of both the CRT and CLL data, suggesting results represent a general property of the methods and not the individual data.

Figures 5 and 6 plot the calculated R(t) measures (Table 1) versus number of imputations (K=10,100,1000) for t=1 and 5 years. The difference between approach 1 versus 2A and 2B is evident. Even at K=10, approach 1 reduces the variation measure ‘R’ to levels which are not matched for approaches 2A and 2B with K=1000. Reductions in predictive variation achieved by using multiple imputation as compared to single-imputation are substantial, particulary for approach 1, which emphasizes the need for multiple imputation in prediction using large values of K.

Fig. 5.

Cross-validation based variation measure of predictions R(t) (as percentages, within the calibration set of 1053 patients), across replicate calibrations for approaches 1, 2A and 2B versus number of imputations at 1 and 5 years follow-up for the CRT data. Dashed lines are for t=5 and solid lines are for t=1. Solid filled dots correspond to results for fully observed records and open dots are for predictions with partially missing records

Fig. 6.

Cross-validation based variation measure of predictions R(t) (as percentages, within the calibration set of 694 patients), across replicate calibrations for approaches 1, 2A and 2B versus number of imputations at 1 and 5 years follow-up for the CLL data. Dashed lines are for t=5 and solid lines are for t=1. Solid filled dots correspond to results for fully observed records and open dots are for predictions with partially missing records

Accuracy for the CRT and CLL data

For the CRT and CLL datasets, we evaluate all methods (1, 2A and 2B) using the Brier scores, as defined in section 3.4 of Houwelingen and Putter [14] at both 1 and 5 years follow-up, accounting for censoring. The scores were calculated using the R package “pec” Gerds [12] for each replicate (see description above on the replication process) and then averaged across replicates, for each value of K=1, 10, 100 and 1000.

Table 2 shows results for K=10 at 1 and 5 years follow-up. We give summary measures on the full set of cross-validated predictions (column “All obs.”). The calculation is repeated using only the predictions corresponding to observations with missing values (“Missing”) and likewise using only the completely observed observations (“Fully obs.”). Note that the latter is not to be confused with so-called “complete-case analysis”, which would instead subset the patients based on the available information prior to the analysis.

Table 2.

Cross-validation based mean Brier score statistics across 20 replications for both the CRT and CLL data at 1 and 5 years follow-up, based on combined cross-validation and multiple imputation across approaches, usingK=10

	20 replicates	CRT data			CLL data
	K=10	Missing	All obs.	Fully obs.	Missing	All obs.	Fully obs.
1 year	Approach 1	0.071	0.067	0.063	0.202	0.189	0.182
	Approach 2A	0.071	0.067	0.063	0.202	0.189	0.182
	Approach 2B	0.071	0.067	0.063	0.202	0.189	0.182
5 years	Approach 1	0.211	0.191	0.176	0.237	0.241	0.242
	Approach 2A	0.211	0.191	0.177	0.237	0.242	0.243
	Approach 2B	0.211	0.191	0.177	0.237	0.242	0.243

All methods have effectively equal performance, whether at 1 or 5 years follow-up. As expected, Brier scores are systematically larger at 5 years follow-up. Results calculated with the full set of predictions appear to be a mixture between the corresponding results calculated using either predictions on partially-missing or completely observed cases only. Higher values would be expected on the missing data portion as a consequence of the increased uncertainty in prediction. The complementary tables for K=100 and 1000 are virtually indistinguishable from results for K=10 and hence not shown to save space. The Monte Carlo standard deviations calculated on the replicate Brier scores were systematically smaller than 0.005 and not shown to save space. Figures 7 and 8 show mean Brier scores versus imputation numbers K, calculated separately on the full set (middle picture), on predictions for partially-missing records (left picture) and for fully observed records (right picture), for the CRT and CLL data separately. The pictures confirm the above discussion, but note that Brier scores are slightly larger for K=1 as compared to K \ge 10.

Fig. 7.

Cross-validation based mean Brier scores across 20 replicate analyses and at 5 years follow-up from approaches 1, 2A and 2B for the CRT data versus imputation number K. The plotting symbols (1, 2, 3) refer to methods 1, 2A and 2B in that order

Fig. 8.

Cross-validation based mean Brier scores across 20 replicate analyses and at 5 years follow-up from approaches 1, 2A and 2B for the CLL data versus imputation number K. The plotting symbols (1, 2, 3) refer to methods 1, 2A and 2B in that order

Validation study results for the CRT data

We applied the described methods to predict survival outcomes for the CRT validation set (216 new patients), based on Cox models fitted using the calibration data (1053 patients). Predictions were generated by adding each (validation) patient’s data record to the calibration set on its own (add-one-in), with outcome data deleted, after which the above cross-validatory approach was applied to predict the outcome for the added patient, by deleting the added observation from the augmented data matrix using the cross-validatory approach described before. This procedure was applied for all 216 validation patients individually. Note this procedure can be described as an “add-one-in” approach, which is the logical counterpart to the more common leave-one-out approach for within-sample validation. The sample size of the initial data matrix prior to prediction of any validation sample is always 1054 with this method.

Figure 9 displays the deviation measure R(t) across 20 replications of approach 1 as described before, versus the number of imputations used. From a qualitative point of view, these results based on a separate validation set are identical to those discussed in Cross-validatory assessments for the CRT and CLL data section using cross-validation in the calibration data. The between-replication variation is somewhat higher than seen in Fig. 5, which is consistent with within-sample cross-validation tending to underestimate variability.

Fig. 9.

Variation measure R(t) (percentages) of predictions in the CRT validation set (216 individuals), based on Cox models fitted in the calibration set (1053 patients), across replicate calibrations for approaches 1, 2A and 2B versus number of imputations at 1 and 5 years follow-up. Dashed lines are for t=5 and solid lines are for t=1. Solid filled dots correspond to results for fully observed records and open dots are for predictions with partially missing records

Simulations

Simulations for imputation with survival outcome

A disadvantage of the above data analytic applications is that we cannot assess bias as we do not know the true population survival probabilities. We therefore conduct a simulation experiment. The below description follows the setup previously described by Mertens et al. [25] but adapted for survival outcome with proportional hazards. It is set up to simulate datasets resembling the CRT data to some extent. For ease of reference, we repeat some of the simulation description from Mertens et al. [25] and explain the modifications for survival analysis.

Simulated data is generated to consist of a follow-up time, a censoring status indicator δ and a predictor matrix consisting of 4 continuous covariates which are drawn from a multivariate normal distribution , with , j=1,..,4. For the covariance matrix we choose the sample covariance matrix of the standardized continuous variables (Age, Egfr, Hb, Lvef) in the CRT data (supplementary materials). The survival times , where i denotes the individual, with i=1,...,n and n=1000 are generated from an exponential density with proportional hazards

with the corresponding row-vector of predictors and fixed baseline hazard λ =0.0073. The hazard ratios are chosen as

such that are fixed, while we set to either a low or high effect equal to łog (1.1) or łog (2) respectively. Censoring times are drawn from a uniform density on the interval [13.5, 167.5], resembling the CRT dataset. The observed follow-up time is generated for each individual by taking the minimum between the simulated survival and censoring times. Finally, administrative censorship is applied to the simulated data at t=84 months, as for the CRT data. The above choices for λ and the hazard ratios simulate data with similar censoring and survival proportions as in the CRT data. The concordance statistic for the low and high scenarios are 0.60 and 0.68 respectively. The apparent Brier scores at 1 and 5 years follow-up are 0.08 and 0.22 when and 0.08 and 0.18 when .

Once the above described simulated outcomes and predictor data have been generated, missing values are introduced by removing observations from predictor , using 4 distinct scenarios. The scenarios are defined as all combinations of either low or high association of with survival outcome () and low or high numbers of missing observations (10 or 50 percent) in as indicated in Table 3. For each of these scenarios, the missing values are introduced either missing completely at random (MCAR, Molenberghs et al. [26]) or missing at random (MAR) (see Carpenter et al. [8], section 1.4 for definitions), such that we have 8 scenarios in total. Again following Mertens et al. [25], for simulation subject to missing at random, unobserved values are introduced by calculating probabilities for each individual, where and M denotes a fraction between 0 and 1 chosen to achieve the desired numbers of missing observations. A draw is then generated from the Bernoulli densities with probabilities for each observation to decide on removal of the corresponding predictor values in .

Table 3.

Definition of 4 simulation scenarios in a design

		Percentage of missing observations ()
		Low [10%]	High [50%]
log(hazard ratio) ()	Low [łog (1.1)]	Scenario 1	Scenario 3
	High [łog (2)]	Scenario 2	Scenario 4

For each of the above described simulation scenarios, we generate 100 simulated datasets, with sample size n=1000. For each simulated dataset, 20 replications of each approach are generated. We calculate average Brier scores , averaging across both the simulations s=1,...,100 and replications r=1,...,20 for each scenario. In simulations, we may investigate bias as well as variance, as we have the true survival fractions available for each simulated individual at any time t, based on the assumed simulation model. We assess these with the following definitions.

Let be the fitted survival probability from any approach at time t and for the individual within the simulated dataset and for the replicate analysis. We then calculate the mean across replications for the individual within the simulation and subsequently, the deviations . In analogy to the above description for the measures D, we now calculate the and percentiles and across all deviations corresponding to individuals within the 20 and 80 percentiles of and for the simulation. We next calculate the range statistics as a measure of variation for each simulation and summarize using the average across simulations s as our final summary measure of variance.

To define a measure of bias, we proceed similarly by first calculating the deviations of the average across replicates from the true survival fraction

within the simulation. We then calculate the mean deviation across i within s and for those individuals with . We finally compute a summary measure of bias by taking the average of the measures across s.

Investigating simulations

We investigate simulations as described in Summary of key results and recommendations section, for both the MCAR and MAR scenarios and applied methods 1, 2A and 2B on each simulated dataset with K=1, 10 and 100. The full set of results are presented in supplementary materials using both tables and figures. For each method, the results summarize the simulation study use the following structure.

• We report results for the variance measure , the bias and the average Brier score for each simulation scenario.

• Results are calculated separately for prediction at either t=1 and also at t=5 years follow-up.

• The statistics are calculated separately on the two subsets having either fully observed records and for the records containing missing information.

The key differences between methods can be found for variation and bias, which we discuss below.

Figure 10 (for prediction from fully observed predictor records) and Fig. 11 (for partially observed predictor records) show results for the R(t) variation measure when predicting at t=5 years follow-up in so-called nested-loop plots [30] and Bonneville et al. [5]]. The figures show results on 8 simulation scenarios (all combinations of MCAR versus MAR, high or low and number of missing values M high or low). For each simulation scenario, 3 connected lines are displayed which show the evolution of the R(t) variation measure for the 3 distinct methods (1, 2A, 2B) as K increases from 1, to 10 and 100. The combination of design parameters defining each simulation pattern is indicated by the step functions at bottom of the plot. The figures for t=1 are in the supplementary materials, and show similar patterns but with lower levels of variation. All other nested-loop plots in the paper and supplementary materials have similar structure.

Fig. 10.

Variation (Monte Carlo) in predictions due to imputation as measured by the R(t) measure at t=5 years for fully observed records. The effect of increasing numbers of imputations on variation is shown through connected lines across K=1, 10, 100, for each prediction approach and within each simulation scenario. The step functions at the bottom of the plot describe the distinct simulation scenarios considered. Each step represents a level of a factor being varied, and is read from left-to-right [M represents the two levels of the amount of missing values (from low to high), the log(hazard ratio (low to high)]. K is the number of imputations used [from 1 - i.e. single imputation to 10 and 100)]. The 4 left-most scenarios are for a MAR missingness mechanism and the four right-most are for MCAR

Fig. 11.

Variation (Monte Carlo) in predictions due to imputation as measured by the R(t) measure at t=5 years for partially observed records. The effect of increasing numbers of imputations on variation is shown through connected lines across K=1, 10, 100, for each prediction approach and within each simulation scenario. The step functions at the bottom of the plot describe the distinct simulation scenarios considered. Each step represents a level of a factor being varied, and is read from left-to-right [M represents the two levels of the amount of missing values (from low to high), the log(hazard ratio (low to high)]. K is the number of imputations used [from 1 - i.e. single imputation to 10 and 100)]. The 4 left-most scenarios are for a MAR missingness mechanism and the four right-most are for MCAR

Several aspects are of note. First, imputation-induced variation in predicted probabilities within individuals is systematically larger for approaches 2A and 2B than for approach 1, when comparing at a fixed number of imputations K. The improvement of prediction-averaging approach as compared to use of a pooled model is particularly pronounced for prediction on fully observed predictor records (Fig. 10). When predicting from partially observed records, the improvements of prediction-averaging are especially for scenarios with low where imputation

Variation is generally much higher for scenarios with high levels of missing data as compared to scenarios with low levels of missing values. This would be expected due to the variation induced through the much higher number of imputations required in the calibration data. As expected, 5-year predictions are much more variable than the 1-year predictions, irrespective of the simulation scenario or whether predicting from fully observed records or not.

Note that the variabilities can be high. For example, for K=1, MCAR and prediction on partially observed predictor records, the R(t) variation measure goes as high as 39% at t=5. This variability is reduced to 5% by increasing the number of imputations to K=100. Variation can be reduced substantially for the prediction-averaging method when increasing numbers of imputations to 100, sometimes by an order of magnitude. Variation improvements are much more resistant to increases in numbers of imputations for approaches 2A and 2B. The results confirm those from the data analytic applications.

Figure 12 (for prediction from fully observed predictor records) and Fig. 13 (for partially observed predictor records) show the average bias when predicting at t=5 years follow-up. The figures for t=1 are in the supplementary materials and show similar patterns but with lower levels of bias.

Fig. 12.

Mean bias in predictions at t=5 years for fully observed records. The effect of increasing numbers of imputations on variation is shown through connected lines across K=1, 10, 100, for each prediction approach and within each simulation scenario. The step functions at the bottom of the plot describe the distinct simulation scenarios considered. Each step represents a level of a factor being varied, and is read from left-to-right [M represents the two levels of the amount of missing values (from low to high), the log(hazard ratio (low to high)]. K is the number of imputations used [from 1 - i.e. single imputation to 10 and 100)]. The 4 left-most scenarios are for a MAR missingness mechanism and the four right-most are for MCAR

Fig. 13.

Mean bias in predictions at t=5 years for partially observed records. The effect of increasing numbers of imputations on variation is shown through connected lines across K=1, 10, 100, for each prediction approach and within each simulation scenario. The step functions at the bottom of the plot describe the distinct simulation scenarios considered. Each step represents a level of a factor being varied, and is read from left-to-right [M represents the two levels of the amount of missing values (from low to high), the log(hazard ratio (low to high)]. K is the number of imputations used [from 1 - i.e. single imputation to 10 and 100)]. The 4 left-most scenarios are for a MAR missingness mechanism and the four right-most are for MCAR

When present, bias tends to be small. It is never above 4% (absolute value) and typically below 2%. The magnitude of these biases is also smaller compared to the levels of variation discussed above and for the same scenarios and number of imputations K. Those biases are somewhat larger when predicting on partially observed records as compared to prediction on fully observed data. Likewise, prediction at later follow-up (5 years) is associated with higher biases as compared to early follow-up (1 year), when bias exists.

An unexpected result is the poor performance of method 2B (pooling of the regression coefficients first and then estimating the Breslow estimator based on these pooled estimates) as compared to method 2A (straightforward averaging of separate Breslow estimates across imputations) for some simulation scenarios. This occurs for the combination where both the regression effect as well as the percentage of missing values M are set to a high value (scenario 4). Remarkably, the problem is not corrected but actually worsens the higher the number of imputations K employed, which gives further evidence against the use of pooled models in prediction.

We did not find substantive differences in Brier scores between the methods, when comparing on the same simulation scenario. Neither did we see noteworthy changes in Brier scores within any method and for a fixed simulation scenario, when increasing the number of imputations K from 1 to 100. Calculated Brier scores are systematically higher for predicting at t=5 as compared to t=1 as expected. We note these results again corroborate findings from the data analytic examples.

Discussion

Current clinical prediction practice and number of imputations for prediction

Molenberghs et al. [26] (page 253, “How many imputations?”) point to the fundamental importance of Monte Carlo error induced by imputations and the need to control this variation by using sufficient numbers of imputations. They state that the early literature may have suggested too small numbers of imputations and point to recent references showing Monte Carlo errors being “visibly affected” when “MI analysis is repeated with a different set of imputations using the same [number of imputations]”. Carpenter et al. [8] (page 55) point to suggestions that 5 imputations may often suffice, but caution this may only apply to effect estimation, while p-value estimation may require more than 100 imputations. Neither of these texts discuss choice of number of imputations for prediction applications.

For prediction, awareness of the need to explicitly address the issue of choosing a sufficient number of imputations appears largely absent. Current perceptions persist that small numbers of imputations would suffice as compared to effect estimation or hypothesis testing applications. For example, recent literature review by Tsvetanova et al. [36] of UK prediction models, documents that single imputation is typically used for prediction studies - when imputations are used - and multiple imputation rarely applied. A similar literature study by Hunt et al. [16] reaches similar conclusions for general use of imputations. A key finding of our paper is that when using imputations in prediction, multiple imputation is required with the numbers of imputations substantially above those of current practice. For optimal prediction, imputation numbers between 100 and 1000 with the prediction-averaging approach may be required.

We documented exceptionally poor prediction performance with single-imputation (where all methods discussed in this paper coincide), as the Monte Carlo replication variation of predictions ranges between 15 and 20% on the probability scale, making it unsuitable for practical application. It is again important to stress that these results apply - not only for prediction on future partially observed records - but even for fully observed records. We are not aware of any previous formal assessment on the performance of single imputation for prediction application with Cox regression.

An alternative option to predictive model calibration with missing values which is sometimes suggested is to restrict to complete-case based analysis. This approach does not solve the problem of missing values in future patient records however. We are not aware of any thorough study of complete-case based analysis for prediction. We carried out a simulation exercise, which indicates the approach could indeed be viable (ignoring the future missing value issue), but provided the missing values are induced from a MCAR process, with substantial (calibration) sample sizes available and the number of incomplete cases restricted. These conditions may be too restrictive for many clinical applications, especially with observational data. The description of this simulation-based assessment is too extensive to be included in this paper and will be made available in a separate report.

The predictive density, model uncertainty and survival prediction

The favourable results for prediction-averaging are no coincidence, but due to the close connection with the so-called predictive density. Recognition of the fundamental importance of the predictive density to prediction application goes back decades and described in texts by Seber [33] (page 292), but also Aitchison et al. [1] and even earlier by Jeffreys, H. [18] and Zellner, A. and Chetty, V. K. [42], among others. Use of the predictive density becomes especially important in problems with substantial model uncertainty. Differences between-models fitted on distinct imputations are a prime example. Extensive discussion of the problems which are induced for prediction can be found in another more established example application in variable selection [6, 15]. As discussed by Bernardo, J.M. and Smith, A.F.M. [4] (section B.4.3.), in such situations, we can no longer plug-in an estimator of the model parameters into the base model assumption, as we would do by using the Rubin’s rule pooled model. Instead, we must apply Rubin’s rules directly to the predictions (distributions), as we have done in this paper, which leads to prediction-averaging and use of the predictive density.

Another less well recognized consequence of applying Rubin’s rules to derive a pooled model for prediction is that predictions may suffer from Jensen’s inequality Jensen, J. [19]. The problem is avoided by directly averaging predictions instead. This problem applies especially to all non-linear models and may explain the discrepancy in Monte Carlo variation between prediction-averaging and the pooled model, especially when increasing the number of imputations as this implies an ever-larger number of models to be combined. Our study was restricted to use of the Cox model, but conclusions should carry over broadly for other survival models too, specifically accelerated failure time modeling, due to the highly non-linear nature of survival model prediction. Indeed, we would expect most of the discussion to apply to any prediction scenario involving a non-linear outcome dependence model with imputations.

Fundamental assumptions, Cox regression and imputation method choice

Basis assumptions required are that the model governing the dependence of lifetime outcomes on predictors is correctly specified and remains valid in prediction. In addition, the process inducing missing values must be MCAR or MAR.

We have used the Nelson-Aalen cumulative hazard estimator and censoring indicator to replace the observed survival outcomes [8, 40] to implement imputations for the Cox model. Our justifications for using this method are that first, it can be easily implemented with standard imputation software (such as the ‘mice’ package) and secondly its computational efficiency. Concerns with this method have been raised about (1) the so-called ‘non-congeniality’ of imputations [2] as well as (2) the approximation of the baseline hazard by the marginal hazard. Concerning issue (1), published evidence in favour of using congenial imputation methods - at time of writing - is in terms of the potential bias on effect estimation but not with respect to prediction. As demonstrated in this paper, prediction implementations based on the predictive density (averaging) are preferable with imputation and not those based on pooled effect estimates. Bonneville et al. [5] demonstrate that the bias effects in prediction due to non-congenial imputation are minor - if present at all. Alternative congenial imputation strategies, such as ‘SMC-FCS’ incur a considerable computational overhead and longer computing times. They are also sensitive to convergence problems and may occasionally break down. As the extensive simulation and data-based investigation in this paper is already computationally demanding, we anticipated such approach would have added a computational burden difficult to overcome. Furthermore, congeniality itself does not ensure compatibility of the model assumptions with the underlying population distribution for which we wish to derive predictions. We anticipate that the latter problem is more important to the viability of practical prediction, with the congeniality problem a relatively minor issue in prediction generally. On the second issue (2), White and Royston [40] for standard survival analysis and Bonneville et al. [5] for competing risks outcomes showed in simulation studies that the performance of an imputation procedure based on iteratively updating the Breslow estimate of the baseline hazard performed equally well as the simplified procedure based on the Nelson-Aalen estimate.

Towards practical clinical application

We studied the scenario where the basic model is fully specified, including the set of predictors, without further uncertainty on the model definition. The CRT data example is of this form, as the original clinical application reported model fitting and assessment conditional on a fixed set of covariates. Of all the potential extensions to address model (predictor) uncertainty, inclusion of shrinkage-based estimation would likely be the most interesting and promising in this context and combine well with the methods discussed in this paper. This is left for future research.

The primary objective of this paper has been to investigate the two basic approaches available for using imputations in the context of missing values in risk prediction. There is a non-negligible computation cost for either method. For the prediction-averaging approach, a parallel computing implementation would be highly suited. We did not investigate such implementation, but found predictions could be generated for a single patient with the prediction-averaging approach, based on 1000 imputations in under a minute on a desktop PC. We found that prediction-averaging performs well generally and is preferable to use of single pooled models. Even with computational performance from basic implementations, it can hence be applied to improve the clinical prediction applications where patients have missing predictor values.

Conclusions

Calibrating a prediction rule on partially observed predictor data and then apply it for the prediction of new observations - when their predictors may themselves also contain missing values - is a complex statistical problem. This paper investigated use of imputations for missing values in the predictors of both the calibration data records, as well as those occurring in future observations to which a prediction rule is applied. We have studied two basic approaches to the combination of multiple imputations for prediction in applications of the Cox-model for lifetime outcomes. The first approach is suggested by theory and directly averages predictions obtained from multiple Cox regression models, each of which is fitted on a distinct imputed calibration data set. At time of writing, the second approach is more conventional and first averages the regression coefficients and baseline hazards across multiple Cox regression models fit on the imputed data, to define a pooled model which is then used for prediction of future records ([8], section 8.4).

Summary of key results and recommendations

We found that methods perform comparably with respect to accuracy. However, when replicating the predictions for the same individuals with a new set of imputations and for the same number of imputations K, the prediction-averaging methodology is superior because of much lower between-replication prediction variation. For prediction-averaging the absolute between-replication variation of predictions approaches 0 when K=1000 imputations are used. When using pooling-based approaches for prediction however, variation of predictions between-replications of the imputation process becomes resistant to further reduction when we increase the numbers of imputations K, once K approaches 100. This suggests it may not be useful to generate more than 100 imputations when Rubin’s rule-pooled models are used for prediction and that the method is suboptimal as compared to prediction-averaging. Interestingly, these results apply to both prediction with future partially observed records, but also when predicting on fully observed records and thus, results carry over to the setting investigated by Bonneville et al. [5] as well (although prediction on fully observed records is less variable as compared to prediction based on partially observed records, as would be expected). Use of the pooled model is discussed as a default approach in the update edition of Carpenter and Kenward’s book (2022, section 8.4). That method should however be treated with caution and predictive averaging used by preference, even in the context as discussed in the book.

Authors’ contributions

BM is responsible for all methodology, software implementation, data analysis and writing. LdW contributed to writing, methodological setup and critical review. LvdB compiled graphical summaries of the simulation. NM and ED provided the data and gave advice on model setup.

Funding

The authors declare that this research received no funding.

Data availability

Software (R package) and analysis protocols can be accesses via Github (https://github.com/BartJAMertens), which also includes a copy of the CLL data. The CRT data is not publicly available but researchers can contact the corresponding author with reasonable request.

Declarations

Ethics approval and consent for publication

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.