Prediction models need appropriate internal, internal-external, and external validation

Recent Editorials in this journal stressed the classical paradigm in clinical epidemiology of insisting on test-retest evaluations for studies on diagnosis and prognosis [1], and specifically prediction models [2]. Indeed, independent validation of previous research findings is a general scientific principle.

Another recent debate was on the interpretation of the lack of external validation studies of published novel prediction models [3][4][5]. One issue is the role that validation should have at the time of model development. Many researchers may be tempted to try to report some proof for external validity, i.e. on discrimination and calibration, in independent samples with their publication that proposes a new prediction model. Major clinical journals currently seem to appreciate such reporting. Another issue is whether external validation should be performed by different authors than those involved in the development of the prediction model [3][6]. We would like to comment on these and related key issues in the scientific basis of prediction modeling.

The recent review confirms that model development studies are often relatively small for the complex challenges posed by specifying the form of a prediction model (which predictors to include) and the estimation of predictor effects (overfit with standard estimation methods) [3]. The median sample size was 445 subjects. The number of events is the limiting factor in this type of research and may be far too low for reliable modeling [4]. In such small samples, internal validation is essential, and apparent performance estimates are severely optimistic (Figure 1). Bootstrapping is the preferred approach for validation of prediction models [7] [8] [9]. A bootstrap procedure should include all modeling steps for an honest assessment of model performance [10]. Specifically any model selection steps, such as variable selection, need to be repeated per bootstrap sample if used.

Figure 1.

Schematic representation of apparent, split-sample, and bootstrap validation. Suppose we have a development sample of 1,000 subjects (numbered 1,2,3,..1000). Apparent validation assesses performance of a model estimated in these 1000 subjects on the sample. Split-sample validation may consider 50% for model development, and 50% for validation. Bootstrapping involves sampling with replacement (e.g., subject number 1 is drawn twice, number 2 is out, etcetera), with validation of the model developed in the bootstrap sample (Sample*) in the original sample.

We recently confirmed that a split sample approach with 50% held out leads to models with a suboptimal performance, i.e. models with unstable and on average the same performance as obtained with half the sample size [11]. We hence strongly advise against random split sample approaches in small development samples. Split sample approaches can be used in very large samples, but again we advise against this practice, since overfitting is no issue if sample size is so large that a split sample procedure can be performed. Split sample approaches only work when not needed.

More relevant are attempts to obtain impressions of external validity: do model predictions hold true in different settings, e.g. in subjects from other centers, or subjects seen more recently? Here a non-random split can often be made in the development sample e.g. by year of diagnosis. For example we might validate a model on the most recent 1/3 of the sample held out from model development. Since the split is in time, this would qualify as a temporal external validation [6]. The disadvantages of a random split sample approach unfortunately equally hold here: a poorer model is developed (since on smaller sample size than the full development sample), and the validation findings are unstable (since based on a small sample size) [9].

We make 2 propositions for validation at the time of prediction model development (Figure 2). First, we recommend an “internal- external” validation procedure. In the context of individual patient data meta-analysis (IPD-MA), internal-external cross-validation has been used to show external validity of a prediction model [12] [13]. In a MA context, the natural unit for splitting is by study. Every study is left out once, for validation of a model based on the remaining studies. The final model is based on the pooled data set, which we label an ‘internally-externally validated model’. In a more general sense, splits by hospital in a multicenter study or by calendar time are attractive options when developing prediction models, with the final model developed on all available data [14].

Figure 2.

Schematic representation of internal-external cross-validation and external validation. Suppose we have 4 centers (a – d) in our development sample. We may leave 1 center out at a time to cross-validate a model developed in the other centers. One such validation is illustrated: for a model based on 750 subjects from centers b, c, and d, on 250 subjects from center a. Since the split is not at random, this qualifies as external validation. The final model is based on all data, and can subsequently be validated externally when new data become available for analysis after publication of the model. This approach is best when there is a large number of small centers.

Second, we may consider more direct tests for heterogeneity in predictor effects by place or time. In IPD-MA with many studies, random effects models can be used to quantify the heterogeneity in predictor effects over studies [15]. The amount of heterogeneity is essential in claims on generalizability of predictions from the proposed model on the pooled data. In the same spirit we may assess heterogeneity over centers in a multicenter study where a prediction model is developed. When few studies are available, testing interaction terms such as “predictor * study” provide valuable insights, and more direct than a global impression of external validity. For temporal validity, a more direct test is provided by the interaction of “predictor * calender time”. In addition to predictor effects, such tests can also assess heterogeneity in baseline risk (main effect of “study”, after adjusting for risk), and in overall risk associations as summarized in the linear predictor (e.g. the “prognostic index * study” interaction). So again, using the maximal sample size in modeling is preferred.

Finally, fully independent external validation with data not available at the time of prediction model development can be important (Figure 2). Such validation should be a test of generalizability. If the external data set is very similar to the development data set, the assessment is for reproducibility rather than for transportability [6]. The similarity of validation to development sets hence is essential for interpretation of an external validation study; either by comparing descriptive data (“Table 1”) or a simple statistical model to predict membership of the development or validation data set [16].

In sum, we recommend that internal validation should always be attempted for any proposed prediction model, with bootstrapping being preferred (Figure 1). Many failed external validations could have been foreseen by rigorous internal validation, saving time and resources [4]. With respect to external validation at the time of model development, we recommend internal- external validation procedures and direct tests for heterogeneity of predictor effects rather than keeping parts of the data out (Figure 2). Such assessments at model development may temper overoptimistic expectations of prediction model performance in independent data. For external validation after the developed model was published, the issue may not so much be that different authors perform the analysis [5]; indeed, no difference was noted between validations performed by overlapping or different authors [3], although selective reporting and publication bias cannot be ruled out. What matters more is the similarity between settings [6][16], and the quality of the validation analysis [7] [8] [17], which may be positively influenced by reporting guidelines such as the recently proposed TRIPOD guidelines for prediction models [18] [19].