Validating prediction models

Lokker et al presented an interesting model to predict citation counts for clinical articles.¹ This topic is so important that the paper will probably attract many citations. We want to clarify some of the nomenclature of validation of prediction models, to avoid confusion in future reporting.

The authors randomly divided 1274 articles into a derivation data set of 757 articles for development of a prediction model and a validation dataset for testing of 504 articles, after exclusion of outliers with >150 citations. This procedure is an example of a split sample approach, but the authors refer to it as cross validation. Cross validation would mean that we develop a model in the first part of the data and test it in the second part, and then repeat the procedure with development in the second part and testing in the first.

The authors report that explained variation (R2) decreased from 0.60 at development to 0.56 at validation, and refer to this decrease as shrinkage. Shrinkage is not an appropriate term for this decrease; a better label is optimism.^{2 3} Optimism is the phenomenon that prediction models tend to perform more poorly in new data than in the data where the model was developed; it occurs especially when many predictors are considered in relatively small datasets.⁴

Ironically, a need for shrinkage is well illustrated in figure 2, where the residuals are generally positive for low predictions (which were often too low), and generally negative for high predictions (which were often too high).¹ Shrinkage should be applied to the regression coefficients for more reliable predictions.^{2 4 5}

How valid is this model to predict citations? Firstly, the authors did not shrink regression coefficients, which implies that high predictions will be too high and low predictions too low for articles fulfilling the inclusion criteria. Secondly, for a future article we cannot know beforehand whether the article is an outlier, i.e. having more than >150 citations. Exclusion of outliers at validation is artificial and should not have been done; it has inflated the R2 of the model. As always with prediction models, future validation is required and may reveal disappointing performance.