Letter to the editor

Li and others (2013) wrote a recent paper which proposes to generalize the Net Reclassification Index (NRI) and the Integrated Discrimination Index (IDI), which have become popular metrics for assessing the value of adding a marker to a model for predicting risk of disease (Pencina and others (2008), Pencina and others (2011)). The authors intend to generalize these measures to the setting where the disease outcome has multiple categories. Asymptotic distribution theory is provided for estimators and evaluated in finite sample simulations. The generalized measures are used to select variables to be added to a risk model. We have several comments and concerns with the methods proposed in the paper.

The measure S defined in expression 2.5 is not a generalization of the NRI statistic, and is not a valid measure for comparing models. For each outcome category, the measure relates the probability of being incorrectly classified by model 1 and correctly classified by model 2; this is the probability of a “good” reclassification. However, in the original NRI this is offset by the probability of a “bad” reclassification: the probability of being correctly classified by model 1 and incorrectly classified by model 2 (Pencina and others (2008)). By only counting reclassifications in one direction, S will be positive as long as model 2 moves subjects between outcome classes. For example, if model 2 is simply model 1 with the addition of a marker unassociated with outcome, S will be positive despite the fact that there is no true difference in performance between the two models.

The measure R defined in expression 2.7 is intended to generalize the IDI statistic to multicategory disease outcomes. This could be rewritten as the (weighted) difference in the mean risk difference between models 1 and 2, where the mean risk difference for a given outcome category m and model M is the difference in mean risk of outcome m between subjects with and without the outcome, i.e. – E{p_m(M)|Y ≠m} (Pencina and others (2011)). This formulation is a bit more intuitive and in the spirit of the original IDI.

Estimators for R and S are proposed that entail plugging in model-based estimates of the probabilities of disease outcomes and estimating model performance empirically. However, as noted by the authors, the asymptotic distribution theory for these estimators does not take into account the variability in the outcome probability estimates, and therefore is of limited use for data analysis. Confidence intervals will not be expected to achieve the correct coverage. As the authors point out, a resampling approach such as the bootstrap would remedy this particular problem.

The asymptotic standard errors are also not useful for hypothesis testing, since, as mentioned by the authors, the asymptotics do not hold under the null of no difference in performance between the two models. The p-values reported in Section 4.2 are presumably based on “Z-statistics” calculated using these standard errors and therefore are not valid.

Finally, the lack of asymptotic normality for the estimators under the null cannot be remedied by resampling, as demonstrated by [Kerr and others (2011), submitted], despite the suggestion otherwise by [Li and others (2013, p. 7]. There is currently no valid approach for doing hypothesis testing for the NRI or IDI (Kerr and others (2011), submitted).

We applaud the authors for addressing the important problem of multicategory disease outcomes, and for further development of the NRI and IDI measures. However, it is our opinion that more methodological development and evaluation is needed before these measures can be advocated for clinical application.