Dear Editor,
With interest we read the article by Raney et al. [1] in which they developed a prediction rule to identify patients with neck pain who are likely to improve with cervical traction. We agree with the authors that external validation is needed before the rule can be implemented in routine clinical practice. However, we doubt that this rule will be valid in an external population, because some analytical choices made by the authors probably hamper external validity.
First, they did not apply the rule of thumb of at least ten events per variable [2], which introduces bias into the tests of significance. The exact number of variables is not mentioned, but exceeds the advised amount of three variables (30 events) by far. Second, continuous variables were dichotomized based on their optimal cutoff point. This inflates the type I error rate and may result in wrongful identification of a variable as prognostically relevant [3]. Furthermore, categorisation of continuous variables before entering them into a stepwise logistic regression selection procedure may also result in the wrong predictors in the final model [4], and a poorer performance of the model [4, 5].
These analytical choices finally lead to the prediction rule through a stepwise logistic regression procedure, of which the selection criteria are not mentioned. The characteristics of the prediction rule are presented in Table 3 and 4 [1]. However, we would like to point out that Table 4 contains several incorrect values (e.g. sensitivity = 0.83, instead of 0.30, and specificity = 0.42, instead of 0.97, if ≥2 predictors are present) [1].
In conclusion, we think that this prediction rule most likely only applies to the development population and will lack external validity.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
References
- 1.Raney NH, Petersen EJ, Smith TA, Cowan JE, Rendeiro DG, Deyle GD, Childs JD. Development of a clinical prediction rule to identify patients with neck pain likely to benefit from cervical traction and exercise. Eur Spine J. 2009;18:382–391. doi: 10.1007/s00586-008-0859-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49:1373–1379. doi: 10.1016/S0895-4356(96)00236-3. [DOI] [PubMed] [Google Scholar]
- 3.Altman DG, Lausen B, Sauerbrei W, Schumacher M. Dangers of using “optimal” cutpoints in the evaluation of prognostic factors. J Natl Cancer Inst. 1994;86:829–835. doi: 10.1093/jnci/86.11.829. [DOI] [PubMed] [Google Scholar]
- 4.Schellingerhout JM, Heymans MW, de Vet HC, Koes BW, Verhagen AP. Categorizing continuous variables resulted in different predictors in a prognostic model for nonspecific neck pain. J Clin Epidemiol. 2009;62:868–874. doi: 10.1016/j.jclinepi.2008.10.010. [DOI] [PubMed] [Google Scholar]
- 5.Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25:127–141. doi: 10.1002/sim.2331. [DOI] [PubMed] [Google Scholar]