Letter to the Editor: Efficacy and Degree of Bias in Knee Injury Prevention Studies: A Systematic Review of RCTs

To the Editor:

As an avid soccer player, I am always interested to learn what I can do to prevent knee injuries from occurring. It was a bit dismaying, then, to find the negative results of the Grimm et al. [5] study. Of course, the statistician in me recognizes that a lack of statistical significance is not the same as “no difference in treatment benefit”, so perhaps there is still hope, and maybe I will continue to train with proprioceptive balance and plyometrics. However, there is also a more subtle methodologic point that needs to be made. Grimm et al. reported that (1) all 10 studies were properly randomized and had allocation concealment; (2) only one was masked; and (3) future trials should comply with CONSORT. It is unlikely that any of the studies were properly randomized, or had the benefits of allocation concealment. Clearly, this statement requires support.

I distinguish the effort to mask from the success of the endeavor [1]. Given the nature of the treatments involved, it is unlikely that even the one study that claimed masking could have been truly masked, although de facto masking [2] is always an option, and probably should have been used to at least conceal which treatment was the active one. Therefore with or without the claim of masking, none of the studies can be taken as perfectly masked, and any unmasking at all, coupled with any form of restricted randomization (such as permuted blocks), precludes the possibility of allocation concealment, because knowledge of even some of the prior allocations can allow for prediction of future ones [1, 4]. Unless the trials used unrestricted randomization, which almost no trials use in practice, there was no allocation concealment. This does not imply that the randomization was improper, as there are also drawbacks to using unrestricted randomization [1, 4]. However, the use of an excessively restricted procedure, such as permuted blocks, is inappropriate when masking is uncertain. Grimm et al. should have displayed in Table 2 the specific randomization procedure used in each trial, but without this, we cannot assume that all 10 research teams knew to use minimally restrictive procedures such as the (paradoxically named, given its introduction here) maximal procedure [1, 4].

We now come full circle to what this means for soccer players. Methodologic issues are not confined to the purview of the statistician, because they have real implications for the consumers of the research. When the trials cannot be trusted to provide valid results, the results cannot be trusted, and we have no real basis for forming any conclusion, other than that future studies need to be better in conduct and reporting. Grimm et al. stated that future studies should comply with CONSORT. This would be a good start, necessary but not sufficient, as even CONSORT-compliant trials can be fatally flawed. No check list can anticipate every possible way a trial may be biased, but some are better than others, and the check list used as a first pass to check for trial quality and integrity must evolve to keep up with changes in trial design, as these changes introduce opportunities for new biases to arise [3]. Researchers and statisticians must remain vigilant to new biases, and always keep an eye open to the need to augment CONSORT or any other check list; the soccer players and other athletes must recognize flawed research for what it is, take it for what it is worth, and, until more definitive evidence can be produced, keep on kicking.