Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Dec 1.
Published in final edited form as: Br J Sports Med. 2019 Jul 9;53(23):1444–1446. doi: 10.1136/bjsports-2019-100906

Keep Calm and Carry On Testing—A Substantive Reanalysis and Critique of “What is the Evidence for and Validity of Return-to-Sport Testing after Anterior Cruciate Ligament Reconstruction Surgery? A Systematic Review and Meta-Analysis”

Jacob J Capin 1,2, Lynn Snyder-Mackler 1,2,3,4, May Arna Risberg 5,6, Hege Grindem 5,7,8
PMCID: PMC6858490  NIHMSID: NIHMS1045142  PMID: 31289039

Clinicians rely on rigorous systematic reviews to guide practice. We therefore suspect many clinicians will note the results of the 2019 systematic review and meta-analysis by Webster and Hewett, “What is the Evidence for and Validity of Return-to-Sport Testing after Anterior Cruciate Ligament Reconstruction Surgery? A Systematic Review and Meta-Analysis.”[1] We agree that it is important to evaluate the association between return-to-sport (RTS) test batteries and outcomes after anterior cruciate ligament (ACL) reconstruction. The third review question in Webster and Hewett (2019) is particularly pertinent: “Is passing RTS test batteries associated with reduced rates of subsequent knee injury (all knee injuries and ACL injury)?”[1] We are authors of several of the original data papers cited in the systematic review and we are concerned about the study methodology and its conclusions. We highlight major problems with including 2 studies and present revised analyses that demonstrate the impact these studies had on the conclusions.

Methodological concerns

First, we question the validity of pooling studies with substantial clinical and methodological diversity[2]. The meta-analysis combined studies where only some athletes returned to sport[3,4] and studies where all, or mostly all, returned to sport[5–7]; studies with skeletally immature patients[5] and studies with elite athletes[6]; and studies where substantially different RTS test batteries were used. Our second concern is that Webster and Hewett[1] did not assess risk of bias, a fundamental precept of systematic review methodology clearly stated in the PRISMA reporting guideline.[8–10] Assessment of study quality (as performed by Webster and Hewett[1]) does not quantify risk of bias.[11] A risk of bias assessment identifies factors within studies that can skew results, and these factors must be considered carefully in the decision to pool data and in the conclusion. Important bias domains for review questions 2 and 3 include (i) study participation, (ii) study attrition, (iii) methods used to ascertain RTS pass status, (iv) outcome (subsequent injury), (v) confounding, and (vi) statistical analysis and reporting.[12]

How 2 studies designed to assess different research constructs impacted contralateral ACL injury results

We believe that Webster and Hewett’s report of 235% greater risk of contralateral ACL injury among those who passed RTS criteria[1] is an artefact of including 2 studies that were not designed to answer the same research question as the 3 other studies in the systematic review. In the studies by Sousa et al.[3] and Wellsandt et al.[4], the RTS test results were used to determine when athletes were cleared to return to sport. Early return to sport is among the strongest risk factors for reinjury[7,13–15], and delaying return to sport in those who initially fail RTS testing is likely to protect them from reinjury. In these 2 studies[3,4], reinjuries were reported irrespective of whether the patients returned to sport after RTS testing. The 3 other studies[5–7] in the systematic review represented athletes who returned to sport even if they failed RTS tests (all patients returned to sport in the studies by Kyritsis et al.[6] and Grindem et al.[7], and 39 of 42 patients returned to sport in Graziano et al.[5]). These patients returned to sport either because of nonadherence to the protocol or because the RTS test battery was not used for sports clearance in the sample. We contend that the clinically relevant question of whether patients should pass RTS tests prior to return to sport cannot be informed by studies where return to sport was delayed if the patient failed the RTS tests[3,4]. Pooling these studies[3,4] in a meta-analysis is therefore inappropriate.

In the study by Sousa et al., confounding may also play a large role and is clearly highlighted in the paper’s conclusion[3]. Patients who passed RTS criteria in their study were younger, had higher preinjury and follow-up activity levels, and returned to sport earlier.[3] Young age,[16–23] high activity levels,[17,18,22] and early return to sport[7,13–15] are very strong risk factors for second ACL injury; we expect higher injury rates in the ‘passed RTS criteria’ group than in the older ‘failed RTS criteria’ group who participated in less knee-demanding sports and had delayed return to sport. The majority (9/16) of the contralateral ACL injuries reported in Webster and Hewett[1] were derived from the Sousa et al. study[3], which was the only individual study that showed a higher rate of contralateral ACL injuries in the group that passed RTS tests. This single study accounted for 77% of the weighting for the contralateral ACL injury meta-analysis and heavily influenced the conclusions drawn by Webster and Hewett.

Four of the remaining 7 contralateral ACL injuries among the athletes who passed RTS testing in the meta-analysis were derived from the study by Wellsandt et al.[4] This study was designed to evaluate estimated pre-injury capacity (EPIC) levels as alternatives to limb symmetry indexes, not the association between the established RTS criteria and second ACL injuries.[4] Patients in this study[4] only received RTS clearance after they passed RTS testing,[24] either 6 months after ACLR (the test point reported by Wellsandt et al.[4]) or at a later time-point after completing additional rehabilitation. Having no contralateral ACL injuries among those who ‘failed’ the RTS test battery 6 months after surgery was not surprising as these patients did not return to sport at that time. Instead, athletes who failed the RTS tests continued rehabilitation and were scheduled for a new test at a later time. This prevents a meaningful comparison because no one in this specific sample returned to sport until they passed RTS criteria. As authors, we acknowledge that this information is not explicitly stated in the paper. Had we known the data would be used for another purpose than the original paper, we would have clarified to avoid this misinterpretation.

Revised analysis and interpretation of results

Here we demonstrate how conclusions change when the two studies [3,4] with critically different study designs are excluded from the meta-analysis.

  1. In the meta-analysis for any knee injury, Sousa et al.[3] and Wellsandt et al.[4] were excluded by Webster and Hewett, so their analysis remains unchanged. Passing RTS test batteries is associated with 72% lower risk of further knee injury (i.e., ACL injury and other knee injuries, 95% confidence interval [CI]: 6–96 % lower risk, p = .09).[1]

  2. Webster and Hewett[1] reported no significant association (p = .68) between successfully passing the RTS test and having a lower rate of subsequent ACL injury (i.e., graft rupture and/or contralateral ACL injury). However, after excluding the studies by Sousa et al.[3] and Wellsandt et al.[4], those athletes who passed the RTS criteria had 75% lower odds (95% CI: 46% to 88% lower odds) of any ACL injury than those who failed (p<0.01, Figure 1).

  3. Webster and Hewett[1] found that those who passed RTS criteria had a 60% lower risk for ACL graft rupture (95% CI: 31% − 77%, p = .003). By excluding the inappropriate studies,[3,4] and pooling remaining data, there is 78% lower odds of graft rupture in those who passed RTS criteria compared to those who failed (95% CI: 52% to 90% lower odds, p<0.01, Figure 2).

  4. By excluding the studies by Sousa et al. and Wellsandt et al.,[3,4] there are only 4 contralateral ACL ruptures left for analysis (among those who passed or failed RTS criteria). These numbers are too low to say whether or not passing RTS criteria influences risk of contralateral ACL rupture.

Figure 1.

Figure 1.

Open in a new tab

Re-analysis showing the risk for any second ACL injury among those who pass versus fail RTS criteria. Abbreviations: RTS, Return-to-Sport; M-H, Mantel-Haenszel; OR, odds ratio.

Figure 2.

Figure 2.

Open in a new tab

Re-analysis showing the risk for an ACL graft rupture among those who pass versus fail RTS criteria. Abbreviations: RTS, Return-to-Sport; M-H, Mantel-Haenszel; OR, odds ratio.

Conclusion

Our reanalysis omits 2 studies[3,4] with designs that addressed a different research construct than the remaining 3 and/or have a high risk of bias. We found that compared with patients who fail RTS tests prior to return to sport, athletes who pass RTS test batteries have (i) a lower risk of (any) knee reinjury, (ii) a lower risk of any second ACL injury, and (iii) a lower risk of ACL graft rupture; (iv) no conclusions regarding contralateral ACL injury risk can be drawn due to insufficient data.

More evidence is needed to refine RTS test batteries to provide greater certainty in their ability to facilitate successful RTS. Future meta-analyses should critically evaluate the study design of each potential contributing paper and include a risk of bias assessment. Meta-analyses should also consider the impact of sparse-data bias[25] and avoid the Firth penalization when events are 0, as this method can change the direction of the reported association[26,27].

In future studies, authors should report and/or control for sport level and athletic exposure. Researchers should aim to rigorously evaluate (i) which tests can help clinicians help athletes return to play successfully, (ii) the optimal values for cut-off scores, and (iii) alternatives to limb symmetry indexes. Importantly, all studies to date are observational, and there is a need for interventional designs (e.g., pragmatic trials or site randomization). Such studies will improve clinicians’ understanding of RTS test batteries and, with appropriate implementation, should reduce secondary knee and ACL injuries.

Clinicians should not fear an increased risk of contralateral ACL injuries on the basis of the current literature, but continue to use RTS test batteries (and appropriate time-frames) to support RTS decision-making.

Acknowledgements:

Thank you to biostatistician, Dr. Mohammad Ali Mansournia, MD, MPH, PhD, for his critical review of our work, and for his helpful suggestions.

Funding: National Institutes of Child Health & Human Development (R37 HD037985).

Competing interests: JJC has received grants from the Foundation for Physical Therapy Research—Promotion of Doctoral Studies (PODS) Level I and II Scholarships and the National Institutes of Child Health & Human Development (F30 HD096830). LSM has received grants from the National Institutes of Health (R37 HD037985, R01 AR048212, T32 HD007490, R44 HD068054, U54 GM104941 and P30 GM103333). MAR and HG have received grant funding from the National Institutes of Child Health & Human Development (R37 HD037985). HG is a BJSM senior associate editor, works as a sports physiotherapist, occasionally receives minor speaking fees for presentations, and holds grants from the International Olympic Committee, the Norwegian fund for post-graduate training in Physiotherapy, and the Swedish Research Council for Sport Science.

List of Abbreviations:

ACL:

anterior cruciate ligament

RTS:

Return to sport

Footnotes

Ethics approval and consent to participate: Not applicable

Consent for publication: Not applicable

Availability of data and material: All data generated or analyzed in this manuscript are included in the systematic review and meta-analysis by Webster and Hewett [1].

References

  • 1.Webster KE, Hewett TE. What is the Evidence for and Validity of Return-to-Sport Testing after Anterior Cruciate Ligament Reconstruction Surgery? A Systematic Review and Meta-Analysis. Sport Med 2019;49:917–29. doi: 10.1007/s40279-019-01093-x [DOI] [PubMed] [Google Scholar]
  • 2.Grindem H, Mansournia MA, Øiestad BE, et al. Was it a good idea to combine the studies? Why clinicians should care about heterogeneity when making decisions based on systematic reviews. Br J Sports Med 2019;53:399–401. doi: 10.1136/bjsports-2018-099516 [DOI] [PubMed] [Google Scholar]
  • 3.Sousa PL, Krych AJ, Cates RA, et al. Return to sport: Does excellent 6-month strength and function following ACL reconstruction predict midterm outcomes? Knee Surgery, Sport Traumatol Arthrosc 2017;25:1356–63. doi: 10.1007/s00167-015-3697-2 [DOI] [PubMed] [Google Scholar]
  • 4.Wellsandt E, Failla MJ, Snyder-mackler L. Limb Symmetry Indexes Can Overestimate Knee Function After Anterior Cruciate Ligament Injury. J Orthop Sport Phys Ther 2017;47:334–8. doi: 10.2519/jospt.2017.7285 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Graziano J, Chiaia T, De Mille P, et al. Return to sport for skeletally immature athletes after ACL reconstruction: Preventing a second injury using a quality of movement assessment and quantitative measures to address modifiable risk factors. Orthop J Sport Med 2017;5:1–10. doi: 10.1177/2325967117700599 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Kyritsis P, Bahr R, Landreau P, et al. Likelihood of ACL graft rupture: not meeting six clinical discharge criteria before return to sport is associated with a four times greater risk of rupture. Br J Sports Med 2016;50:946–51. doi: 10.1136/bjsports-2015-095908 [DOI] [PubMed] [Google Scholar]
  • 7.Grindem H, Snyder-Mackler L, Moksnes H, et al. Simple decision rules can reduce reinjury risk by 84% after ACL reconstruction: the Delaware-Oslo ACL cohort study. Br J Sports Med 2016;50:804–8. doi: 10.1136/bjsports-2016-096031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 5. The Cochrane Collaboration; 2011. www.handbook.cochrane.org [Google Scholar]
  • 9.Springer Manuscript Guidelines Journals: Instructions for Authors. https://www.springer.com/authors/manuscript+guidelines?SGWID=0-40162-6-1397843-0 (accessed 28 Mar 2019).
  • 10.Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 2009;6:e1000097. doi: 10.1371/journal.pmed.1000097 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Boutron I, Ravaud P. Classification systems to improve assessment of risk of bias. J Clin Epidemiol 2012;65:236–8. doi: 10.1016/j.jclinepi.2011.09.006 [DOI] [PubMed] [Google Scholar]
  • 12.Hayden JA, van der Windt DA, Cartwright JL, et al. Assessing Bias in Studies of Prognostic Factors. Ann Intern Med 2013;158:280–6. doi: 10.7326/0003-4819-158-4-201302190-00009 [DOI] [PubMed] [Google Scholar]
  • 13.Laboute E, Savalli L, Puig P, et al. Analysis of return to competition and repeat rupture for 298 anterior cruciate ligament reconstructions with patellar or hamstring tendon autograft in sportspeople. Ann Phys Rehabil Med 2010;53:598–614. doi: 10.1016/j.rehab.2010.10.002 [DOI] [PubMed] [Google Scholar]
  • 14.Capin JJ, Khandha A, Zarzycki R, et al. Gait mechanics and second ACL rupture: Implications for delaying return-to-sport. J Orthop Res 2017;35:1894–901. doi: 10.1002/jor.23476 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Joreitz R, Lynch A, Rabuck S, et al. Patient-specific and surgery-specific factors that affect return to sport after ACL reconstruction. Int J Sports Phys Ther 2016;11:264–78. [PMC free article] [PubMed] [Google Scholar]
  • 16.Nelson IR, Chen J, Love R, et al. A comparison of revision and rerupture rates of ACL reconstruction between autografts and allografts in the skeletally immature. Knee Surgery, Sport Traumatol Arthrosc 2016;24:773–9. doi: 10.1007/s00167-016-4020-6 [DOI] [PubMed] [Google Scholar]
  • 17.Mohtadi N, Chan D, Barber R, et al. Reruptures, reinjuries, and revisions at a minimum 2-year follow-up: A randomized clinical trial comparing 3 graft types for ACL reconstruction. Clin J Sport Med 2016;26:96–107. doi: 10.1097/JSM.0000000000000209 [DOI] [PubMed] [Google Scholar]
  • 18.Webster KE, Feller JA, Leigh WB, et al. Younger patients are at increased risk for graft rupture and contralateral injury after anterior cruciate ligament reconstruction. Am J Sports Med 2014;42:641–7. doi: 10.1177/0363546513517540 [DOI] [PubMed] [Google Scholar]
  • 19.Maletis GB, Chen J, Inacio MC, et al. Age-Related Risk Factors for Revision Anterior Cruciate Ligament Reconstruction: A Cohort Study of 21,304 Patients From the Kaiser Permanente Anterior Cruciate Ligament Registry. Am J Sport Med 2016;44:331–6. doi: 10.1177/0363546515614813 [DOI] [PubMed] [Google Scholar]
  • 20.Paterno MV, Rauh MJ, Schmitt LC, et al. Incidence of second ACL injuries 2 years after primary ACL reconstruction and return to sport. Am J Sports Med 2014;42:1567–73. doi: 10.1177/0363546514530088 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Wiggins AJ, Grandhi RK, Schneider DK, et al. Risk of Secondary Injury in Younger Athletes After Anterior Cruciate Ligament Reconstruction: A Systematic Review and Meta-analysis. Am J Sport Med 2016;44:1861–76. doi: 10.1177/0363546515621554 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Kaeding CC, Pedroza a. D, Reinke EK, et al. Risk factors and predictors of subsequent ACL injury in either knee after ACL reconstruction: Prospective analysis of 2488 primary ACL reconstructions from the MOON cohort. Am J Sport Med 2015;43:1583–90. doi: 10.1177/0363546515578836 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Sanders TL, Pareek A, Hewett TE, et al. Long-term rate of graft failure after ACL reconstruction: a geographic population cohort analysis. Knee Surgery, Sport Traumatol Arthrosc 2017;25:222–8. doi: 10.1007/s00167-016-4275-y [DOI] [PubMed] [Google Scholar]
  • 24.Adams D, Logerstedt D, Hunter-Giordano A, et al. Current concepts for anterior cruciate ligament reconstruction: a criterion-based rehabilitation progression. J Orthop Sports Phys Ther 2012;42:601–14. doi: 10.2519/jospt.2012.3871 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Greenland S, Mansournia MA, Altman DG. Sparse data bias: A problem hiding in plain sight. BMJ 2016;352:i1981. doi: 10.1136/bmj.i1981 [DOI] [PubMed] [Google Scholar]
  • 26.Greenland S, Mansournia MA. Penalization, bias reduction, and default priors in logistic and related categorical and survival regressions. Stat Med 2015;34:3133–43. doi: 10.1002/sim.6537 [DOI] [PubMed] [Google Scholar]
  • 27.Mansournia MA, Geroldinger A, Greenland S, et al. Separation in Logistic Regression: Causes, Consequences, and Control. Am J Epidemiol 2018;187:864–70. doi: 10.1093/aje/kwx299 [DOI] [PubMed] [Google Scholar]

RESOURCES