“Garbage in, garbage out”—the importance of detailing methodological reasoning in orthopaedic meta-analysis

Dear Editor,

We thank Mr. Khan for his letter regarding our recent meta-analysis “Minimally invasive versus conventional exposure for total hip arthroplasty: a systematic review and meta-analysis of clinical and radiological outcomes” [1]. We would like to take this opportunity to clarify the points he has raised to defend the methodological approaches taken in our meta-analysis.

Firstly, Mr. Khan rightly identified that we pooled all minimally invasive surgery approaches in this review. This therefore included mini-anterior, mini-posterior, and mini-anterolateral approaches, as documented in Table 2 [1]. As stated in the Introduction, the purpose of this review was to determine whether there was a difference in outcome between conventional and minimally invasive procedures. Accordingly, it was essential to investigate whether there was a difference in treatment effect between studies, irrespective of the surgical approach adopted. Whilst there may, or may not be, a difference between the different specific approaches, it was not the remit of the meta-analysis to answer that question. To highlight this point, we also acknowledged in our Discussion that further study is indicated to determine the clinical importance of different minimally invasive approaches.

The issue of inclusion of non-randomised controlled trials in meta-analyses is a controversial topic. Meta-analyses and systematic reviews which are solely based on randomised controlled trials may be able to make more confident inferences on the results as their data is based on higher level evidence, but the quality of the study will influence the effect size [2]. By including all other study designs, the conclusions drawn from this analysis represent the evidence-base as a whole. Accordingly, this ‘best-evidence synthesis’ can then not be accused of selection or allocation biases [3, 4]. Furthermore, as stated in our Introduction, a previous meta-analysis solely assessing the findings from randomised controlled trials had been undertaken on this topic [5]. Interestingly, both our paper and the meta-analysis of Cheng et al. [5] conclude that there was little difference in the clinical or radiological outcomes following MIS compared to standard exposure total hip arthroplasty. Therefore by consulting these complementary meta-analyses of both randomised and non-randomised controlled trials, orthopaedic surgeons can now base their clinical decision-making of minimally invasive surgery on a stronger research synthesis evidence-base. This would not have been possible if we had simply re-reported the findings of randomised controlled trials after the work of Cheng et al. [5].

Finally, Mr. Khan correctly identified that two studies reported data in four publications [6–9]. He did not however appreciate that we also identified this during our analysis, which is irksome. These publications presented some of the same, but also different outcomes, in each paper. Since it is good practice to include all pertinent and appropriate data in an analysis to avoid selective reporting of outcomes, data was extracted from each paper, whilst the findings from each of these cohorts was analysed only once for each outcome, therefore not duplicating the results.

We agree with Mr. Khan that meta-analyses are an important means to assess orthopaedic practice, and that data should be handled with care to prevent a “garbage in, garbage out” scenario becoming the basis of clinical decision-making. However, Mr. Khan has failed to acknowledge that the design was compliant with internationally agreed PRISMA guidelines [10], with the decisions made during the development and analysis of this meta-analysis respecting both population and methodological heterogeneity, in order to answer this important clinical question in hip surgery. Ratio et consilium propriae ducis artes.