Answer to the Letter to the Editor of Mingbo Tian et al. concerning “Body mass index and risk of surgical site infection following spine surgery: a meta-analysis” by D.Y. Abdallah, M.M. Jadaan and J.P. McCabe; Eur Spine J, doi:10.1007/s00586-013-2890-6

We would like to respond to the Letter to the Editor concerning our meta-analysis [1].

For the electronic search, we used PubMed, which provides a search interface to Medline, and Embase. These two search engines complement each other [2], and, in addition to CENTRAL, contain the largest number of studies according to the Cochrane Handbook of Systematic Reviews of interventions [3]. We did not search CENTRAL, as the focus of our research was on observational studies. The search, supplemented by screening of reference lists, identified 34 studies relevant to our research question, which is not a small number. However, it was not appropriate to pool results from all 34 studies. It is not uncommon to pool results from <10 studies. Between 50 and 75 % of meta-analyses contain <10 studies, including Cochrane reviews [4, 5]. While we know that a single search engine does not yield all pertinent studies, we are not aware of guidelines that specify at least three databases to be searched. Nevertheless, during literature search we used Google Scholar to identify additional studies. Despite the large number of results, the search did not yield relevant studies beyond those already identified. We did not formally include Google Scholar, as the search engine has low specificity and does not utilise controlled vocabulary relevant to the research question [6].

Ideally, to circumvent the problem of publication bias, a meta-analysis should include unpublished literature and non-English language studies, but this is not always feasible and is a limitation of many published meta-analyses, not only ours. We conducted a formal assessment of publication bias using more than one method and did not rely on visual inspection of funnel plots. Applying these methods when the number of studies is small is not wrong, but has limited power [4]. We reported the results of these tests and were careful in our interpretations by acknowledging the low power of Egger’s test. We never claimed that there was no publication bias, and we stated that “we were unable to reliably assess the presence of publication bias due to the small number of studies included” [1]. We would also like to draw the attention of the writers of the Letter that we explicitly discussed English language bias and the potential over-estimation of results in the discussion section.

In relation to assessment of study quality, first, we assigned the ‘level of evidence’ to each study, which is a “hierarchical rating system for classifying study quality” [7]. It is a well-established scoring system and is used by several journals. Second, to minimise bias, we used several inclusion criteria, and listed all excluded studies as well as the reasons for their exclusion. We reported all information that would potentially introduce bias, such as study design, categorisation of BMI, and adjustment for confounders. No subjectivity was involved in extracting the data. Therefore, data extraction was reviewed by the second author to ensure correctness rather than any other degree of agreement between reviewers. The purpose of the study was to provide a quantitative estimate free from subjectivity, and quality scores were not used when pooling results as these have been shown to produce bias [8]. Further, methodological reviews of most of the studies included in the meta-analysis have been done twice previously [9, 10].

It is common practice for researchers to rely on the test for heterogeneity for choosing between fixed-effect and random-effects models. However, this approach has been strongly discouraged, and  it is recommended that the selection of a model should be based on understanding of underlying differences between studies [11]. Although we used several inclusion criteria, other sources of heterogeneity between studies would be expected, such as patient population, hospital setting, and surgical procedures performed. Thus, it would be wrong to assume that all studies have the same underlying effect; rather it is the mean effect we were looking for. The lack of evidence for heterogeneity could be due to the low power of the heterogeneity test [11, 12]. A recent re-analysis of thousands of Cochrane meta-analyses highlighted evidence of previously undetected heterogeneity and consequent bias in estimates [12]. In addition, the random-effects model yields wider confidence intervals, which makes our estimates more conservative, as well as more generalizable [11]. Yet, to avoid misleading generalisations, our final recommendation was for future prospective studies to confirm results.