Editorial: CORR® Criteria for Reporting Meta-analyses

Meta-analyses and systematic reviews provide ways of synthesizing literature to clarify issues on which there is controversy or to confirm generally held views. They are used increasingly in orthopaedic surgery and other fields: a PubMed search of orthopaedic meta-analyses using “(orthopaedic[ad] OR orthopedic[ad]) AND meta-analysis[ti]” yielded 145 articles, 104 of which were published since 2008. The quality of these reviews has varied greatly. Because of the varying quality of review articles, numerous groups have established criteria to aid standardized methods of reporting. In 1999, a working group of 30 clinicians, epidemiologists, statisticians, and other methodologists [2] proposed an approach to enhance meta-analyses based on randomized controlled trials. They suggested authors develop a flow diagram describing the methods of identifying and selecting articles, and then complete a checklist of 21 items from the individual studies that should be included in a meta-analysis. They referred to their process by the name, “Quality of Reporting of Meta-analyses” or “QUOROM.” A working group of 29 participants updated their suggestions in 2009 and renamed the standards, “Preferred Reporting Items for Systematic Reviews and Meta-analyses” or “PRISMA” [3].

Even given these guidelines, the quality of the selected studies for inclusion remains an obstacle to high-quality meta-analyses. Oxman and Guyatt [4] in 1991 were perhaps among the earliest to propose a method of judging the quality of articles using a 13-item questionnaire. In 2003 Whiting et al. [5] proposed a checklist of nine items that reflected the quality of a study to be included in a systematic review (“quality assessment of studies of diagnostic accuracy” or “QUADAS”), and in 2011 [6] this proposal was updated.

These efforts to judge the value of a study are crucial to enhancing the quality of medical evidence through meta-analyses, but do not necessarily ensure the worth of such reports. Wright et al. [8] recently proposed criteria the Journal of Bone and Joint Surgery (JBJS), American volume, will use for considering meta-analyses or systematic reviews:

“First, authors will be expected to identify all meta-analyses and systematic overviews published in the past five years on related or identical topics. JBJS will not accept meta-analyses or systematic reviews on the same topic published within five years unless the authors can demonstrate that the literature has dramatically changed. Second, meta-analyses or systematic reviews will not be accepted if the same (or largely the same) papers are used to arrive at similar conclusions. Third, for meta-analyses in which the authors use statistical methods to combine and summarize results, only summaries of randomized trials will be accepted. Moreover, only studies with sufficient homogeneity of inclusion and exclusion criteria will be considered appropriate for meta-analysis. In addition to these basic requirements, authors should familiarize themselves with reporting checklists such as the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) to improve the quality of reporting. We believe that these criteria will improve the reporting of systematic reviews and meta-analyses in JBJS and throughout the orthopaedic literature.”

We at CORR® concur on the need for such guidelines and generally will adhere to the above criteria. We realize, though, that the mere publication of a meta-analysis in the previous 5 years does not mean such a study was well-conducted or reported, so we will consider this when determining whether to publish a meta-analysis submitted to CORR®. We also concur with the need for clear and appropriate inclusion and exclusion criteria, and for reasonable homogeneity of the studies.

We make clear distinctions between requirements for meta-analyses and systematic reviews. Although meta-analyses with their stringent statistical analyses frequently provide the best available evidence, prospective randomized trials (ie, Level of Evidence I or II [1, 7]) are not always available in sufficient number or quality to address many important questions in surgical disciplines. Nonetheless, answers based on the best currently available evidence are needed to guide physicians in treating patients. Therefore, for systematic reviews (in which little if any attempt is made to extract and statistically analyze individual bits of data from individual studies) we will allow high-quality retrospective trials (ie, Level III comparative studies or Level IV studies) to be included when of sufficient documented quality. (Readers should be aware that Levels of Evidence do not necessarily reflect the quality of a study, but rather reflect the relative risks of bias.) In addition to the modified criteria above, we also will continue to require authors to provide a PRISMA flow chart and checklist and appropriate measures of study quality. We believe use of these guidelines will enhance the value of systematic reviews for current and future readers of CORR®.