Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports

The authors of the study “Evidence-Based Retrieval in Evidence-Based Medicine” in this issue of the Journal of the Medical Library Association (JMLA) examined the replicability of the literature search strategies provided in meta-analyses indexed in PubMed [1]. The authors found that only 6.7% of a random sample of articles with meta-analyses had “both reported a retrieval strategy in sufficient detail such that it could be repeated and reported evidence of the effectiveness of that strategy.” The authors concluded that “[p]eer-review standards must be developed and applied that require authors of meta-analyses to report evidence for the effectiveness of the retrieval strategies they employ.” They also concluded that “[r]eports of bibliographic-based meta-analyses that do not report the retrieval strategy in sufficient detail to be repeated run counter to the basic tenets of meta-analysis research and evidence-based medicine.”

This study points to a shortcoming of meta-analysis that happily is easy to correct. Both conditions should be part of any literature review. It is important to know if the researchers' findings are generalizable to similar shortcomings in search strategies used for other purposes. For example, is there evidence that the literature reviews in Cochrane Collaboration reports have replicable and effective search strategies? A news story in BMJ in February 2003 by White [2] highlighted a Cochrane study on the effectiveness of editorial peer review. White's news story focused on the conclusion of the Cochrane review: “the practice of peer review is based on faith in its effects, rather than on facts” [3], which came from a review of 21 studies selected from a set of 135 studies that were identified through a “comprehensive” literature search.

A careful examination of data published in the Cochrane study [4] on the effectiveness of editorial peer review suggested that there, too, search strategies were not as rigorous as they might be. Even though the search strategy was replicable, the methodology for the literature search for this report was not comprehensive. In the Cochrane study, all searches of online databases used only the term “peer review.” Studies of peer review cover a range of subjects (e.g., rejection rates, reviewer agreement, reviewer bias, statistical review, blind or anonymous review); many of these terms would not be retrieved with only the term “peer review.” In addition, until fairly recently, the term “refereeing” was more commonly used.

The Cochrane authors searched a number of medically related online databases and provided the coverage years. However, the Cochrane authors did not provide criteria for database selection or for the years searched, and not all databases were searched fully. For example, Current Contents is available online from 1993 but was only searched from 1999 to 2000. ISI's Web of Science was not searched and is an excellent tool for retrieving cited references. The most recent amendment to the report was November 18, 2002. However, none of the databases were searched beyond 2000.

The Cochrane authors hand-searched a number of monographs and journals but did not explain their selection criteria. Some relevant reviews of the editorial peer-review process were not included: (1) Weller [5] undertook an analysis of studies of editorial peer review and identified, for example, 32 studies of reviewer agreement and 34 studies in the medical literature on statistical review; (2) Weeks and Kiner [6] published proceedings from a conference aimed at disseminating research on editorial peer review; and (3) Speck [7] annotated 780 publications on the subject of peer review. The Cochrane example is important, because editorial peer review is considered pivotal to the publication of solid scientific studies. The study of that process should be done with as much care and thoroughness as possible.

The Cochrane Reviewers' Handbook [8] gives explicit steps to follow for designing search strategies, and The Cochrane Manual [9] provides principles for searching. In an article in the Bulletin of the Medical Library Association, Helmer and colleagues [10] stated that “guidelines [for search strategies] generally require that researchers and librarians search” [emphasis added]. A search of both the Cochrane handbook and manual revealed that the Handbook twice (Appendix B and Appendix 5B) suggested that librarians be consulted for particularly complex searches. Neither the handbook nor the manual suggested that a librarian be a member of the team of researchers.

The findings of the JMLA researchers and the Cochrane example convincingly illustrate the need for librarians to be on any team that sets out to undertake meta-analyses or Cochrane reviews. Not only does the literature search strategy need to be reproducible, documentation should be provided that a comprehensive search was done, as pointed out by the JMLA authors. Had this been done with the peer review study, it might have drawn a different conclusion. The role of librarians in this process is essential.