Evidence-based retrieval in evidence-based medicine

Abstract

Objective: Clinical decisions based on a meta-analysis that is based on an ineffective retrieval strategy may have serious negative consequences for patients. The study objective was to investigate the extent to which meta-analyses report proof of their retrieval strategies' effectiveness.

Methods: The authors examined a random sample (n = 100) of articles in the 1996 to 2002 full-text subset of Ovid MEDLINE indexed as “meta-analysis.” We classified the articles in three ways: the article (A) reported both a retrieval strategy in sufficient detail (such that it could be repeated) and with evidence of the strategy's effectiveness, (B) reported a retrieval strategy in sufficient detail but not with evidence of the strategy's effectiveness, or (C) neither reported a strategy in detail nor evidence of the strategy's effectiveness. Articles classified as (A) were further classified according to the level of evidence reported.

Results: Of the eighty-nine articles in our final analysis, six (6.7%) were classified as category (A), fifty-seven (64%) as (B), and twenty-six (29%) as (C). Articles in category (A) reported a previously validated search, a published strategy, or strategy based on expert opinion.

Conclusion: Peer-review standards must be developed that require authors of meta-analyses to report evidence for the effectiveness of their retrieval strategies.

INTRODUCTION

Evidence-based medicine implies the need to base clinical decisions on the results of prior scientific study. It also implies the need for clear indication of the effectiveness of the strategies and methods used to retrieve that evidence. Such a need is increasingly recognized [1–3]. Typically, according to standards for evidence-based medicine (such as the “Levels of Evidence” developed by the Centre for Evidence-Based Medicine [4]), stronger evidence is based on rigorous scientific study, while weaker evidence is based on expert opinion. By analogy, strong evidence of the effectiveness of a bibliographic retrieval strategy might be based on prior testing of the strategy, while weaker evidence might be an expert searcher's opinion of its effectiveness. The potential negative consequences of ineffective retrieval strategies in health care are well demonstrated by the death of a healthy research volunteer at Johns Hopkins in 2001 [5]. The US Department of Health and Human Services Office for Human Research Protection (OHRP) Compliance Determination Letter of July 19, 2001, regarding the incident specifically cited the inadequacy of the bibliographic retrieval strategy used, saying

In particular, OHRP notes the following: (a) Prior to the research being approved by the IRB [internal review board], the investigators and the JHBMC [Johns Hopkins Bayview Medical Center] IRB failed to obtain published literature about the known association between hexamethonium and lung toxicity. Such data was readily available via routine MEDLINE and Internet database searches, as well as recent textbooks on pathology of the lung. [6]

Meta-analyses of prior studies reported in the medical literature are an increasingly important method of generating evidence to support clinical decision making. In addition to emphasizing the statistical methods used to compare studies in a meta-analysis, the focus needs to be on the bibliographic strategy used to retrieve reports of these studies. Quality control of a meta-analysis needs to focus on the statistical methods used to compare reports as well as the bibliographic strategy used to retrieve these reports. Clinical decisions based on a meta-analysis that is itself based on an ineffective retrieval strategy can have serious negative consequences for patients.

The authors sought to investigate the extent to which such meta-analyses themselves report proof of the effectiveness of their retrieval strategies. We examined a sample of peer-reviewed journal articles indexed in MEDLINE as publication type “meta-analysis” to determine whether they reported evidence of the effectiveness of their retrieval strategies.

METHODS

According to the Medical Subject Headings scope notes provided by the National Library of Medicine, articles reporting the details of a meta-analysis are indexed with the publication type “meta-analysis” [7]. (Articles indexed with the main subject heading of “meta-analysis” are articles about the topic meta-analysis, rather than reports of specific meta-analyses.) Thus, to construct a sample of peer-reviewed articles reporting the methods and results of specific meta-analyses, on November 21, 2002, we searched the 1996 to 2002 subset of the MEDLINE database through OVID for articles with publication type “meta-analysis.”

We used the Ovid MEDLINE search query “meta-analysis.pt,” producing 4,740 results. We limited the results to the years 1997 to 2002, producing 4,258 results. Next, we limited the results to English, producing 4,027 results. Of these results, 664 were available in Ovid full text, covering 52 peer-reviewed general and specialty journals and including 28 journals out of the 120 core clinical journals of the Abridged Index Medicus (AIM) [8]. We limited the results to Ovid full text, and we took a random sample (n = 100) of the 664 Ovid full-text results. Four authors (Patrick, Tao, Folk, and Moxley) independently classified each article in the sample in one of three ways.

Category (A) indicates that the article both reported a retrieval strategy in sufficient detail such that it could be repeated and reported evidence of the effectiveness of that strategy. We considered that a strategy was reported in sufficient detail to be repeated if the report at least included the databases searched, the years searched, and all of the search terms used. An example of a retrieval strategy reported in sufficient detail is the following statement:

We identified potential English-language sources from the MEDLINE database for the years 1966 through 1997 using the search terms physical examination, palpation, breast, breast diseases, diagnosis, diagnostic tests, and sensitivity and specificity. [9]

An example of a report of evidence of the effectiveness of a retrieval strategy, in this case that of a previously tested search strategy, is the following statement:

We searched Medline for 1966 to December 1995 using a combination of the March 1996 update of the optimally sensitive search strategy for trials from the Cochrane Collaboration. [10]

Category (B) indicates that the article reported a retrieval strategy in sufficient detail such that it could be repeated but did not report evidence of the effectiveness of that strategy. Finally, articles classified as category (C) neither reported a retrieval strategy in sufficient detail such that it could be repeated nor reported evidence of the effectiveness of a reported strategy. For example,

We performed searches of the MEDLINE database and reviewed the bibliographies of review articles, to identify studies describing treatment for acute myocardial infarction. We used population-based studies that reported at least 10 years of data whenever possible to determine changes in intervention rates for different therapies. Published meta-analyses of randomized controlled trials were used to estimate the average benefit from changes in these various interventions. [11]

In this case, the authors did not report the years searched, all of the search terms used, or any evidence of the effectiveness of the search strategy.

After completing the independent classifications, we compared results. When researchers classified articles differently, the criteria and article were discussed to reach consensus. Such cases were few and typically concerned whether a strategy was reported in sufficient detail such that it could be repeated. Finally, articles assigned to category (A) were further classified according to level of evidence cited. We used a one-dimensional classification consisting of three levels of evidence: (1) previously validated search strategy (strongest), (2) previously published search strategy, and (3) expert opinion (weakest).

RESULTS

Eleven of the 100 articles were excluded from further analysis. One article was excluded from further analysis, because it was a brief consumer-oriented summary. One article was excluded, because it was a letter commenting on another study. Nine articles were excluded from further analysis, because they did not involve a meta-analysis based on retrieval of studies from the medical literature but instead reported a meta-analysis based on reports contained in non-bibliographic databases. In one case, an article in our original sample was a “Patient-Oriented Evidence that Matters” (POEM) article [12] summarizing another article. Miser 1998 [13] was a POEM summary of a specific reference [14], and, while Miser 1998 was included in our original sample, that reference was not. We excluded Miser 1998 from further analysis and included the specific reference. The remaining 89 articles included in the analysis were contained in 25 peer-reviewed general and specialty journals covering at least 15 health care specialties (Table 1).

Table 1 Twenty-five journals included in random full text sample (n = 100)

Of the eighty-nine randomly selected articles that met the inclusion criteria, only six (6.7%), were assigned to category (A): those that both reported a retrieval strategy in sufficient detail such that it could be repeated and reported evidence of the effectiveness of that strategy. Fifty-seven articles (64%), were assigned to category (B): those that reported a retrieval strategy in sufficient detail such that it could be repeated but did not report evidence of the effectiveness of that strategy. Finally, twenty-six articles (29%), were assigned to category (C): those that neither reported a retrieval strategy in sufficient detail such that it could be repeated nor reported evidence of the effectiveness of a reported strategy.

Of the six articles assigned to category (A), four reported evidence that we classified as previously validated search strategy (level 1), one reported evidence that we classified as previously published search strategy (level 2), and one reported evidence that we classified as expert opinion (level 3).

DISCUSSION

Of the twenty-five journals containing articles included in our final sample, eighteen (72%) make an explicit reference in their statement of instructions for authors to the “International Committee of Medical Journal Editors, Uniform Requirements for Manuscripts submitted to Biomedical Journals” (ICMJE) [15]. The ICMJE directs authors to

[i]dentify the methods, apparatus (give the manufacturer's name and address in parentheses), and procedures in sufficient detail to allow other workers to reproduce the results,

implying a requirement that a meta-analysis report its retrieval strategy in sufficient detail such that it can be repeated. The ICMJE does not, however, explicitly require authors to report evidence of the effectiveness of their retrieval strategies. That eighteen journals containing articles in our final sample themselves make an explicit reference to the ICMJE statement appears to be consistent with our finding that 64% of the articles fell into category (B), those that report a retrieval strategy in sufficient detail such that it could be repeated but do not report evidence of the effectiveness of that strategy. However, ten journals that include an explicit reference to the ICMJE statement also include articles classified as our category (C), those that neither report a retrieval strategy in sufficient detail such that it could be repeated nor report evidence of the effectiveness of a reported strategy.

Two statements of requirements for authors of meta-analyses, the QUOROM and MOOSE statements, are mentioned in the author instructions of one journal, and the QUOROM statement was mentioned by one additional journal. Both of these journals include articles that we classified as category (C). The QUOROM statement for meta-analyses requires an author to include a description of the search strategy used,

in detail (e.g., databases, registers, personal files, expert informants, agencies, hand-searching), and any restrictions (years considered, publication status, language of publication). [16]

The MOOSE statement requires the author to similarly describe details of the bibliographic retrieval methods used including

—Qualifications of searchers (e.g., librarians and investigators)

—Search strategy, including time period included in the synthesis and keywords

—Databases and registries searched

—Search software used, name and version, including special features used (e.g., explosion) [17]

Because the MOOSE statement requires mention of the qualifications of searchers, it requires an indication of at least indirect evidence for the effectiveness of the retrieval strategy used.

Reports of bibliographic-based meta-analyses that do not report the retrieval strategy in sufficient detail to be repeated run counter to the basic tenets of meta-analysis research and evidence-based medicine. Furthermore, any reporting of a retrieval strategy that does not also report evidence of the effectiveness of that strategy is similarly at odds with the basic tenets of evidence-based medicine. More effort should be directed toward establishing clear standards of evidence of the effectiveness of search retrieval strategies in evidence-based medicine. In addition, editorial and peer-review standards must be developed and applied that require authors of meta-analyses to report evidence for the effectiveness of the retrieval strategies they employ.