Analysis of the reporting of search strategies in Cochrane systematic reviews

Abstract

Background:

The Cochrane Handbook for Systematic Reviews of Interventions provides instructions for documenting a systematic review's electronic database search strategy, listing elements that should be in the description. Complete documentation of the search strategy allows readers to evaluate the search when critically appraising a review's quality.

Objective:

The research analyzed recently published Cochrane reviews to determine whether instructions for describing electronic database search strategies were being followed.

Methods:

Eighty-three new reviews added to the Cochrane Database of Systematic Reviews in the first quarter of 2006 were selected for analysis. Eighteen were subsequently excluded because their searches were conducted only in the specialized registers of Cochrane review groups. The remaining sixty-five reviews were analyzed for the seven elements of an electronic database search strategy description listed in the Cochrane Handbook, using dual review with consensus.

Results:

Of the 65 reviews analyzed, none included all 7 recommended elements. Four reviews (6%) included 6 elements. Thirty-two percent (21/65) included 5 or more elements, with 68% (44/65) including 4 or fewer. Three included only 2 elements. The 65 reviews represented 41 different Cochrane review groups.

Conclusion:

The instructions from the Cochrane Handbook for reporting search strategies are not being consistently employed by groups producing Cochrane reviews.

Highlights

• Cochrane guidelines for reporting electronic database search strategies are not being consistently employed by groups producing Cochrane systematic reviews.

• None of the Cochrane reviews analyzed in this study included all seven recommended elements in the search strategy description.

• Most Cochrane reviews in this analysis listed the databases searched, years covered, and details of search terms. Most reviews did not state the exact date the searches were performed or the database hosts.

Implications

• Librarians who conduct searches for systematic reviews should try to ensure that the search process is carefully tracked and completely reported when published.

• Librarians should convey standards for complete tracking and reporting of the search process when they consult about search methods with systematic review researchers.

• Librarians should use high-quality search reports in published systematic reviews as models when teaching how to conduct these searches.

Introduction

Systematic Reviews

Systematic reviews serve as a preeminent source of synthesized knowledge for evidence-based practitioners. As the number of published research articles grows, practitioners find it increasingly difficult to locate and assimilate current knowledge. By identifying, analyzing, and synthesizing the best research on a specific topic, well-conducted systematic reviews can save practitioners' time, support evidence-based clinical practice, facilitate health policy decision making, and serve as a resource in the development of practice guidelines.

Conducting a systematic review starts with the articulation of a clear research question. Next, a study protocol is written that defines inclusion and exclusion criteria to guide the search and help reduce bias in the selection process. Working at times closely with an experienced searcher, reviewers create a strategy to comprehensively identify and retrieve relevant research on a topic. The search strategy lists the databases to search and articulates the complete set of terms and Boolean logic combinations to use. The search is then adapted to match the specific structure of each database. Comprehensive searching of electronic databases rarely retrieves all the pertinent literature and should be supplemented by checking the reference lists on relevant search results [1]. Publication bias, the recognized tendency for negative results to be published less frequently, should be addressed through additional searching of the gray literature [2,3]. The goal of this core portion of the systematic review method is to assure that the highest quality evidence available on the topic is included in the review.

Importance of High-Quality Search Strategies

The goal of a systematic review is the complete and unbiased identification of relevant studies. Errors in electronic database search strategies can impact the recall (sensitivity) of the search, leading to missed research and incomplete conclusions. Sampson et al. sought to identify the elements of a systematic review search strategy associated with accuracy and completeness [4]. Based on a thorough review of the literature, these authors identified the search errors that have the greatest impact on recall: errors in conceptualization of the search, logical operator errors, missed index terms, spelling errors, and errors in adapting search syntaxes for different database structures. To detect such errors, the search strategy needs to be made completely transparent to readers.

Importance of Search Strategy Reporting

For a number of reasons, the search strategy used to gather research for a systematic review must be fully reported when the review is published. First, as just mentioned, explicit reporting of the search strategy serves as a mechanism to evaluate the quality of the search. It enables readers to judge the credibility and methodology of the review [5]. Second, explicit reporting of the search strategy allows replication of and consistency in the search approach employed when the review is updated. Few systematic reviews discover that the existing research evidence is conclusive; most make recommendations for further research. Ideally, all systematic reviews that do not find conclusive evidence should be updated, because results from systematic reviews are most useful when they represent the current state of research. There is, however, no consensus on the timing of or the method for updates [6,7]. Finally, explicit reports of a search strategy can be used as a building block in developing search strategies for reviews on related topics, facilitating the creation of a shared knowledgebase.

The Librarian's Role in Search Strategy Reporting

Librarians have been identified as key members in the systematic review process [8–10]. As Beverley et al. pointed out, “information professionals have evolved from simply acting as ‘evidence locators’ and ‘resource providers’ to being quality literature filterers, critical appraisers, educators, disseminators, and even change managers” [11]. They identified ten possible roles for information professionals in the systematic review process: project leader, project manager, literature searcher, reference manager, document supplier, critical appraiser, data extractor, data synthesizer, report writer, and disseminator. Given the importance of search strategy reporting mentioned above, librarians who participate in the systematic review process should be aware of the evolving standards for complete reporting and work to convey those standards to others. Librarians who participate as part of a systematic review team should try to ensure that a search process is tracked and completely reported during publication. Librarians called in as consultants on the search process for systematic reviews should be sure that the researchers are informed about standards for reporting search strategies. When providing instruction on how to conduct a search for a systematic review, librarians should include information about standards for search strategy reporting and point out examples of systematic reviews with high-quality search reports.

Overview of Research on Search Strategy Reports

The definition of a systematic method for reviewing and synthesizing clinical literature grew out of the recognition that traditional narrative reviews lacked scientific rigor [12,13]. As an increasing number of reviews adopted a more systematic process, researchers began to focus on assessing the quality of the methods used to conduct the review and the quality of the search strategy report. However, relatively few of these researchers have focused specifically or in any detail on the issue of the completeness of the search strategy report. Table 1 presents an overview of the research related to assessing the completeness of search strategy reports discussed below.

Table 1.

Summary of research related to reporting of search strategies for systematic reviews

The articles in Table 1 that are most focused on search strategy reporting are those by Major, Flores-Mir, and Major who used three criteria derived from the Cochrane Handbook for Systematic Reviews of Interventions [14] to evaluate the reporting of search methods in dental systematic reviews. Initially focusing on orthodontic systematic reviews published between January 2000 and December 2004, they found that 88% reported search terms [15]. However, database names and search dates were not documented in 37.0%, and 62.0% failed to document the complete search strategy. When their analysis was expanded to include all dental systematic reviews published between January 2000 and July 2005, they found substantial improvement over this period [16]. One hundred percent of reviews published in 2005 reported database names and search dates. Ninety-five percent provided search terms. However, 20.0% still failed to provide a complete search strategy. Next, while analyzing the differences between dental specialties, they extended the end publication date to June 2006 and found that, over all years and specialties, 49.6% of the reviews did not provide a complete search strategy [17].

Moher et al. also focused in some detail on search reporting in an analysis of a set of 125 Cochrane systematic reviews issued in November 2004 combined with 175 systematic reviews indexed in MEDLINE during the same month [18]. They used an independently created and pilot-tested form to assess reporting of search strategies by analyzing reports of the number of databases searched, the number of other sources searched, the years covered, and the extent to which and how search terms were reported. Their analysis revealed that 83.2% of the Cochrane therapeutic reviews reported dates searched compared with 58.0% of the non-Cochrane, while 78.3% of the Cochrane therapeutic reviews provided the full Boolean search strategy, compared to only 18.2% of the non-Cochrane. Overall, these authors found little consistency in the reporting of search strategies in this group of systematic reviews.

Sampson et al. published an analysis of the same reviews used by Moher et al. (less three reviews) using an assessment tool developed after evaluating eleven search reporting scales [19]. These scales included AMSTAR, Cochrane Handbook, STARLITE, Oxman and Guyatt, MOOSE, Quality of Reporting of Meta-analyses (QUOROM), and more. Items appearing in three or more of the scales were combined to create an evaluation checklist consisting of databases used, database vendor, dates covered, date of the search, search terms, full search strategy, key terms, statement of non-database search methods, language limits, publication status limits, list of excluded references, and qualifications of searchers. They found that 100% of the Cochrane reviews and 98.2% of the non-Cochrane reviews in this set reported the databases that were searched. Only 42.6% of Cochrane and 31.0% of the other reviews clearly stated the database vendor. The dates covered by the search were reported by 82.9% of Cochrane and 59.5% of the other reviews. A statement of the search terms used was included in 98.4% of the Cochrane and 75.6% of the other reviews, while the full electronic search strategy was included for 78.3% of the Cochrane reviews and only 15.5% of the other reviews. The qualification of the searchers was the least frequently reported item, with only 18.6% of the Cochrane reviews and 6.0% of the others including that information.

Research focused on the quality of search strategies used in systematic reviews also provided some information on the completeness of search strategy reports. Sampson and McGowan reported that they had to exclude 30.0% of the 105 MEDLINE search strategies they extracted from the Cochrane reviews released in issue 3, 2002, because not enough detail was reported to assess the quality of the search [20]. Patrick et al. analyzing the search strategies of 89 randomly selected meta-analyses published from 1996 to 2002, found that only 6.7% reported both a search strategy in sufficient detail that it could be repeated and evidence of the effectiveness of that strategy [21]. Sixty-four percent reported a retrieval search strategy in sufficient detail but no evidence of effectiveness, and 29.0% reported neither. Golder et al. examined the search strategies of 277 systematic reviews of adverse effects and found that only 4.7% provided sufficient information for the searches to be replicated [22]. Most reviews in Golder's sample reported databases searched; 76.0% provided years covered by the search; most reviews did not state language restrictions; and only 11.0% included information on the database host.

It is also often possible to find data on the quality of search strategy reporting embedded in an analysis of the overall quality of systematic reviews. The following articles provided some information about the quality of search reporting in Cochrane reviews. Shea et al. [23] used the Overview Quality Assessment Questionnaire (OQAQ) [24] to compare Cochrane reviews published in 1996 with those published in journals over multiple years prior to 1996. They found only 34.6% of Cochrane reviews, as compared to 67.3% of journal reviews, reported search methods. The discussion section of the article by Shea et al. mentioned an effort by the Cochrane Collaboration to improve the quality of its reviews. Jadad et al. compared Cochrane reviews (issue 3, 1998) with those published in journals between 1988 and 1998 that focused on asthma research [25]. Using OQAQ, they found that 100% percent of the Cochrane reviews, compared to only 55.0% of journal reviews, reported search methods. In a later study by Shea et al., the quality of reporting in Cochrane Musculoskeletal Group reviews from 2002 also demonstrated improvement from the 1996 study [26]. Again using OQAQ, they now found that 88.0% of the Cochrane reviews reported search methods. Shea et al. [26] also used the QUOROM checklist [27] for analysis and found that 97.0% of the reviews described the databases used in the search. In the discussion, this article mentioned that both of these instruments were hard to use because they lacked clear published guidelines. It should also be noted that not all Cochrane review groups have demonstrated the same level of quality in reporting databases used. A QUOROM assessment of Cochrane Multiple Sclerosis Group reviews for 2006 found a lack of search strategy description in 40% [28].

Many more examples in the literature provide limited data on the quality of search strategy reporting as part of a broader assessment of systematic review quality; however, the authors do not intend to provide a comprehensive review of that topic. Rather, this overview of related research should make it clear that few articles specifically analyze reporting of search strategies in any depth and that the poor quality of search reporting has created problems for studies analyzing the quality of systematic review searches. The quality of search reporting may be improving, particularly in Cochrane reviews; search reporting is generally of higher quality in Cochrane than non-Cochrane reviews. A variety of assessment tools have been used to analyze search reporting as part of overall systematic review quality.

Evolution of the Research Topic

The authors are a group of librarians from across the United States who met while participating in a continuing education course focused on the librarian's role in systematic reviews. We discovered a shared frustration with the inadequate reporting of search strategies in published systematic reviews. At the same time, we were all struggling with how to adequately report search strategies within the page limit confines of a journal article when we were part of systematic review teams.

Chalmers and Haynes acknowledged the difficulty of fully reporting a systematic review search process in print journals early in the evolution of systematic reviews and indicated that it was a contributing factor to distributing Cochrane reviews in electronic format [29]. A lengthy complete report could be disseminated through the Cochrane Database of Systematic Reviews, while an abbreviated version could be published in standard print medical journals.

Because the Cochrane Database of Systematic Reviews has not posed the same space limitations as traditional journals, we were interested in determining how completely Cochrane reviews incorporated the Cochrane instructions for search strategy reporting. As noted above, while the literature indicated that search strategy reporting in Cochrane Reviews might be improving, only a few studies have looked at this issue in any depth, and none of these focused on compliance with the Cochrane instructions. We analyzed eighty-three new reviews from the Cochrane Database of Systematic Reviews published in the first quarter of 2006 to determine whether the guidelines for reporting search strategies from the Cochrane Handbook were followed [14].

Methods

The Cochrane Handbook is a core publication articulating the methods for systematic reviews [14]. It is the work of many researchers involved with the Cochrane Collaboration, an international nonprofit organization dedicated to creating and disseminating systematic reviews. The Cochrane Handbook is continuously updated in response to recommendations from reviewers and empirical evidence.

Selection of Reviews for Analysis

All new Cochrane reviews for first quarter of 2006 (eighty-three reviews) were selected for analysis. The titles were randomly divided into six sets using a spreadsheet random number generator. An initial analysis of thirty reviews by two authors revealed that those reviews that relied solely on searches of a Cochrane review group's internal specialized register could not be analyzed for the purposes of this study. Those reviews (eighteen) were eliminated, leaving sixty-five reviews to be evaluated. These reviews represented forty-one of the fifty-one different specific health care topic Cochrane review groups.

Evaluation Process

The evaluation tool was developed from section 5.2.2.1 of the Cochrane Handbook for Systematic Reviews of Interventions, version 4.2.5 [14], which described the elements that must be included when documenting electronic database searches. In accordance with the requirements outlined in this section, search strategy descriptions were checked for the presence or absence of the following seven items:

• databases searched

• name of host

• date search was run

• years covered by search

• complete search strategy

• one or two sentence summary of the search strategy

• language restrictions

According to the Cochrane Handbook, databases searched refers to the names of the databases, such as MEDLINE, and the name of the host refers to the platform on which the database is provided, such as Ovid, SilverPlatter, and EBSCO. The date a search was conducted should include the month, day, and year. Years covered by the search are those years the search was limited to or the years covered by the database, if no limits were applied. The Cochrane Handbook indicates that search strategies should be copied and pasted rather than retyped. The one or two sentence search strategy summary should describe which lines in the detailed search were included to locate research related to the health condition or intervention and which lines were used to identify studies of an appropriate design. Finally, the Cochrane Handbook [14] asks reviewers to provide information on the “absence of language restrictions.”

A template code-sheet and codebook were developed based on these seven elements. Authors worked individually and then sent their evaluations to one other author for review. Dual review with consensus was used. When consensus could not be reached, questions were submitted to the entire group for resolution.

Results

Of the analyzed Cochrane reviews (n = 65), none contained all 7 search description elements and only 4 included 6 of the 7 elements (Figure 1). Twenty-one included 5 or more elements, with 44 including 4 or fewer. Elements most frequently included were: 100% listed the databases searched, 91% included the years covered by searches, and 88% included the detailed search strategy (Figure 2). Elements most frequently missing were: 89% did not include the month, day, or year of the search; 83% did not mention the database host; 74% did not provide a search summary; and 31% failed to include a statement about language restriction.

Figure 1.

Number of reviews by number of elements included

Figure 2.

Percentage of reviews with each element

Discussion

This study demonstrates that authors of Cochrane reviews newly published in the first quarter of 2006 did not consistently follow the Cochrane guidelines for reporting electronic database search strategies. The Cochrane Handbook, section 5.2.2.1, states that “the search strategy for electronic databases should be described in sufficient detail in a review that the process could be replicated” and lists the seven elements of a search that should be included in the search description [14]. The Cochrane Handbook also provides an example of how to report an electronic search strategy. Our analysis of sixty-five Cochrane Reviews revealed that none included all elements as outlined in the Cochrane Handbook.

Confirmation of Others' Findings

The results of this study were consistent with the findings of 2 previously mentioned studies that specifically examined search reporting and included a large set of Cochrane reviews on a wide range of topics [18,19]. In analyzing 125 Cochrane reviews, Moher et al. found that most (83%) reported the date ranges of the searches, and 78% provided the full Boolean search strategies [18]. The rate of reporting for these 2 elements was found to be somewhat better in the reviews studied here: 91% reported the date ranges of the searches, and 88% provided the complete search strategy. Analyzing the same cohort of systematic reviews as Moher et al., Sampson et al. found that 62% of Cochrane reviews reported a statement about language restrictions and that all Cochrane reviews reported the databases searched [19]. These results were similar to ours, in which 69% reported a statement about language restrictions and all reported the databases searched.

Significance of Individual Search Elements

Forty-four, or almost two-thirds, of the reviews examined here included only four of the elements that the Cochrane Handbook [14] designates as necessary in the description of search strategies. Given the importance of most of these elements for demonstrating the quality of the search and permitting its replication, this finding indicates significant problems in reporting of search strategies in Cochrane reviews. With the possible exception of the name of the host, all of the elements have an important role:

• Databases searched: This element is essential. Research has shown that searching only one database, such as MEDLINE, does not adequately capture the literature for a systematic review [30,31]. Searching more than one database is almost always necessary and is an important step in the process. The reader needs this information to assess the quality of the review.

• Years covered by search: This element is essential for enabling the reader to assess the comprehensiveness of the search as well as for providing a reference point when a review is revised or updated.

• Date search was run: Stating the month and year the search was run clearly defines the end point of the search and provides necessary information when updating occurs. However, the exact day the search was executed is not absolutely necessary, because it is good practice to overlap the time period of the new search with the old search to account for inconsistencies in database updates.

• Complete search strategy: The complete search strategy should be included so that the reader can assess the quality of the search and, thus, the quality of the review. Inadequate searches or errors in search strategies can affect the quality of the review and lead to bias. A complete search strategy is also essential when a review is revised or updated.

• One or two sentence summary of the search strategy: A narrative explanation of the search provides the reader, especially one with limited detailed search experience, with an overview of how the search was conducted. It is also helpful information when a review is being revised or updated because it provides a clear explanation of the thought process underlying the design and scope of a search.

• Language restrictions: Language restrictions can lead to the omission of relevant research and reduce the comprehensiveness of the review. Any restrictions, or lack thereof, should be clearly stated.

• Name of host: This element is less important but is appreciated by searchers, because the information helps make a search truly replicable. The input syntax of bibliographic systems varies. The name of the host system gives information necessary to locate host-related syntax information when a search is reported with unfamiliar syntax.

Reasons That Search Strategies are Poorly Reported

There are several reasons why search strategies may be poorly reported. Sampson et al. have suggested that search strategy reporting is inconsistent because there is no agreement on optimum reporting of search methods. Their systematic review identified eleven instruments that provided guidance in reporting search strategies and found that only one criterion, databases searched, was common to all the instruments. A total of eighteen search-related items were included in the instruments, and recent instruments tended to include a higher number of elements. The authors of this study suggest that both systematic review researchers and readers would benefit from a single, consensus-based, high-quality guide and assessment tool for reporting search strategies [19].

In the case of the Cochrane reviews analyzed here, we theorize 2 additional possible reasons for the identified inadequate reporting of electronic database search strategies. First, though Cochrane reviews do not have a formal word limit, an upper limit of 10,000 words is recommended. This limit may lead authors to skimp in their descriptions of search strategies. Second, section 3.4 of the Cochrane Handbook, version 4.2.5, provides guidance in preparing the text of a review. Concerning the search strategy, it states, “the data sources used to identify studies should be summarized…the bibliographic databases searched, the dates and periods searched and any constraints, such as language should be stated. The full search strategies for each database should be listed here or in an Additional table” [14]. The author is then directed to section 5.2.2 for further details. Because the elements we found most frequently missing were those not mentioned in section 3.4, we suspect that authors might be relying on that section rather than using the full instructions in section 5.2.2.

Recent Changes to the Cochrane Handbook for Systematic Reviews of Interventions

An updated version of the Cochrane Handbook (version 5.0.0) was released during the preparation of this manuscript [32]. A new bulleted list in section 6.6.2.2, “Reporting the Search Process in the Review,” includes all of the electronic database search elements in the earlier version on which this study was based, except for the exact day of the search, the name of the host, and the summary of the search strategy. As mentioned earlier, the exact day appears to be unnecessary and the name of the host, while helpful, is probably also not necessary. However, the short summary of the search strategy is useful for novice readers of systematic reviews, because it provides a way to easily understand the strategy employed in the complicated searches typical of systematic reviews. The new bulleted list also includes gray literature sources, hand-searching, personal contacts, and any other sources. These items were mentioned in the earlier version of the Cochrane Handbook but were in separate sections from the bulleted list. Placing these elements all in one list concisely represents all the aspects of a comprehensive search strategy.

Author confusion may still arise when comparing the information in section 6.6.2.2 and the information about the contents for the methods section of a review described in section 4.5. These separate sections of the latest Cochrane Handbook describe the same process, how to report the search in the review; however, these two sections have subtle differences in the language used and no clear cross-references.

Recommendations

Given the importance of clear and complete search strategy reporting, we recommend that future versions of the Cochrane Handbook integrate the reporting guidelines in one section. Additionally, both a checklist and a template would be useful to reinforce inclusion of all necessary elements.

We concur with Sampson et al. that consensus agreement is needed to develop, validate, and promote the use of a single instrument for assessing search reporting for all systematic reviews [19]. The elements found in the Cochrane Handbook provide an essential core for what is needed, but that list could be improved by items included in other instruments.

We highly recommend including a statement identifying who devised the search strategy. Reporting the qualifications of the person responsible for searching has been correlated with the quality of those strategies [33].

Limitations of This Study

This study reviewed a small number (sixty-five) of Cochrane reviews, all of which were published at the same time (first quarter of 2006). A larger sample drawn from all current Cochrane reviews instead of from only newly published ones might have indicated different patterns. A larger sample would also have made it possible to determine if the quality of search strategy reporting over time and/or among different Cochrane review groups differed.

Conclusion

Search strategy reporting in systematic reviews plays a critical role in the quality assessment, reproducibility, and updating of reviews. Our study shows that the Cochrane guidelines for reporting electronic database search strategies are not being consistently followed. Explicit search strategy reporting is crucial for a number of reasons: it serves as a mechanism to evaluate the quality of a search, ensuring a fair judgment of the credibility and methodology of a review; it allows replication and consistency in the search approach to be employed when updating a review; and it can be used as a building block in the development of search strategies for reviews on related topics, thus facilitating the creation of shared knowledge.

By focusing specifically on detailed descriptions of search strategies as an essential part of systematic reviews, we have highlighted the required core elements and emphasized the need to raise the reporting standards adopted by all systematic review searchers and authors, as well as by their peer-reviewers.