Systematic reviews need systematic searchers

Abstract

Purpose: This paper will provide a description of the methods, skills, and knowledge of expert searchers working on systematic review teams.

Brief Description: Systematic reviews and meta-analyses are very important to health care practitioners, who need to keep abreast of the medical literature and make informed decisions. Searching is a critical part of conducting these systematic reviews, as errors made in the search process potentially result in a biased or otherwise incomplete evidence base for the review. Searches for systematic reviews need to be constructed to maximize recall and deal effectively with a number of potentially biasing factors. Librarians who conduct the searches for systematic reviews must be experts.

Discussion/Conclusion: Expert searchers need to understand the specifics about data structure and functions of bibliographic and specialized databases, as well as the technical and methodological issues of searching. Search methodology must be based on research about retrieval practices, and it is vital that expert searchers keep informed about, advocate for, and, moreover, conduct research in information retrieval. Expert searchers are an important part of the systematic review team, crucial throughout the review process—from the development of the proposal and research question to publication.

INTRODUCTION AND BACKGROUND

Librarians have much to contribute to health care research, as they have a broad knowledgebase and skill set that can be applied to research in many areas. Librarians have skills in complex bibliographic retrieval, organization of large amounts of data, and identification and verification of information. They must understand the complex information problems of health care professionals and be able to analyze and appraise research in their own discipline as well as in their users' disciplines. Recognizing the knowledge and skills of librarians, the Medical Library Association (MLA) has produced a research policy stating that health librarians need to contribute to health and to information policy [1]. Librarians can use their health information science knowledgebase to design, develop, and market new health information systems and services, including those that integrate scientific literature with other types of heath-related information. They must also be able to add to the health sciences information knowledgebase by carrying out research in their areas of expertise.

Librarians use their knowledge and skills by participating in the process of creating systematic reviews that will inform health care. Systematic reviews and meta-analyses are very important to health care practitioners. These reviews can help practitioners and decision makers keep abreast of the medical literature, because the reviews summarize large bodies of evidence and help to explain apparently different results among studies addressing the same question [2].

A systematic review is a review that uses systematic and explicit methods to identify, select, and critically appraise relevant research and to collect and analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and pool the results of the included studies [3]. Systematic reviews (also known as systematic overviews, evidence summaries, and integrative reviews) use recently developed scientific methods to summarize results from multiple research studies [4]. This paper will use the term “review” to encompass both systematic reviews and meta-analyses.

Searching is a critical part of conducting the systematic review, as errors in the search process potentially result in a biased or otherwise incomplete evidence base. Searches for systematic reviews need to be constructed to maximize recall and deal effectively with a number of potentially biasing factors. Therefore, librarians who conduct the searches for systematic reviews must be experts.

What follows is a description of the methods, skills, and knowledge of expert searchers working on systematic review teams. Regardless of background, librarians become experts by keeping abreast of the information retrieval literature and applying that knowledge. Expert searchers work closely with the review team through an iterative process and have the skills to develop, conduct, and revise a search strategy as well as determine which sources of information will be searched and how. Expert searchers understand the technical aspects of data structure, function of bibliographic and specialized databases, reference management, and document retrieval. Their expertise is not free, and financial compensation should be factored into project planning and budgeting. Also, their expertise should continue to develop and be based on research about search methodology.

WHO ARE THE EXPERT SEARCHERS?

The MLA Task Force on Expert Searching defines the expert search as “a mediated process in which a user with an information need seeks consultation and assistance from a recognized expert” [5]. The task force goes on to describe the skills and knowledge that are required and notes that the expert searcher is usually a highly trained and experienced librarian. The librarian has a combined set of skills and knowledge that most health care professionals and researchers do not have. In this article, the term “expert searcher” will describe a librarian with a master's level degree from a library school program accredited by the American Library Association.

Librarians who become expert searchers in medical librarianship come from a variety of backgrounds, experiences, and training. Some librarians may have health-related backgrounds or undergraduate degrees in the health sciences. Certain library school programs provide courses on medical librarianship, and some medical school libraries provide internship programs. MLA and other local and national associations provide continuing education sessions on various aspects of the searching process. Self-instruction and reading the literature to keep up to date are valuable options as well. Much information has been written on how to search, and expert searchers should be aware of this literature [6, 7]. While no specific designation indicates who is a qualified medical librarian, MLA has a peer-reviewed credentialing and accreditation program for librarians called the Academy of Health Information Professionals. Membership recognizes professional development activities.

Librarians who specialize in searching work in many situations. They work in traditional hospital or university libraries. They also work outside of libraries with established epidemiology groups, technology assessment programs, and other types of research institutes that require specialized searching skills to create reviews.

HOW CAN EXPERT SEARCHERS DEVELOP AND CONDUCT THE SEARCH?

The reference interview

When working with a review team on a systematic review, the librarian's first step is to understand the research scenario and questions that determine why a systematic review is required. In the equivalent of the reference interview, the librarian works closely with the review team to refine questions and characterize them in terms of elements—such those as used in the population, intervention, comparison, and outcome (PICO) elements for a clinical question:—even if not all of these elements are used in the formal search strategy. Any exclusion criteria that may have an impact on the search parameters are explored. These criteria include age groups, gender, or time periods of the search. The reference interview usually involves several discussions with different members of the review team. If, at any point in the review process, any criteria are changed, the librarian must be kept aware of these changes, as they may influence the search strategy and search results.

Based on discussions with the review team, the librarian selects the key resources to search, such as electronic databases and Websites. Decisions about sources are also based on knowledge of the available sources and testing of sources to determine completeness of coverage. Systematic reviewers are usually experts in the subject of the review, often world leaders in their field. Working with them, the librarian can identify major studies as well as the leading journals and conferences. The librarian can verify that these are comprehensively indexed in the selected electronic sources. If the indexing appears weak or incomplete, supplemental techniques such as hand searching can be used to ensure good coverage of these sources.

Few librarians rely on a single electronic database for a search, even if one database is expected to provide good coverage of the field. Because high recall is needed, and every retrieval can be expected to have less than perfect recall, some redundancy is usually introduced. A relevant paper could be indexed in MEDLINE but not be retrieved by the search. Because of differences in indexing, the translated search might retrieve that paper from EMBASE.

A comprehensive selection of sources for a systematic review often includes two or more bibliographic databases such as MEDLINE and EMBASE, a trials registry, conference proceedings, specialized subject bibliographies, reference lists of review articles, and contact with researchers and companies working in the area.

The iterative process

Electronic database searches provide the majority of material [8], and the strategies are generally developed in MEDLINE first. The MEDLINE retrieval, or a portion of it, is often given to the reviewers for feedback. The objective of the search is to retrieve all relevant material, and this high recall often leads to precision of less than 10%. However, reviewers may find that the search is off topic, and the librarian can refine the search according to the reviewers' feedback before proceeding. The librarian often tests the retrieved set to verify that major studies identified by the subject experts are included. Once the reviewers are satisfied with the MEDLINE strategy, that strategy can be tailored to and executed in the indexing language of the other databases to be searched.

The results of the database searches are usually quite large compared to the results from a clinical query. Numerous evidence reports prepared by the Agency for Healthcare Research and Quality (AHRQ) have been found to have initial retrieval of more than 4,000 items [9]. An example of present efforts to conduct research that studies the efficiency of search strategies is the authors' current investigation of efficiency in evidence reports prepared for the AHRQ.

Once the searches have been run, the retrieved records are usually stored in citation management software (e.g., Reference Manager or TrialStat's SRS), where references from different sources are converted to a common format, duplicate references are removed, and all references are assigned a number identifying the study. Careful counts are made at each step along the way. These citations counts are important for final reporting, when all identified citations must be accounted for (i.e., whether they were duplicates, screened out by reviewers, or were ultimately included in the review).

The authors know of no studies reporting average time spent on the various parts of the review. Based on the authors' experience, it takes from several hours to several days to define the search and retrieve an initial set of abstracts for screening by the reviewers.

The complexity of developing the strategy

Expert searchers must know the “ins and outs” of the various databases. Drawing on MEDLINE as an example, it is vital for the searcher to understand the differences between fields. One can look at the subject heading (sh) field and the publication type (pt) field. An article with the subject heading of randomized controlled trial is an article about randomized controlled trials. An article with the publication type of randomized controlled trial is a randomized controlled trial—exactly what is needed for the review! When searching for articles about meta-analyses and their methodologies, a subject heading is available. However, when searching for actual meta-analysis studies, a publication type is more appropriate. As shown in the above example, being aware of the meaning and implications of searching different fields will determine the quality of a search strategy.

The searcher must be familiar with the sensitivity and specificity of features such as age tags and subheadings: are these suitable for use in a very high recall strategy? In a recent systematic review of treating infertility subsequent to spinal cord injury, the searcher hypothesized that using the subheading “rehabilitation” with keywords would have been successful in retrieving studies, however, only 12 of 118 studies found through MEDLINE had a rehabilitation subheading (rh). In other topics, the rehabilitation subheading may perform well when used as a floating subheading. The librarian must have the expertise to develop test strategies to verify the performance of terms and elements of the search, adjusting or abandoning nonperforming elements. Often these tests rely on comparison against a strategy from a previously published review or the recall of a set of key references supplied by subject experts.

The librarian constantly balances the challenges of maintaining high recall to ensure that no relevant articles are missed, without overwhelming the resources of the review team. Consequently, search strategies for systematic reviews can become quite large. For example, in the Complementary Medicine field of the Cochrane Collaboration, their search strategy for topics in complementary medicine in MEDLINE contains 248 lines, not including the Cochrane Collaboration search filter for randomized controlled trials, which contains 29 lines in Ovid.

The sources of information other than bibliographic databases

Other sources, besides bibliographic databases, can provide important material for the review and must be searched to protect against potential publication and database bias [10]. Librarians involved in systematic reviews are expected to have a basic appreciation for the main elements of epidemiological bias [11]. The Medical Subject Headings (MeSH) scope note for Bias (Epidemiology)/explains it as one-sided or systematic variations in measurement from the true value (systematic error). For example, the tendency for positive findings to be preferentially published means that the published literature is biased—relying on only published studies gives a predictably exaggerated impression of how well a treatment works. Expert searchers are responsible for guarding against bias in their conduct of the search. A database-only search would generally be considered a mark of low quality [12].

While centrally responsible for the development and execution of the electronic search, the librarian ideally works hand in hand with the reviewers to define and undertake the balance of the search strategy. Discussing other sources of information with the review team is essential. The subject experts might have conference proceedings in their collections and might have established connections with key researchers in the field. Subject experts can correspond with colleagues to identify unpublished manuscripts and other fugitive or gray literature, as well as published works that may not have been retrieved by the database search. Pharmaceutical companies and other manufacturers can be approached for study results.

Expert searchers might find specialized registers of clinical trials helpful in locating studies. Specialized registers have been developed largely to support the needs of systematic reviewers. Most notable is the CENTRAL database of the Cochrane Collaboration, a compilation of the many subspecialty registers. These registers are compiled from specialty searches run regularly against major databases and from material identified through a coordinated hand searching effort [13]. They are maintained by trial search coordinators of Cochrane Review Groups. Many of these coordinators are librarians. Additional hand searching may be warranted in the most recent issues of key journals, when key journals are not covered by the Cochrane Collaboration efforts or when research designs other than controlled trials are required.

The focus of the review is to retrieve primary studies (i.e., original reports of randomized controlled trials). Although review articles are excluded, the reference lists of review articles can help identify additional primary studies to be reviewed by a member of the review team. As additional references are identified, they are added to the citation manager software with an indication of where they were located.

WHAT SHOULD BE RECORDED?

As the work progresses, the librarian records search strategies used, databases searched, dates of coverage provided by each database at the time of the search, number of items downloaded from each database (often records are tagged with the source database), order of precedence in removing duplicate citations, and number of duplicate items removed. All of this information is needed for final reporting.

WHAT ARE THE TECHNICAL ASPECTS?

By now, it should be clear that systematic reviews are quite a bit of work. The main intellectual components of the review process are laid out in Table 1, but supporting these efforts are all the technical components: running the search in multiple databases, downloading the bibliographic records and importing them into the citation manager, and finding and removing duplicate records. Reviews can take a year or more to conduct, and it is common to rerun the search toward the end of the review to identify new material.

Table 1 Characteristics of the review

Consider also that, if the initial retrieval contains 4,000 items, the reviewers will want to see a full copy of many of these articles (thankfully, many can be excluded on the basis of information in the bibliographic record). Obtaining a full copy leads to a great deal of work ordering, logging, and labeling articles for distribution to the reviewers. Other technical work includes handling the non-database portion of the search, such as locating copies of conference proceedings and creating bibliographic records for unpublished studies that are found. All of this work must be done accurately and often quickly.

Librarians can contribute to technical aspects of the review beyond the traditional information science domain. Librarians are often more technologically sophisticated than the subject experts on the team and so may contribute to information management in the reviewing stages, advise on software, and set up forms for electronic data abstraction. Such participation can be helpful, as it keeps the librarian in touch with the progress of the review once the initial search has been completed.

WHEN DOES THE ROLE OF THE LIBRARIAN IN THE REVIEW PROCESS END?

Table 1 also documents the main elements of the review and the players associated with each element. The librarian is a key player on the team and needs to be an integral player in all meetings. The librarian's expertise can be helpful at many stages of the review to manage ongoing issues, from the initial development of the review to publication. For example, changes in scope or in the focus of questions might require that the search be modified to provide a sound evidence base for the review or that more sources need to be searched. The librarian can also help manage the volume of the material needing to be retrieved and create strategies for rapid document retrieval as deadlines loom.

WHAT ARE THE COSTS?

All reviews take time for all members of the review team. Some reviews are formally funded, while others are not. In the case of the Cochrane Collaboration, some reviewers work on a voluntary basis, while other systematic reviews can cost up to a quarter of a million dollars. How does one estimate the costs for a review? Allen suggests that to estimate costs, one correlates the time necessary to carry out the review by analyzing the total body of literature in the area [15].

Regardless of whether or not the review is funded, it does require the time of a librarian, which directly translates into costs. In estimating the total number of hours required of the librarian, all aspects of the librarian's involvement must be accounted for. Time includes hours spent in all meetings and discussions, research into the subject area or on databases, development and implementation of the search, record keeping, reporting, and writing. In addition to the librarian's hourly rate, other issues such as overhead for computers, database access, office space, and continuing education must also be taken into account to give a fair view of the true cost of librarian services. Other library services also need to be taken into account for other items, such as library technician costs for processing interlibrary loans and photocopying and the costs of document delivery.

WHAT ABOUT REPORT WRITING AND AUTHORSHIP?

The record keeping that has been referred to throughout this paper culminates in preparing the manuscript: the librarian writes a description of the search for the methods section of the review. This section describes the development of the search, major aspects of the search including databases and date ranges searched, and any restrictions (such as age or language limits). In most Cochrane reviews and an increasing number of journal-published reviews, the electronic search strategy is reproduced in the published report, often as an appendix. Counts are reported in the text and the quality of reporting meta-analysis (QUORUM) flow chart. A sample flow chart is shown in Figure 1.

Figure 1.

Example of a quality of reporting meta-analysis (QUORUM)-style flow chartSource: Sampson M, Campbell K, Ajiferuke I, Moher D. Randomized controlled trials in pediatric complementary and alternative medicine: where can they be found? BMC Pediatr 2003;3(1):1. Available at: <http://www.biomedcentral.com/1471-2431/3/1>.

Several reasons exist for this detailed reporting. The first is transparency: others should be able to satisfy themselves that the review is not open to bias. The second is reproducibility: others should be able to replicate the methods and arrive at the same results, and, most importantly, the same team or another review team should be able to repeat the search to update the review at a later date by incorporating new studies.

One issue for librarians is whether or not to be included as an author on the published review. Inexperienced reviewers might not fully appreciate, or even understand, the librarian's intellectual contribution to the review. In fact, the librarian's contribution is central. A flawed or biased search can render the review useless, and, as shown above, a great deal of specialized knowledge goes into developing a search that results in a valid evidence base. Librarians approached by systematic reviewers for involvement in a review project should not be shy to negotiate authorship up front.

Understanding what constitutes authorship is helpful when negotiating authorship. A common starting point for discussion is the “Uniform Requirements for Manuscripts Submitted to Biomedical Journals” (also known as the “Vancouver Statement”):

Authorship credit should be based on (1) substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; (2) drafting the article or revising it critically for important intellectual content; and (3) final approval of the version to be published. Authors should meet conditions 1, 2, and 3. [16]

For more about information about authorship, Davidoff provides an excellent discussion of its subtleties and politics [17].

HOW DO EXPERT SEARCHERS ADVANCE INFORMATION RETRIEVAL PRACTICES FOR SYSTEMATIC REVIEWS?

The MLA policy, “Using Scientific Evidence to Improve Information Practice” [1], calls upon health sciences librarians to conduct research relevant to health information problems. Eldredge draws direct parallels between evidence-based health care and evidence-based librarianship, holding up the latter as the standard to which health sciences librarians ideally practice [18]. Those librarians involved in systematic reviews (the top of the evidence hierarchy [19]) need to base their practice on the evidence and be active in researching those aspects of practice that are still based on expert opinion. For instance, the authors are currently researching some of the issues raised in this paper: cost implications of searching decisions, qualifications of searchers, and authorship status of searchers in systematic reviews.

Participation in reviews and other research efforts expands options for librarians [20]. One great advantage for librarians who become involved in systematic reviews is that they learn a powerful methodology for answering questions that arise in information science. As an example, we used the methods of systematic reviews to answer the question “where can randomized controlled trials in pediatric complementary and alternative medicine be found?”; were able to make evidence-based recommendations to librarians regarding searching and collection development; and were able to advise practitioners regarding the most productive journals to read [21].

But research is expensive. It is unrealistic to expect that librarians can build the evidence base of the profession in their free moments. Building the evidence base relies on the same approaches used in other fields: experts prepare research proposals and compete for peer-reviewed funding. Success in obtaining grant funding gives the experts resources to carry out the research, often by employing research assistants.

We must end by stressing the important role MLA has to play as an advocate in lobbying national governments and Health Technology Assessment groups to support methodological research in information science by providing grant support.