Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014

Abstract

Objectives

The purpose of this study was to determine whether the number of bibliographic databases used to search the health sciences literature in individual systematic reviews (SRs) and meta-analyses (MAs) changed over a twenty-year period related to the official 1995 launch of the Cochrane Database of Systematic Reviews (CDSR).

Methods

Ovid MEDLINE was searched using a modified version of a strategy developed by the Scottish Intercollegiate Guidelines Network to identify SRs and MAs. Records from 3 milestone years were searched: the year immediately preceding (1994) and 1 (2004) and 2 (2014) decades following the CDSR launch. Records were sorted with randomization software. Abstracts or full texts of the records were examined to identify database usage until 100 relevant records were identified from each of the 3 years.

Results

The mean and median number of bibliographic databases searched in 1994, 2004, and 2014 were 1.62 and 1, 3.34 and 3, and 3.73 and 4, respectively. Studies that searched only 1 database decreased over the 3 milestone years (60% in 1994, 28% in 2004, and 10% in 2014).

Conclusions

The number of bibliographic databases searched in individual SRs and MAs increased from 1994 to 2014.

Systematic reviews (SRs) use formal methods to search, critically appraise, and synthesize the literature to provide summaries of medical research reports on specific clinical questions [1]. SRs bring together many separately conducted studies, sometimes with very different sample sizes and conflicting findings, and incorporate their results [2]. They employ reproducible methodology to identify all relevant studies, assess the validity of their findings, and combine the results to provide conclusive answers to clinical questions [3]. Once the literature has been systematically reviewed, quantitative data from individual studies may be pooled and reanalyzed using established statistical methods [4]. These reports are called meta-analyses (MAs), which are considered a subtype of SRs. The rationale for MAs is that combining samples of individual studies increases overall sample size, thereby improving the statistical power of the analysis and the precision of the estimates of treatment effects [5]. SRs and MAs are foundations of evidence-based medicine, as they rely on balanced inference generated from collated evidence instead of commentaries made by experts, as in the case of narrative reviews [6]. SRs and MAs have become increasingly popular for providing evidence of the effectiveness of medical interventions to support the creation of clinical practice guidelines. In 2010, an average of eleven new SRs or MAs were published each day [7].

High-quality SRs require high-quality literature searches and accurate reporting. Searching a single database identifies a maximum of one-third of all relevant articles, and searching more databases may identify only half of all available articles [8]. A search conducted solely in MEDLINE results in database bias, as only 30%–80% of all trials are identifiable through MEDLINE [9]. Although the Cochrane Handbook for Systematic Reviews of Interventions identifies MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials as the three main databases for searching any topic on medical intervention [3], the selection and usage of databases in some SRs and MAs still falls below this minimum recommendation.

In this study, the authors examined the number of databases searched in SRs and MAs during three milestone years (1994, 2004, and 2014) across a twenty-year period related to the official 1995 launch of the Cochrane Database of Systematic Reviews (CDSR). Our objective was to establish a reference point for future authors of SRs and MAs and library and information professionals on the historical mean and median number of bibliographic databases searched.

METHODS

The search was conducted in Ovid MEDLINE using the Scottish Intercollegiate Guidelines Network (SIGN) search filter for SRs and MAs [10]. A search filter is a pretested strategy that identifies high-quality evidence from databases. The SIGN filter was modified by the addition of keyword search terms “Web of Science,” “Web of Knowledge,” “Scopus,” and “Google Scholar” to the original list of database names to reflect changes in database availability (Table 1). This strategy was chosen instead of relying on the SR and MA publication type filters available in MEDLINE, because the SR publication type in MEDLINE is not a real publication type tag that indexers assign manually but rather an embedded, hidden search strategy that was originally designed by Shojania and Bero [11, 12], and the MA publication type in MEDLINE has been criticized for being inaccurate and inconsistent [13]. The SIGN strategy shows better sensitivity, specificity, and precision than the filters available in MEDLINE [14].

Table 1.

Literature search strategy

The search was limited to 1994, 2004, and 2014, and a set of results was generated for each year. By default, the Ovid search interface sorted all resulting records by reverse chronological order, with an arbitrary number assigned for each record. Randomization software [15] was used to produce a random sequence of integers for each of the 3 years. For instance, in 1994, we retrieved 747 records. The randomization software generated a random sequence of integers from 1 to 747. The MEDLINE search results were rearranged according to this randomized sequence. These steps were repeated for each year until all records were listed in a random sequence.

The records were then examined one by one. Records were included if they were a self-proclaimed SR or MA or if they were tagged by MEDLINE as an SR or MA in the publication type field. In addition, because the terminology and concepts of SRs were made popular by the launch of the CDSR, which did not occur until 1995, reviews that did not explicitly use the terms “SR” or “MA” but that used reasonably comprehensive search strategies with the purpose of finding all relevant studies were also included.

Records were excluded if they did not involve literature searching or if the number and/or names of databases searched were not reported. Information regarding database usage was first sought in the abstract, and, if absent, the full text was consulted. In cases in which the required information was not in the full text or the full text was not available from the local medical library, the record was excluded, and the next one on the random list was moved up to fill the void. These steps were repeated until 100 records with the desired information were examined and analyzed for each of the 3 years of interest (Figure 1).

Figure 1.

Research process flowchart

The number and names of bibliographic databases used in the studies were counted, recorded, and analyzed. Other information such as the number of authors and the location of first authors was also recorded for analysis.

RESULTS

The search strategy retrieved 747 records for 1994, 4,112 records for 2004, and 16,433 records for 2014. A total of 339, 157, and 127 records for 1994, 2004, and 2014, respectively, were examined to achieve 100 qualifying records for each year. Seventy-three (21.5%), 15 (9.5%), and 12 (9.4%) records from 1994, 2004, and 2014, respectively, were excluded due to poor reporting in the abstract and full text (i.e., no mention of the number and/or names of databases used) or the lack of full text availability. The remaining excluded studies were determined not to be SRs or MAs upon examination of the abstracts.

The mean and median numbers of bibliographic databases used in the randomized samples of SRs and MAs were 1.62 and 1 for 1994, 3.34 and 3 for 2004, and 3.73 and 4 for 2014. Of the 100 records from 1994, 60% had literature searches that involved only 1 bibliographic database. This dropped to 28% in 2004 and 10% in 2014.

A total of 116 bibliographic databases were named in the 300 randomly selected records. Among all databases, MEDLINE/Pubmed/Index Medicus was the most frequently used database in all 3 years combined, followed by Embase/Excerpta Medica and the Cochrane Library (Table 2).

Table 2.

Top ten most frequently used databases for systematic reviews (SRs) and meta-analyses (MAs) in 1994, 2004, and 2014

The first authors of the selected studies were from 26 different countries. The mean numbers of coauthors per study were 3.46, 4.24, and 4.87 in 1994, 2004, and 2014, respectively, suggesting an upward trend.

DISCUSSION

Our results indicate that the literature searching practice for SRs and MAs has changed substantially since 1994. We found that many SRs and MAs from 1994 were not based on any literature searching but were instead based solely on private collections of clinical trials reports known to the authors and gathered by unknown methodologies. Even in cases in which bibliographic databases were searched, the databases were often unnamed, a practice that undermines the role of the literature searching. By the years 2004 and 2014, however, the mean and median number of databases searched and reported (3.4 and 3, 3.73 and 4, respectively) increased and exceeded the minimum of 3 databases (MEDLINE, Embase, and Cochrane) recommended by the Cochrane Handbook. This emerging trend of multi-database SRs and MAs is further confirmed by a decrease in the number of single database studies from 60% to 28% to 10% in 1994, 2004, and 2014, respectively. We believe this is the first quantitative study on the topic.

This change in database searching practices might be due to a number of reasons, including strengthened recognition of the importance of literature searching, more involvement of library and information professionals in the research process, a growing number of opportunities for training in SR methodology, better availability of databases or computers, a transition from print-based indexes to digital databases, increased emphasis on the quality of reporting, the launch of the CDSR, and enhanced awareness and adoption of emerging guidelines, including the Cochrane Handbook in 1994 [16], Quality of Reporting of Meta-Analyses (QUOROM) statement in 1999 [17], and Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) in 2009 [17]. These guidelines for conducting and reporting SRs and MAs minimize bias by promoting thorough, objective, and reproducible multi-database searches [3]. The observed increase in the number of databases searched might imply that higher-quality, more comprehensive SRs and MAs are being published, which in turn contributes more meaningful knowledge to the practice of evidence-based medicine.

The top ten databases searched in the three milestone years can be categorized into three groups. The first group consists of comprehensive medical or health care databases: MEDLINE, Embase, CINAHL, Cochrane Library, and Cochrane Central. The second group consists of specialized databases focusing on narrow disciplines within medicine or health care: PsycLIT/PsycINFO and Cochrane Specialized Registers. The third group consists of general science databases: Web of Science, Current Contents, and the Chinese database CNKI. The first group includes the top five databases from our top ten database list (Figure 2), indicating that most researchers prefer using comprehensive medical or health care databases for SRs and MAs and only use specialized or general science databases when extra coverage is required.

Figure 2.

Most frequently used databases

Along with an increase in the number of databases searched in SRs and MAs since 1994, the number of coauthors of SRs and MAs has also increased. SRs take varying amounts of time but usually require between nine months to one year to complete. With an increasing awareness of proper SR methodology, the number of coauthors may also be increasing to cope with the raised standards and more extensive research process [18].

Our study has some limitations. Only a relatively small sample of 100 records was examined from each of the 3 milestone years, and only 3 years were examined as opposed to the entire 20-year span from 1994 to 2014. As the purpose of this study was to analyze the usage pattern of bibliographic databases only, hand searching of journals or books; searching of individual conference proceedings, private bibliographies, private databases, and reference lists; or consultation with experts in the field were not considered, although these practices were common among the examined studies and could be important sources of information for SRs or MAs. Another possible source of bias was the exclusion of records for which database usage information was not available in the abstracts and the full text was not available from the local medical library, which might have led to a bias toward studies published in relatively well-established journals that are more likely to be available in library collections.

Biography

Michael T. Lam, BMSc(C), michaelthomaslam@gmail.com, Undergraduate Student, Bachelor of Medical Science Program, Elgin Hall, Western University, 1151 Richmond Street, London, ON N6A 4K7, Canada; Mary McDiarmid, MISt, mcdiarmidm@ontarioshores.ca, Library, Ontario Shores Centre for Mental Health Sciences, 700 Gordon Street, Whitby, ON L1N 5S9, Canada