Manual search approaches used by systematic reviewers in dermatology

Abstract

Objective

Manual searches are supplemental approaches to database searches to identify additional primary studies for systematic reviews. The authors argue that these manual approaches, in particular hand-searching and perusing reference lists, are often considered the same yet lead to different outcomes.

Methods

We conducted a PubMed search for systematic reviews in the top 10 dermatology journals (January 2006–January 2016). After screening, the final sample comprised 292 reviews. Statements related to manual searches were extracted from each review and categorized by the primary and secondary authors. Each statement was categorized as either “Search of Reference List,” “Hand Search,” “Both,” or “Unclear.”

Results

Of the 292 systematic reviews included in our sample, 143 reviews (48.97%) did not report a hand-search or scan of reference lists. One-hundred thirty-six reviews (46.58%) reported searches of reference lists, while 4 reviews (1.37%) reported systematic hand-searches. Three reviews (1.03%) reported use of both hand-searches and scanning reference lists. Six reviews (2.05%) were classified as unclear due to vague wording.

Conclusions

Authors of systematic reviews published in dermatology journals in our study sample scanned reference lists more frequently than they conducted hand-searches, possibly contributing to biased search outcomes. We encourage systematic reviewers to routinely practice hand-searching in order to minimize bias.

Well-conducted systematic reviews are the apex of the evidence hierarchy and routinely used for developing care guidelines and informing clinical decision making [1]. While each aspect of systematic review methodology is important, the search process, when thorough and well produced, leads to a set of research evidence to consider for inclusion that minimizes bias. There is a substantial body of evidence pointing to the importance of thorough and prespecified search strategies involving multiple databases to locate relevant studies and minimize the potential for publication and language bias [2]. As part of the search process, systematic reviewers often review reference lists of other studies or conduct hand-searches to identify additional primary studies. It is the authors' experience that reviewing reference lists and conducting hand-searches are often considered the same, yet we argue that these processes are quite different and lead to different outcomes.

Our objective was to assess how often systematic reviewers in dermatology actually conducted hand-searches when performing a systematic review.

METHODS

We conducted a PubMed search on January 4, 2016, for systematic reviews and meta-analyses published in the top 10 dermatology journals (January 2006–January 2016) according to the Science Citation Index 2015 impact factor. We used systematic reviews published in the dermatology literature due to the clinical interests of this team. We used a search string based on Montori et al. that has demonstrated high sensitivity to identifying such studies [3] (available on figshare <https://dx.doi.org/10.6084/m9.figshare.2073472.v1>).

Our search yielded 357 citations (356 after removing duplications). The second author (Atakpo) screened the full-text version of all studies to ensure that they were either systematic reviews, meta-analyses, or both. The PRISMA flow diagram detailing article exclusions is available on figshare <https://dx.doi.org/10.6084/m9.figshare.2073472.v1>. Systematic reviews were defined as studies that summarized research evidence from multiple studies and included information regarding inclusion and exclusion criteria, and search strategies. A meta-analysis was defined as a quantitative synthesis of primary study outcomes [4]. There were 293 systematic reviews and meta-analyses in our final sample, of which 292 were analyzed.

After screening, the third author (Kash) inspected each systematic review and extracted the verbatim statements from the reviews' “Methods” sections that discussed hand-searching, manual searching, inspection of reference lists, and other like terms. After completing this process, the first author (Vassar) and the third author (Kash) independently categorized each statement as either “Search of Reference List,” “Hand Search,” “Both,” or “Unclear.” After we performed these independent classifications, we reviewed the ratings and resolved any disagreements by mutual consensus. Stata 13.1 was used to produce descriptive statistics.

RESULTS

Of the 292 systematic reviews included in our sample, we found that 143 (48.97%) reviews that did not report a hand-search or scan of reference lists from primary studies. One-hundred thirty-six reviews (46.58%) only reported searches of reference lists from other primary studies. Four reviews (1.37%) reported systematic hand-searches, and 3 reviews (1.03%) reported use of both hand-searches and reference list scans. Six reviews (2.05%) used vague wording to describe these searches and were classified as unclear.

DISCUSSION

A majority of systematic reviewers in the dermatology journals composing our sample searched reference lists, which in turn, can contribute to biased search outcomes. Only a small percentage hand-searched journals using the systematic process that the Cochrane Collaboration recommends. The term “hand-search” was often used in conjunction with scanning reference lists. In some instances, we found that the language reported in these reviews was vague. For example, Miller et al. noted that “additional relevant articles were found by manual inspection” [5]. Such statements are not sufficient to determine the true nature of these manual searches. We recommend more detailed statements be included.

In another case, Baillie et al. noted that “hand-searching of journals was not performed as the key journals are all contained within the electronic databases searched for this review” [8]. Yet this statement contradicts advice provided in the Cochrane Handbook for Systematic Reviews of Interventions. According to section 10.2.2.3 of the Cochrane Handbook, “The perusal of reference lists of articles is widely used to identify additional articles that may be relevant although there is little evidence to support this methodology. The problem with this approach is that the act of citing previous work is far from objective and retrieving literature by scanning reference lists may thus produce a biased sample of studies” [6]. For example, studies with statistically significant outcomes are cited at a higher rate across studies of many disorders, making such studies more likely to be located through searching reference lists and, in turn, more likely to be included in a systematic review [7]. On the other hand, hand-searching is a manual page-by-page examination of entire journal issues or conference proceedings over a particular time period. The Cochrane Handbook recommends hand-searching as a useful adjunct to searching electronic databases because not all trial reports are included in electronic databases and, even if these reports are indexed, they might not use relevant search terms in the titles or abstracts or might fail to include search terms that allow them to be easily identified [6].

We noted a few cases of systematic reviewers who performed hand-searches and reported them well. For example, Mohan and Silverberg reported that “We also hand-searched 12 journals: Acta Dermato-Venereologica (1998–present), Archives of Dermatological Research (1975–present), JAMA Dermatology (formerly Archives of Dermatology; 1960–present), British Journal of Dermatology (1998–present), BMJ (1840–present), Clinical and Experimental Dermatology (1997–present), Dermatology (1894–present), Journal of Investigative Dermatology (1998–present), Journal of the American Academy of Dermatology (1979–present), JAMA (1883–present), Lancet (1991–present), and the New England Journal of Medicine (1928–present)” [9]. Such statements allow readers to easily determine that an actual hand-search had been conducted as well as the specific journals and time periods examined.

Based on our findings, we encourage reviewers to move away from manual inspection of reference lists due to the potential bias that this creates. Hand-searches, however, nicely supplement database searches and should be considered when feasible. Statements regarding such searches should be clear, be complete, and avoid vague language.

Biography

Matt Vassar, PhD, matt.vassar@okstate.edu, Office of Institutional Research and Analytics; Paul Atakpo, BS, patakpo@okstate.edu; Melissa J. Kash, AHIP, Melissa.kash@okstate.edu; Oklahoma State University Center for Health Sciences, 1111 West 17th Street, Tulsa, Ok 74107