Impact of librarians on reporting of the literature searching component of pediatric systematic reviews

Abstract

Objective

A critical element in conducting a systematic review is the identification of studies. To date, very little empirical evidence has been reported on whether the presence of a librarian or information professional can contribute to the quality of the final product. The goal of this study was to compare the reporting rigor of the literature searching component of systematic reviews with and without the help of a librarian.

Method

Systematic reviews published from 2002 to 2011 in the twenty highest impact factor pediatrics journals were collected from MEDLINE. Corresponding authors were contacted via an email survey to determine if a librarian was involved, the role that the librarian played, and functions that the librarian performed. The reviews were scored independently by two reviewers using a fifteen-item checklist.

Results

There were 186 reviews that met the inclusion criteria, and 44% of the authors indicated the involvement of a librarian in conducting the systematic review. With the presence of a librarian as coauthor or team member, the mean checklist score was 8.40, compared to 6.61 (p<0.001) for reviews without a librarian.

Conclusions

Findings indicate that having a librarian as a coauthor or team member correlates with a higher score in the literature searching component of systematic reviews.

There is ample literature assessing the quality of systematic reviews across many disciplines [1–16], and a common theme that has emerged from a number of these studies has been the need for improving the conduct and reporting of systematic reviews. Many studies have advocated for and described various roles that librarians and information professionals could play on a review team [17–26]. Several standards and organizations also suggest that a librarian or information professional be involved in the review process [27–30]. Documents produced by the Campbell Collaboration and a recent Canadian Institute of Health Research (CIHR) Knowledge Synthesis grant competition strongly recommend that an information professional or librarian be included among the members of a review team [31, 32].

However, to date, there has been very little empirical evidence on whether the presence of a librarian or information professional on a systematic review team contributes to the quality of a systematic review. Golder et al. found that only a very small percentage of reviews reported their search strategy with enough detail to be reproducible, and of those with reproducible searches, nearly half employed an information professional [33]. In the same study, the authors noted that literature searches performed by information professionals tended to be carried out using more resources than those without. More recently, it was suggested that the involvement of a librarian not only helps with selection of databases and other sources, but also with the development of a search strategy to retrieve eligible studies [34]. In a recent article, Rethlefsen et al. showed that librarian participation was significantly associated with search reproducibility and better reporting in general internal medicine systematic reviews [35]. To our knowledge, no studies have surveyed the authors of systematic reviews to compare the literature searching component of systematic reviews with and without librarian involvement. This study is aimed at addressing this gap in the literature.

METHODS

This study received approval from the Health Sciences Research Ethics Board at Western University.

The study did not differentiate between librarians and information professionals or specialists. Therefore, for the balance of this report, where reference is made to a “librarian” signifies either a librarian or information professional or specialist.

Development of the assessment instrument

A reporting methodology checklist was created as the assessment instrument (Appendix A, online only) by adapting the Institute of Medicine (IOM) standards for systematic reviews. The IOM standards are mapped to the methodological advice of the Agency for Healthcare Research and Quality Effective Health Care Program, Centre for Reviews and Dissemination, and the Cochrane Collaboration [29]. Although not validated, the checklist is thought to be reliable because it is based on the IOM standards that are generally considered methodologically sound.

The IOM standards provide specific guidelines for reporting systematic reviews (standard 5.1 and, specifically, element 5.1.6). However, additional elements of the standards were incorporated into the checklist to allow more detailed evaluation of the literature search reporting. Elements from IOM standards 2.6 (“Develop a systematic review protocol”), 3.1 (“Conduct a comprehensive systematic search for evidence”), and 3.2 (“Take action to address potentially biased reporting of research results”) were included in the checklist. Element 3.1.3 (“Use an independent librarian or other information specialist to peer review the search strategy”) was purposely excluded because the peer review of search strategies is a relatively new recommendation, and many methodological guides for systematic reviews do not mention peer review of search strategies [36].

The reporting score, which is the sum of the elements reported for each of the reviews, was used as a surrogate for the rigor of the literature search methodology reporting.

Collection of the systematic reviews

To have a manageable sample, we decided to choose one area of medicine that covered a broad range of medical topics in a well-defined discipline, pediatrics. The sample consisted of systematic reviews published between 2002 and 2011, representing a decade of reviews in the top-twenty pediatrics journals, ranked by impact factor, according to the 2010 science edition of Journal Citation Reports (Thomson Reuters). This time period saw the introduction of three evaluation instruments: the measurement tool for the “assessment of multiple systematic reviews” (AMSTAR) [37], PRISMA [38], and most recently, the standards reported by IOM [29]. Of note, the QUOROM statement [39] and MOOSE [40] preceded this period by several years. The sample was retrieved from the Ovid MEDLINE In-Process & Other Non-Indexed Citations, Ovid MEDLINE Daily, and Ovid MEDLINE (1946 to present) databases, according to the strategy reported in online only Appendix B. Only those articles that were described by their authors as systematic reviews (i.e., in the title, abstract, or methods) or that had an identifiable methodology section consistent with a systematic review were included in this study.

Collection of additional data from systematic review authors

Some systematic reviews in the sample did report whether or not a librarian was involved. To get a more complete understanding of the role assumed and the duties performed by the librarian, a questionnaire was created and sent by email to the corresponding authors of each systematic review article in the data pool in June and July, 2013. The questionnaire was resent by email to authors where no reply was received in October and November 2013. Questionnaire replies were received from corresponding authors until January 2014 (Appendix C, online only).

Pilot tests

We conducted pilot tests of the assessment instrument on samples of systematic reviews that were not part of our study sample. These tests allowed us to create a guide (Appendix A, online only) for applying the checklist, identifying and refining elements with unacceptable inter-reviewer agreement, and calibrating the 2 reviewers evaluating the retrieved reviews. To determine inter-rater reliability, we calculated Cohen's kappa and selected a kappa score of ≥0.70 as the acceptable level of inter-rater reliability.

Assessment of the systematic reviews

All reviews in the data pool were scored on their reporting practices for the literature searching component of the systematic review. The scoring was conducted independently by 2 of the study authors (Costella and Torabi). Each scored 60% of the articles in the pool, with blinded duplicate scoring performed for 20% of the articles (n=99).

Data analysis procedures

The overall reporting in each category was expressed as the mean value of the total score, and the analysis of variance (ANOVA) and pairwise comparisons were used for multiple comparisons. We used the Bonferroni correction [41] to adjust the p values and avoid the increased risk of a type I error due to multiple statistical tests. The frequency of reported methodological elements was recorded and expressed as a percentage. We compared the frequency of reported methodological elements for the “Without Librarian” group with either the “Librarian as Consultant” or “Librarian as Coauthor/Team Member” groups using Pearson chi-square analysis, and expressed as the risk ratio (RR) with 95% confidence interval (CI). Use of RR for the purpose of comparing various categories of articles has been described elsewhere [16, 42]. Two-tailed tests were performed, and p values <0.05 were considered statistically significant. All of the statistical analysis was conducted using Excel and SPSS software.

RESULTS

The search strategy returned 617 citations on April 25, 2012. Following review, we excluded 122 citations on the basis of not being systematic reviews, which left a total of 495 citations that we reviewed using the assessment checklist. Among the 495 articles reviewed, 11% mentioned librarian involvement with the study in the article. However, very often there was no indication of the role or to what extent the librarian was involved.

Author survey response

The response rate to the email survey was 38%. Of those authors who responded, 44% indicated that a librarian was involved with the research. Including the studies identified from the review and the survey responses, the final data pool of reviews with known librarian contribution was 22%.

Among the 53 original authors who reported librarian involvement in their articles, only 19 responded to the survey. Because we did not have full information about the librarian contribution to those studies (n=34) and the Cochrane Handbook for Systematic Reviews of Interventions recommends analyzing only the available data as one technique for dealing with missing data [28], we limited our analysis to data collected from the survey. The sample size for each category was still large enough to allow for statistical analysis. A flow diagram of the article workflow can be seen in online only Figure 1.

Figure 2.

The percentage of frequencies of databases reported in each category of systematic reviews (Without Librarian, Librarian as Consultant, Librarian as Coauthor/Team Member)

To understand the magnitude of the librarian contribution to the studies, we asked respondents to indicate the librarian's role and function in the study. Although the survey included 3 options for the librarian's role (a consultant, team-member, or coauthor), many of the responses indicated that the librarian acted as both a team member and coauthor, with very few responses for coauthor alone, so the studies selecting these 2 options were combined. Ultimately, the included systematic reviews were categorized into 1 of 3 groups: “Without Librarian,” “Librarian as Consultant,” and “Librarian as Coauthor/Team Member.” The results from survey question #2 (role) are summarized in Table 1. Survey question #3 dealt with librarian functions. Authors could select as many options as appropriate from the list of 9 functions, and 189 different functions were selected by the authors. The results from survey question #3 (functions) of the survey can be seen in Table 2. Of note, 57% reported that the librarian had more than 1 function.

Table 1.

Survey results for librarian's role (n=81)

Table 2.

Survey results librarian's functions (n=189

Reporting scores for methodological elements

To minimize potential bias, we scored the reviews without prior knowledge of the involvement of a librarian, unless it was specifically reported in the publication. The kappa statistics ranged from 0.64 to 1.00, indicating good inter-rater reliability for most of the reporting elements. However, element #5—the keywords, subject headings, and search terms that were used—had a kappa below 0.7 and was therefore excluded from the analysis. We calculated the sum of the reported elements for each article to obtain the reporting score for that article. Then, the mean reporting score was calculated for all articles, which was 7.04, 95% CI (6.77–7.32), n=186. All elements in the instrument were equally weighted. Table 3 shows the number of times each element was reported in the sample data.

Table 3.

Reporting scores of methodological elements in all systematic reviews

The most significant differences were observed when we compared reviews in the “Librarian as Coauthor/Team Member” and “Without Librarian” categories. The presence of a librarian as a coauthor/team member was associated with better reporting scores in various components of the methodology, particularly the flow diagram of the study inclusion process, the date that the search was updated, the full search strategy, the use of subject-specific and regional bibliographic databases for data collection, searches of gray literature, and web searches. Although the mean reporting score for reviews “Without Librarian” and those with “Librarian as Consultant” was not statistically significant, it was higher for the systematic reviews with “Librarian as Consultant.” When the two groups of “Librarian as Consultant” and “Without Librarian” were compared, a statistically significant difference was only observed in the proportion of reviews that reported their search strategies and the use of subject-specific and regional bibliographic databases for data. All data for this analysis are summarized in Table 4.

Table 4.

Reporting scores of methodological elements in each category of articles (n=186)

Specific characteristics of included systematic reviews

The reporting of a standard and the number of databases used in all were recorded and compared. Table 5 summarizes the overall score for each category, based on the use or lack of use of a standard. While 15% of the reviews without librarian contribution used a standard, 18% of reviews with a “Librarian as Consultant” and 16% of reviews with a “Librarian as Coauthor” reported the use of a standard. Of interest is that the use of a standard might slightly impact the overall mean score of reporting. However, there were no significant differences in total score between those who used the standard and those who did not, regardless of category.

Table 5.

Reporting scores based on the use of a standard in each category

The results also suggest that librarian involvement is associated with an increase in the number of databases searched and prevention of authors searching only one database for article collection. Two reviews did not report the number of databases they used. The raw data are presented in online only Table 6, and the percentage of frequencies of databases reported in each category of systematic reviews are plotted in Figure 2.

Table 7.

Reporting scores based to the average number of databases used for data collection in each category

We investigated the number of databases (general and subject-specific) used in each study and compared them across each category of systematic reviews. In general, the systematic reviews without librarian involvement used fewer databases; the authors who consulted a librarian used more databases; and those reviews that had a librarian as a coauthor or team member used the highest number of databases. We further investigated the correlation between the number of resources that the authors used and the reporting score totals for all reviews, as well as the three systematic review groups (i.e., “Without Librarian,” “Librarian as Consultant,” “Librarian as Coauthor/Team Member”). There were significant positive correlations between the total number of resources used and the total reporting scores across all four groups. The most significant correlation was for the “Librarian as Coauthor/Team Member” group. Table 7 shows the comparison of overall reporting scores in each group of systematic reviews and the correlation with the average number of databases used for conducting the study.

We also looked at whether reporting the use of a standard might be influenced by the journals in which the studies were published. The instructions for authors pages of each of the 22 titles searched in this study were explored for mention of applying a specific guideline or standard for reporting. The findings of this investigation can be found in online only Table 8, but in summary, 35% of the journals referred to the need to adhere to a specific guideline. The most common one required was the PRISMA Checklist and Flow Diagram, followed by the MOOSE checklist, and several publishers directed authors to consult the EQUATOR Network for guidelines [43].

DISCUSSION

The present study investigated the involvement of librarians in conducting systematic reviews and whether their participation might have an impact on the reporting of the literature search component of the systematic review process.

In a recent article, Rethlefsen et al. showed that librarian participation was significantly associated with search reproducibility and better reporting in general internal medicine systematic reviews [35]. Their findings were very similar to the findings of this study. They demonstrated that systematic reviews have greater compliance with IOM standards (eight of the thirteen elements of the standard) when a librarian coauthored the systematic review. We did not review the quality of search strategies in our sample because many factors contribute to the development of a final search strategy. Our methodology did not allow us to assess those various factors, including research team dynamics, level of collaboration, and the magnitude of librarian contribution to the resulting search strategy.

Our goal in applying the assessment instrument was to determine, as objectively as possible, whether an element was or was not reported in the publication in a transparent and reproducible manner. With two authors assessing each of the captured studies, identifying the reporting of keywords used (reporting element #5) was an ongoing source of difficulty. Despite multiple attempts to refine the evaluation criteria for this element, we were unable to achieve consistently acceptable inter-reviewer agreement, so much so that it was left out of our results. However, we feel that it is important to bring this situation to the attention of all systematic reviewers, teams, and publishers, with or without the presence of a librarian. Readers are at a disadvantage in trying to appraise a search strategy when complete information is not provided. From a reproducibility standpoint, the reader should be able to see explicitly which keywords and subject headings are used and should not have to make assumptions or contact authors for more detailed information. The situation is exacerbated by the fact that some reviews also failed to include a search strategy in the publication.

The authors of systematic reviews and meta-analyses must provide a complete picture of literature search methods and search strategies because a systematic review should be reproducible [44]. Our findings show that a librarian can greatly facilitate the reporting practices for some of the methodological elements, such as the flow diagram of the study inclusion process, the date that the search was updated, the full search strategy, the use of subject-specific and regional bibliographic databases for data collection, and searches of the gray literature. Most of the remaining methodological elements in our instrument were reported consistently, regardless of the presence of librarian involvement (e.g., language restrictions selection criteria, gathering unpublished data/information). On the other hand, some elements were poorly reported by the researchers (e.g., forward citation tracking), whether or not there was librarian involvement. Seven elements showed overall reporting at less than 50%, including the date the search was updated, search strategy, forward citation tracking, gray literature searching, gathering of unpublished data or information, hand-searches of selected journals and conference abstracts, and web searches. This finding correlates with previous studies that looked at similar elements as our study [45, 46]. Our results show that there is still room for improvement in reporting practices.

A previous study showed that the number of databases searched for conducting systematic reviews of adverse events has increased since 1994. The same study reported that searches of only MEDLINE or only one database are declining; however, the number of non-database sources searched throughout the studied time period remained constant [47]. A similar study reported the increase in the number of databases for conducing meta-analysis in leading journals since 1993 [48]. Our findings demonstrate that librarian involvement is associated with authors searching more databases. All systematic reviews with a librarian as a coauthor used more than one database. The AMSTAR checklist specifically requires the use of at least two electronic databases for retrieving eligible studies [37]. Also, with the presence of a librarian as a consultant or a coauthor, more studies searched subject-specific or regional databases and gray literature resources. Usually librarians are more informed about the existence of these sources of information and can direct researchers on how to access and use them. This is particularly true for gray literature sources, which can be difficult to identify and navigate.

Our survey results demonstrated that collaboration with librarians was still quite low in conducing systematic reviews. Among the functions contributing intellectual content for a systematic review, only 28% (n=52) of reviews consulted a librarian for resources and strategies, 15% (n=28) asked a librarian to create the search strategy completely, 14% (n=27) asked a librarian to review the search strategy, 13% (n=24) asked a librarian to compile resources for searching, and 7% (n=14) asked a librarian to draft sections of the manuscript. Possible reasons for these results might be that the authors of systematic reviews were not familiar with the extent to which they could collaborate with a librarian or they did not have access to a health sciences library or contact with a librarian.

Creation of optimal search strategies for systematic reviews should not be taken lightly. Literature searching involves an intellectual and iterative process and a good knowledge of databases [49]. The role of a librarian in conducting systematic reviews has been discussed in the literature [34].

Acknowledging that information reported in a systematic review is affected by a number of interdependent factors, it can be argued that responsibility for the overall quality of a published review is shared amongst all stakeholders in the process from idea conception through to final dissemination. Moher et al. suggest that the situation might be improved if authors and publishers would accept and adhere to evidence-based reporting guidelines [38]. This fact is borne out in our study by looking at the number of journals in our sample that required adhering to a specific checklist. As the number of published meta-analyses is increasing, a routine checklist for scientific quality should be used in the peer-review process to ensure methodological standards for publication [50]. Liberati et al. suggest that publishers should find ways to make search strategies accessible to readers [51]. Consumers of information need to assess the findings of a systematic review against the comprehensiveness and rigor of the methods used to conduct the review, including the identification of studies. It is difficult to make these assessments if the information is incomplete, ambiguous, or absent.

Limitations

We decided, for objectivity and reproducibility reasons, to obtain our sample from systematic reviews published between 2002 and 2011, representing a decade of reviews, in the top 20 pediatrics journals, ranked by impact factor, which is a proprietary measure specific to Journal Citation Reports that is derived from citation data extracted from Web of Science. As such, the possibility exists that some high-profile pediatrics journals might have been missed simply because they are not indexed in this database. Even for those journals that were identified, our sample might have included titles where publication of systematic reviews fell outside the scope of that journal. For example, there was only one systematic review identified for the journal Seminars in Fetal and Neonatal Medicine. The search strategy used to collect our sample was not intended to identify all of the systematic reviews in the twenty journals selected, which would have been outside the scope of the study, but to obtain a sample representative of the population we were using for the study.

Some characteristics of the assessment instrument may impose limitations on the study. The assessment instrument assigned equal weight to each element in order to reduce subjectivity. For example, lack of one element might prevent the reproducibility of the article or failure in searching gray literature for one systematic review might not be as problematic as it might be for another systematic review. Also, the instrument elements were scored using a binary response (yes/no), which does not give a credit for partial reporting of elements.

Because we used self-reported data from the authors of systematic reviews, there is an inherent limitation in determining whether there is consistent interpretation and selection of survey choices. For example, in our choices for roles, what defined librarian as consultant could vary from one respondent to the next. Therefore, high-level generic terms were used in an attempt to minimize interpretation inconsistencies, as well as aggregation of the categories of librarian as coauthor and librarian as team member in the analysis of the data. Also, since not all authors that reported librarian involvement in their publications responded to the survey, we decided to limit our analysis to data collected only from the survey.

The present study investigates the reporting of the literature search component of the systematic review process. Inconsistency in the reporting practices and insufficiency of information provided in the systematic reviews challenges the reproducibility of most studies published in some of the highest impact factor journals. Our findings demonstrate a positive association between the involvement of librarians, as team members or coauthors, and the reporting of the literature search component.

Biography

Deborah Meert, MLIS (corresponding author), dmeertiwi@uwo.ca, Head, Research & Instructional Services, Allyn & Betty Taylor Library, University of Western Ontario, 1151 Richmond Street, London, ON, Canada, N6A 3K7; Nazi Torabi, MSc, MLIS, nazi.torabi@mcgill.ca, Liaison Librarian, Schulich Library of Science and Engineering, McGill University, 845 Sherbrooke Street West, Montreal, QC, Canada, H3A 0G4; John Costella, DDS, MSc, MLIS, jccostel@uwo.ca, Research and Instructional Librarian, Allyn & Betty Taylor Library, University of Western Ontario, 1151 Richmond Street, London, ON, Canada, N6A 3K7