The effect of librarian involvement on the quality of systematic reviews in dental medicine

Jana Schellinger; Kerry Sewell; Jamie E Bloss; Tristan Ebron; Carrie Forbes

doi:10.1371/journal.pone.0256833

. 2021 Sep 1;16(9):e0256833. doi: 10.1371/journal.pone.0256833

The effect of librarian involvement on the quality of systematic reviews in dental medicine

Jana Schellinger ¹, Kerry Sewell ², Jamie E Bloss ^2,^*, Tristan Ebron ³, Carrie Forbes ²

Editor: Lisa Susan Wieland⁴

PMCID: PMC8409615 PMID: 34469487

Abstract

Objectives

To determine whether librarian or information specialist authorship is associated with better reproducibility of the search, at least three databases searched, and better reporting quality in dental systematic reviews (SRs).

Methods

SRs from the top ten dental research journals (as determined by Journal Citation Reports and Scimago) were reviewed for search quality and reproducibility by independent reviewers using two Qualtrics survey instruments. Data was reviewed for all SRs based on reproducibility and librarian participation and further reviewed for search quality of reproducible searches.

Results

Librarians were co-authors in only 2.5% of the 913 included SRs and librarians were mentioned or acknowledged in only 9% of included SRs. Librarian coauthors were associated with more reproducible searches, higher search quality, and at least three databases searched. Although the results indicate librarians are associated with improved SR quality, due to the small number of SRs that included a librarian, results were not statistically significant.

Conclusion

Despite guidance from organizations that produce SR guidelines recommending the inclusion of a librarian or information specialist on the review team, and despite evidence showing that librarians improve the reproducibility of searches and the reporting of methodology in SRs, librarians are not being included in SRs in the field of dental medicine. The authors of this review recommend the inclusion of a librarian on SR teams in dental medicine and other fields.

Introduction

Systematic reviews (SRs) serve an important role in evidence-based practice across the health professions. Initially established within the health disciplines by the founding of the Cochrane Collaboration in the early 1990s, in the ensuing decades SRs have been produced at an exponential rate. The increase in the number of published SRs has not necessarily been marked by an increase in quality. Studies of published SRs indicate major shortcomings, ranging from the failure of reviewers to adequately report methodology [1] to inadequate adherence to standards [2].

In response to these evaluations, international groups have created guidance statements for SRs and meta-analyses. The primary reporting guideline, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, comprises 27 required reporting items and a flow diagram to track the review process [3]. Although widely accepted, meta-research indicates that PRISMA adherence is spotty. Reviews that mention use of PRISMA demonstrate no better adherence to guidelines than reviews without mention of reporting standards [4].

The methodological quality of SRs can be highly variable across health professions [2, 5, 6], despite the widespread use of international guidelines on SRs such as those from the Cochrane Collaboration, the National Academy of Medicine, AHRQ, MOOSE, and others. Many studies of methodological and reporting quality in SRs and meta-analyses come from the domains of medicine, leaving a paucity of knowledge of these issues across other health professions such as dental medicine. More limited studies nonetheless indicate that shortcomings in reporting and methodological quality extend to reviews published in dental medicine [7].

PRISMA use in dentistry SRs varies greatly, promulgated by lack of required reporting standards in journals [8, 9]. Even published dental SRs that assert the use of PRISMA guidelines are often inadequate in adherence, with screening methods underreported and inclusion criteria and data extraction methods missing [10]. Analyses of the methodological quality of SRs in dentistry and endodontics document equally low methodological quality, finding poor reporting and conduct of SRs [10]. Poor adherence to SR guidance is found as early as the registration process for orthodontics SRs [11]. Adherence to other markers of methodological rigor in dental SRs is equally problematic, with issues identified in screening and data extraction processes, inclusion of gray literature, use of quality assessment, and inclusion of a risk of bias assessment [7]. An examination of methodological quality of meta-analyses of periodontal treatment of glycemic control in diabetic patients revealed only 33% of the included meta-analyses met criteria for high methodological quality [12].

The quality of searches underlying dental medicine SRs also suffers from shortcomings. An assessment of the reproducibility of search strategies for 530 dental medicine SRs found that none had complete reporting of the search strategies and selection process [1]. Another study examining SRs in prosthodontic and implant journals found that fewer than 5% of the studies employed ‘systematic’ searches, with issues identified in the rigor and sensitivity of the searches and lack of inclusion of gray literature [13]. Dental medicine SRs are not unique in search strategy shortcomings. A study of 137 SRs published January, 2018, and identified through the PubMed database indicated a high level of search errors which adversely affected the information retrieved from the searches [14]. A study of Cochrane systematic reviews in 2002 contained similar errors [15].

Various influential organizations supporting SRs and meta-analyses strongly recommend that a librarian or information specialist be included in the research team as a means of addressing such issues. Notably, the Cochrane Handbook for Systematic Reviews of Interventions and the 2011 guidelines on SRs from the Institute of Medicine (now the National Academy of Medicine) strongly recommend including a librarian or information specialist from the start of the review process [16, 17]. Other guidelines are less specific on timing, but suggest that information specialists be involved [18–21]- recommendations that stem from evidence indicating that the inclusion of librarians in SR teams can help improve the quality of searches, the quality of methodological reporting, and the inclusion of gray literature in the search [22–26]. Additional guidance, directed specifically toward librarians and information specialists, suggests peer review of SR searches and provides the Peer Review of Electronic Search Strategies (PRESS) instrument for evaluation of searches in hopes of improving quality [27]. Although librarian roles in SRs have generally focused on ensuring a rigorous and reproducible search, librarians may be involved in numerous parts of the review [28]. Rethlefsen, et al, take librarian inclusion even further, noting that librarian co-authors were correlated with improved reporting of search strategies in SRs [26]. When examining the International Committee of Medical Journal Editors (ICMJE) recommendations of who should be a named author [29], Rethlefsen’s assertion that librarians should be named co-authors makes sense for those SRs that have librarian participation in formulating a search and reporting methodology, among other tasks.

Evidence of the effect of librarian inclusion on the quality of SRs has come from studies utilizing SRs from clinical medicine. Published literature has not documented the extent and effect of the inclusion of librarians on SRs in dental medicine. Given the recognized need for the inclusion of librarians [30], this study aims to address this gap in the literature. The research questions addressed are:

Does having a librarian involved improve search strategy reproducibility?
When a librarian is involved, does having a librarian as a coauthor improve the reproducibility of the search more than just having the librarian acknowledged?
Does librarian involvement in the review improve reporting quality?
Does the involvement of a librarian make it more likely that grey literature is searched?
Does having a librarian involved lead to the use of at least 3 databases searched?

Methods

Database search and selection of SRs

This study utilized published SRs and meta-analyses from the twelve most highly cited dental medicine journals. The top ten journals were selected from two sources: Journal Citation Reports (JCR) and Scimago. As the top ten journals differed between the two journal indices, a compound list of the top ten journals from each database was created and duplicates were removed. The list of journals and the number of articles examined from each journal can be found in S1 Appendix.

One author (KS) conducted a search for SRs in each of these journals within the PubMed database on June 18, 2018. The search string consisted of the PubMed journal title abbreviations combined with the SR search string in the PubMed Clinical Queries tool (prior to the 2020 update to PubMed). The combined search is available in S2 Appendix. No date restrictions were used, but results were limited to English language articles.

The search results were loaded into Rayyan for initial screening. Two authors (JS, KS) independently reviewed the articles to identify SRs or meta-analyses. An article was considered a SR if it met at least one of the following inclusion criteria:

Article title or abstract indicated the study was a SR or meta-analysis
The authors described the use of a systematic search
The authors described the use of multiple databases to identify all relevant literature on a topic

Exclusion criteria:

Article explicitly described the methodology as a non-SR expert-level review, such as a scoping review or narrative review, even if a systematic search was employed
Article described qualitative or quantitative study methods for human subjects (cohort studies, randomized controlled trials, retrospective chart reviews, etc.)

Conflicts were resolved by discussion. After review, 430 articles were removed, leaving 913 articles included in the analysis.

Method for reproducibility assessment and data extraction

Articles were randomized using a random sequence generator in Excel and divided into two sets of articles. The two sets were assigned to pairs of reviewers (JS, KS, TE, CF). Assessment was blinded and performed in duplicate using an assessment form in Qualtrics. Any conflicts were resolved by a third reviewer not involved in the initial assessment (JB, JS, KS).

The assessment instrument was adapted from one used by Rethlefsen and colleagues (2015) in their study of the effect librarian co-authorship had on the quality of reported search strategies in internal medicine SRs. Questions were adapted to yes/no format wherever possible. Questions focused on reporting areas that are recommended in the PRISMA 2009 recommendations. The adapted survey consisted of 18 questions [S3 Appendix]. This assessment survey was used primarily to determine which SRs had reproducible search strategies and which SRs included a librarian or information specialist (subsequently collectively referred to as ‘librarians’) in any capacity. These primary data points allowed for further analysis of SRs with searches deemed reproducible and SRs with librarian participation. To determine if a paper included a librarian as author or in acknowledgement, reviewers examined names and titles of all included authors and those mentioned in acknowledgements. Where an author had an MLS, MLIS, MSIS or other library degree, or where the text of the methods section or acknowledgement mentioned a librarian, reviewers automatically recorded the librarian’s participation as an author or as acknowledged. Where degrees or titles were not provided for authors or those acknowledged, the reviewers made every attempt to determine the author’s credentials, from searching the author’s name to investigating any affiliated institutions. Other data points of interest in this first analysis included reporting methods listed where the search information was located, if inclusion and exclusion criteria were reported, number of reviewers who conducted each part of the SR screening, the number of articles found in database searching and at each level of screening, and whether a risk of bias assessment was performed. To limit bias, articles were identified using PubMed Identification (PMID) numbers and all articles went through this first assessment before the second assessment was conducted.

A reproducible search was one that could be copied and pasted into the search bar with minimal changes and produce similar results to those reported in the published SR. Reasonable leeway for reproducibility was considered to be either the absence of a date filter or the inclusion of each concept group in a table, with clear direction on how to combine the concept groups. Where there were questions about the reproducibility of a search through visual assessment, the search was copied and pasted into the database mentioned in text to determine reproducibility. Any articles that used the PubMed database were analyzed using the version of PubMed that was available as default prior to 2020, as the new version of PubMed (2020-present) would provide different results. Searches were considered irreproducible if they did not: state the database the search was conducted in, specify how parts of the search were combined, or indicate what tags were attached to the search terms (i.e. mentioned the use of MeSH terms and keywords in description, but did not note MeSH terms in the provided search).

After analysis, 452 articles were determined to have reproducible search strategies.

A second evaluation instrument was used to further analyze articles with searches deemed reproducible [S4 Appendix]. The purpose of the second evaluation was to analyze the quality of the reproducible searches. Reviewers were blinded to all previous data points (except search reproducibility) and were unaware of which articles had librarian participation. Data points included the number of databases searched, names of databases searched, if searches were reproducible exactly as written or if they required manipulation, whether search terms were reasonably developed and complete, overall quality of the search (subjective), errors in the use of parentheses and brackets, and overall syntax errors. Reviewers also assessed whether gray literature was searched. For the purpose of this analysis, gray literature included conference abstracts, dissertations and theses, discipline-specific publications that are not indexed in major databases, trials registries, institutional repositories, and other sources outside the scope of major databases. The articles were divided into three sets and three authors (JS, KS, JB) were assigned two of the three sets to review so that each article was independently reviewed by two reviewers. The evaluation was performed using Qualtrics. Conflicts were resolved by a third author. In cases where all three authors disagreed, conflicts were resolved by discussion. To limit bias, articles were again identified using PMID numbers. Reviewers were instructed to only examine the searches in this step and not to check authors or affiliations. Finally, reviewers were blinded to all data points from the first assessment survey.

Data analysis

Data were combined and coded in Excel and analyzed using SPSS statistical software version 26. Data visualizations were created in Excel and Google Spreadsheets. Data is available via OSF (https://osf.io/qsua8/files/?view_only=ea7a550400f4403eb280b1d383ec11fa). Statistics were collected and analyzed as planned and included descriptive statistics for all data points. Crosstabs with Chi-Square tests for significant differences were used to analyze all data points that included the dependent variable of reproducible searches. Independent variables included whether a librarian assisted or co-authored a review, whether gray literature was searched, number of reviewers who examined titles, abstracts, and full texts, whether reviewers were blinded to each others’ work, and whether or not a risk of bias assessment was conducted. Bayesian ANOVA of Likert data (overall quality of the search–subjective score) was also conducted with Chi-Square tests for significance. Any significance was determined by p value < .05 [Table 1].

Table 1. Data analysis.

Research Question	Associated Questionnaire Question(s) (see S3, S4 Appendices)	Independent Variable(s) (Code) With Response Code(s)	Dependent Variable(s) (Code) With Response Code(s)	Variable Type	Analysis
Does having a librarian involved improve search strategy reproducibility?	Questionnaire 1 Question 3, 5 Questionnaire 2 Question 3	Search Reproducibility (SRCHREPROD) 1 = Yes 2 = No	Librarian Participation (LIBPARTICP) 1 = Librarian/Information specialist as author 2 = Librarians/information specialist is mentioned in text or acknowledged 3 = No or unclear	Categorical/ Nominal	Crosstab with Chi Square
When a librarian is involved, does having a librarian as a coauthor improve the reproducibility of the search more than just having the librarian acknowledged?	Questionnaire 1 Question 3, 5 Questionnaire 2 Question 3, 5	Search reproducibility (SRCHREPROD) 1 = Yes 2 = No	Librarian participation (LIBPARTICP) 1 = Librarian/Information specialist as author 2 = Librarians/information specialist is mentioned in text or acknowledged 3 = No or unclear	Categorical/ Nominal	Crosstab with Chi Square
Does librarian involvement in the review improve reporting quality?	Questionnaire 1 Question 3–18 Questionnaire 2 Question 3, 4, 6, 7	Location of search Info. (SRCHLOCINF) 1 = Full search strategy in text 2 = Description in text 3 = In appendix–in article 4 = In appendix–web only 5 = No search information 6 = Contact author for full search strategies 7 = Other All items below coded as (unless otherwise noted): 1 = Yes 2 = No Search reproducibility (SRCHREPROD) Inclusion criteria listed (INCCRITINC) Exclusion criteria listed (EXCCRITINC) Were researchers blinded to each others’ responses (BLINDINGYN) Was the number of reviewers title/abstract Reported (REVTANUMRP) Number of reviewers title/abstract (NUMREVWRTA) Was the number of reviewers full text reported (REVFTNUMRP) Number of reviewers full text (NUMREVWRFT) Did they report number of studies included (REVSTNUMRP) Did they report number of duplicates (RPDUPREMVD) Did they report number of titles/abstracts screened (RPTIABSCRN) Did they report number of full texts screened (RPFLTXSCRN) Did they report number of studies included (RPARTFULIN) Was risk of bias assessed (RSKBIASPRF) 1 = Yes 2 = No or unclear	Librarian participation (LIBPARTICP) 1 = Librarian/Information specialist as author 2 = Librarians/information specialist is mentioned in text or acknowledged 3 = No or unclear	Categorical/ Nominal	Crosstab with Chi Square
Does the involvement of a librarian make it more likely that grey literature is searched?	Questionnaire 1 Question 3, 5 Questionnaire 2 Question 6	Did the reviewers conduct a gray literature search? (DIDGREYLIT) 1 = Yes 2 = No	Librarian Participation (LIBPARTICP) 1 = Librarian/Information specialist as author 2 = Librarians/information specialist is mentioned in text or acknowledged 3 = No or unclear Search Reproducibility (SRCHREPROD) 1 = Yes 2 = No
Does having a librarian involved lead to the use of at least 3 databases searched?	Questionnaire 1 Question 3, 5 Questionnaire 2 Question 3, 4	Number of Databases Searched (NUMDBSRCHD) Where is search information located (SRCHLOCINF_	Librarian Participation (LIBPARTICP) 1 = Librarian/Information specialist as author 2 = Librarians/information specialist is mentioned in text or acknowledged 3 = No or unclear Search Reproducibility (SRCHREPROD)	Categorical/ Nominal	Crosstab with Chi Square

Open in a new tab

Indicates how data was analyzed for each question of this review.

Results

Librarian participation

Few published dental medicine SRs reported the inclusion of a librarian in any capacity on the team. Of the 913 SRs examined, 2.5% (n = 23) included a librarian as a co-author, 9% (n = 82), mentioned or acknowledged a librarian, and in 88.5% of the SRs, inclusion of a librarian in the SR was not reported.

Inclusion of search strategy

Inclusion of a librarian on the team is associated with improved reporting of at least one search strategy [Fig 1], particularly when a librarian is included as an author. Of the 23 articles with a librarian co-author, 87% (n = 20) included at least one full search strategy in the text or appendix. The remaining 13% (n = 3) provided a description of the search but not a full search strategy. Of the 82 articles that acknowledged or mentioned a librarian, 51% (n = 42) included at least one full search strategy in text or appendix, 43% (n = 35) described the search, and 6% (n = 5) did not include a search strategy or description. Of the 808 articles that did not mention or acknowledge librarian participation, 50% (n = 401) included at least one full search strategy, 46% (n = 372) described the search, and the remaining 4% (n = 35) did not include a search strategy or a description.

Search quality

Subjective analysis of overall quality of the search indicated that librarian participation on the review team increased the quality of the search. Each reviewer provided a score (1–5). When scores were within 2 points, average score was calculated. When there was a spread of more than 2 points, a third reviewer scored the article and all 3 scores were averaged. Based on Bayesian ANOVA assessment of Likert quality scoring 1(low) - 5(high), when a librarian was a coauthor, the mean score was 3.5. When a librarian was acknowledged, the mean score was 3.4. When a librarian was not mentioned, the mean score was 2.814.

Reporting in all reviews

Information reported in reviews

Among the 913 reviews, only 13.3% (n = 121) of the reviewers reported and searched for grey literature. We conducted initial analysis of five additional reporting metrics: reporting of inclusion and exclusion criteria, blinding during the review, reporting the number of title/abstract reviewers, and reporting the number of full text reviewers. On average, only 1.2 of these items were included. Of the 913 articles reviewed, only 33 (3.6%) included all five reporting factors. Almost half (46%) did not report any of these metrics [Fig 2].

Fig 2 — Illustrates the percent of all articles examined that included the information listed. Grey Literature was reported in 56.5% of reviews with a librarian co-author, 69.5% of reviews with an acknowledged librarian, and 72% of articles with no mention of a librarian. Inclusion criteria was reported in 95.7% of reviews with a librarian co-author, 100% of reviews with an acknowledged librarian, and 94.4% of articles with no mention of a librarian. Exclusion criteria was reported in 78.3% of reviews with a librarian co-author, 75.6% of reviews with an acknowledged librarian, and 74.6% of articles with no mention of a librarian. Whether authors were blinded to each other’s work was reported in 82.6% of reviews with a librarian co-author, 85.4% of reviews with an acknowledged librarian, and 71% of articles with no mention of a librarian. Number of reviewers who examined titles and abstracts of articles was reported in 69.6% of reviews with a librarian co-author, 79.3% of reviews with an acknowledged librarian, and 67.1% of articles with no mention of a librarian. Number of reviewers who examined full texts of articles was reported in 87% of reviews with a librarian co-author, 81.7% of reviews with an acknowledged librarian, and 66.8% of articles with no mention of a librarian.

Databases

Librarian participation was associated with at least 3 databases searched, as the Cochrane Handbook for Systematic Reviews of Interventions recommends the use of the following databases in Systematic Reviews: CENTRAL, MEDLINE, and Embase (where available as this is a subscription database) [17]. Of the 913 SRs, only 42% (n = 386) specified the number of databases searched. Reviews that included a librarian co-author included a mean average of 5.2 databases searched and 88% (n = 14/16) reported searching at least three sources. Reviews that acknowledged or mentioned a librarian included an average of 4.7 databases and 91% (n = 31/34) reported searching at least three sources. Reviews that were unclear about librarian participation included an average of 3.6 databases searched and 71% (n = 240/336) reported searching in at least three sources.

When examining only the reproducible searches that reported the number of databases searched (n = 385), the numbers are similar. When a librarian was a co-author, 88% (n = 14/16) reported searching at least three sources. When a librarian was acknowledged, 91% (n = 31/34) reported searching at least three sources. When there was no report of a librarian involved, 71% (n = 239/335) reported searching at least three sources.

Reporting the numbers

We examined five metrics that are commonly reported within the methodology of a SR. A librarian co-author was associated with better reporting of numbers of articles identified through searching, duplicates removed, full texts screened, and articles included in the analysis. In only one category—number of titles/abstracts screened—did groups without librarian participation include better reporting [Fig 3]. On average, articles reported 1.1 of these metrics and only 31 (3.4%) of the 913 articles reported all five metrics. Over one third of the reviews (38%) reported none of these metrics.

Fig 3 — Illustrates the percent of articles that reported the following: number of articles identified from database searching, number of duplicates removed, number of titles and abstracts screened, number of full text articles screened, and the number of articles included in the final review. Each of these was broken down by whether a librarian was included as a co-author, if a librarian was acknowledged, or if there was no mention of a librarian.

Reproducibility

Librarian co-authors are associated with a higher rate of reproducible searches. There were 23 articles that listed a librarian as an author. Of those, 69.6% (n = 16) were reproducible. When a librarian or information specialist was acknowledged, 51.2% (n = 42) of the 82 articles were reproducible. Among the articles that did not acknowledge a librarian as an author or contributor, 48.8% (394) were reproducible.

Analysis of reproducible searches

The subset of articles with at least one reproducible search was further analyzed using SPSS statistical software version 26. Where multiple searches were reported, the first search strategy (usually MEDLINE via PubMed or Ovid) was analyzed. In articles where the first search strategy was not reproducible, the first reproducible search was analyzed.

Librarian assistance

Crosstabulation analysis revealed, of the 452 articles with reproducible searches, only 3.5% (n = 16) specified a librarian or information specialist as an author. In 9.3% (n = 42) a librarian was acknowledged. The other 87.2% did not specify the inclusion of a librarian.

Information reported in reproducible searches

Librarian co-authors were not associated with more frequent reporting of whether grey literature was searched or how many reviewers examined titles and abstracts of articles. Librarians were, however, associated with higher rates of blinding among reviewers, reporting the number of reviewers who examined full texts of articles, and risk of bias assessments on included articles [Fig 4].

Fig 4 — Within reproducible searches, illustrates the percent of articles that included the information listed. Grey Literature was reported in 37.5% of reviews with a librarian co-author, 40.5% of reviews with an acknowledged librarian, and 42.4% of articles with no mention of a librarian. Whether authors were blinded to each other’s work was reported in 81.3% of reviews with a librarian co-author, 83.3% of reviews with an acknowledged librarian, and 78.4% of articles with no mention of a librarian. Number of reviewers who examined titles and abstracts of articles was reported in 62.5% of reviews with a librarian co-author, 76.2% of reviews with an acknowledged librarian, and 76.6% of articles with no mention of a librarian. Number of reviewers who examined full texts of articles was reported in 87.5% of reviews with a librarian co-author, 81% of reviews with an acknowledged librarian, and 74.9% of articles with no mention of a librarian. Whether a risk of bias assessment was completed was reported in 81.3% of reviews with a librarian co-author, 88.1% of reviews with an acknowledged librarian, and 77.4% of articles with no mention of a librarian.

Search details

Librarian co-authors were associated with fewer mistakes in the use of brackets, Boolean operators, and proximity operators. Spelling was also slightly better with a librarian co-author. The inclusion of a librarian co-author was not associated with fewer errors in system syntax or line numbering [Fig 5]. Speculation about the reason for higher librarian mistakes in this section will be addressed in the discussion.

Fig 5 — Within reproducible searches, illustrates the percent of articles with errors in the following categories: Line numbers (only searches with line numbers were assessed), System syntax errors, Spelling errors, Errors in the use of proximity operators (only searches with proximity operators were assessed), Errors in the use of Boolean operators, Errors in the use of brackets.

When the search terms were examined, librarian co-authors were associated with fewer heading mistakes and irrelevant headings, fewer missing spelling variants, and about equal use of unwarranted limits when compared to other reviews; however, librarian co-authors were associated with more missing and irrelevant natural language terms and less optimal use of truncation [Fig 6].

Fig 6 — Within reproducible searches, illustrates the percent of articles with errors in the following categories: Use of MeSH or Emtree headings (only searches with headings were assessed), Inclusion of irrelevant MeSH or Emtree headings (only searches with headings were assessed), Missing natural language variants, Missing spelling variants (such as pediatric/paediatric), Inclusion of irrelevant natural language variants, Errors in the use of truncation (only searches with truncation were assessed), Inclusion of unwarranted limits (date, language, etc.).

Significance

Analysis of most of our data points in SRs did not demonstrate statistical significance (p < .05). This is likely due to the small sample size of SRs that included librarians in any capacity and the even smaller proportion of reproducible searches that included a librarian. We were, however, able to demonstrate statistical significance when examining the effect of a librarian on whether at least three databases were searched. Based on chi square testing, when a librarian was a co-author, it was 11 times more likely that at least 3 databases were searched than when a librarian was not a co-author with a p value of .003.

Discussion

Principal findings

There is very little evaluation of SRs in the field of dental medicine, however our findings are consistent with existing research from other disciplines. Despite guidelines from the Cochrane Collaboration, AHRQ, and CRD suggesting consulting a librarian on a systematic review search is valuable [3, 13, 16–19], inclusion of librarians in dental medicine SRs is extremely limited. Reporting of methodology in dental medicine is not ideal [7–10]. Our findings suggest that having a librarian as part of the review team is associated with improved methodological reporting of SRs, consistent with similar foundational studies in other disciplines, such as those by Koffel and Rethlefsen [23–27]. Although subjective, analysis of search quality indicates librarian co-authorship is also associated with a better-quality search, supporting prior research in medicine [23–26]. Librarian co-authors were associated with a higher inclusion rate of grey literature, although looking only at SRs with reproducible searches, librarian co-authors were not associated with a higher rates of grey literature inclusion. We notice numerous librarian errors in the search details section (Fig 5). We posit two possible reasons for errors in line numbers, system syntax, and proximity operators. Although statistical analysis did not focus on these data points, we suggest that SR teams with librarian inclusion were more likely to utilize a search strategy that included line numbers and proximity operators than SR teams without librarian assistance. This possibility should be examined further, as this could have skewed the data. Another possible reason for errors in this section is that librarians are more likely, because of their familiarity with a search or database, to neglect or overlook reporting certain data. More research is needed in this area as well, although librarians should be more vigilant about checking their wording in the methods section. Due to the small sample size and not knowing the extent of librarian roles, it is unclear whether librarian co-authorship is associated with improved reporting of numbers; however, when taken together, articles with librarian co-authorship and librarian participation performed overall better than those without a librarian.

Implications

A sensitive search in the SR process takes on special importance because it determines the set of articles on which all other parts of the review depend. As articles were collected prior to the publication of PRISMA-S guidelines in 2021 [31], those conducting SRs had only their knowledge of PRISMA 2009 guidelines and search methodology to guide them. Given the methodological knowledge and search expertise a librarian could add to a SR team, this raises questions of why they are not being utilized on dental SR teams. One issue could be the level of librarian support available at dental schools, private practices, or hospitals. For instance, the Medical Library Association (MLA) dental caucus only has 93 members while other caucuses have up to 500 members. There are 68 accredited dental schools in the United States, and 10 in Canada [32]. Even a designated librarian for a dental school may not be able to support all systematic reviews due to workloads. While there may be enough support for the 78 dental schools in the US and Canada, librarian assistance outside of academia may be problematic. Even within academia, not all librarians have formal SR training.

The explosion of publications of SRs of low quality is well documented [6]. As many authors have concluded, there is a lack of awareness in the broader field of dentistry on the methodology and execution of an SR and what is required for a truly sensitive search. When journals do not require use of reporting standards, the number of high quality reviews in dentistry research decreases [7]. The lack of reproducible searches found could indicate a lack of knowledge of what is needed to re-run a search, limiting the ability to validate and update the research.

Limitations

There were some limitations to this review. First, the rate of librarian assistance may be higher than reported in articles, and therefore, data may be skewed [23]. Second, most authors that acknowledged a librarian did not explain the librarian’s role. Due to the yes/no formatting of many questions, there is no allowance for degrees or levels of compliance; therefore, for example, an SR that was missing only a couple of keywords had the same result as one that was missing many. The subjective rating of the search quality was an attempt to adjust for this weakness; however, this is an area that would benefit from further study and analysis.

Although many data points were collected about the quality of reporting and search terms, overall quality of searches was determined through subjective rating. Reviewers limited bias where possible by being blinded to each other’s work. Articles were identified using PMID numbers within search instruments, and the two-step review process with librarian participation evaluated in the first step and search quality evaluated in the second allowed for blinding to which articles were deemed to have librarian participation when reviewers assessed search quality. The number of SRs was substantial; however, the number with reproducible searches and librarian participation was too small to run meaningful statistical significance analysis. Searches in SRs were evaluated prior to the release of PRISMA-S guidelines for searches [31] and PRISMA 2020 reporting guidelines [33]. The publication of these new guidelines may positively affect the quality of future searches and the quality of reporting in SRs, and should be investigated.

Conclusion

Overall findings suggest that involvement of a librarian is associated with more reproducible searches and improved methodological reporting in dental medicine SRs, though the association in the current review is not statistically significant. Efforts should be made to consult a librarian and include them as a co-author or acknowledge them in the article for their assistance in crafting sensitive, reproducible searches and for their knowledge of SR methodology and reporting standards.

Supporting information

S1 Appendix

(DOCX)

Click here for additional data file.^{(12.3KB, docx)}

S2 Appendix

(DOCX)

Click here for additional data file.^{(13.5KB, docx)}

S3 Appendix

(DOCX)

Click here for additional data file.^{(13.8KB, docx)}

S4 Appendix

(DOCX)

Click here for additional data file.^{(14KB, docx)}

Acknowledgments

The authors thank Dr. Hui Bian for her recommendations and expertise on assessment tool creation and statistical analysis. The authors also thank Dr. Julia Castleberry for her recommendations and expertise on statistical analysis.

Data Availability

All data is available via OSF: https://osf.io/qsua8/files/?view_only=ea7a550400f4403eb280b1d383ec11fa.

Funding Statement

The author(s) received no specific funding for this work.

References

1.Faggion CM Jr, Huivin R, Aranda L, Pandis N, Alarcon M. The search and selection for primary studies in systematic reviews published in dental journals indexed in MEDLINE was not fully reproducible. Journal of clinical epidemiology. 2018Jun1;98:53–61. doi: 10.1016/j.jclinepi.2018.02.011 [DOI] [PubMed] [Google Scholar]
2.Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Systematic reviews. 2017Dec;6(1):1–4. doi: 10.1186/s13643-016-0385-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 2009Jul21;6(7):e1000097. doi: 10.1371/journal.pmed.1000097 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Nawijn F, Ham WH, Houwert RM, Groenwold RH, Hietbrink F, Smeeing DP. Quality of reporting of systematic reviews and meta-analyses in emergency medicine based on the PRISMA statement. BMC emergency medicine. 2019Dec;19(1):1–8. doi: 10.1186/s12873-018-0218-x [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS medicine. 2016May24;13(5):e1002028. doi: 10.1371/journal.pmed.1002028 [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007Mar27;4(3):e78. doi: 10.1371/journal.pmed.0040078 [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Hua F, Walsh T, Glenny AM, Worthington H. Surveys on reporting guideline usage in dental journals. Journal of Dental Research. 2016Oct;95(11):1207–13. doi: 10.1177/0022034516657803 [DOI] [PubMed] [Google Scholar]
8.Sarkis-Onofre R, Cenci MS, Moher D, Pereira-Cenci T. Research reporting guidelines in dentistry: A survey of editors. Brazilian dental journal. 2017Feb;28(1):3–8. doi: 10.1590/0103-6440201601426 [DOI] [PubMed] [Google Scholar]
9.Bassani R, Pereira GK, Page MJ, Tricco AC, Moher D, Sarkis-Onofre R. Systematic reviews in dentistry: current status, epidemiological and reporting characteristics. Journal of dentistry. 2019Mar1;82:71–84. doi: 10.1016/j.jdent.2019.01.014 [DOI] [PubMed] [Google Scholar]
10.Sideri S, Papageorgiou SN, Eliades T. Are orthodontic systematic reviews registered a priori in PROSPERO?. Journal of orthodontics. 2017Dec;44(4):249–55. doi: 10.1080/14653125.2017.1370773 [DOI] [PubMed] [Google Scholar]
11.El-Rabbany M, Li S, Bui S, Muir JM, Bhandari M, Azarpazhooh A. A quality analysis of systematic reviews in dentistry, part 1: meta-analyses of randomized controlled trials. Journal of Evidence Based Dental Practice. 2017Dec1;17(4):389–98. doi: 10.1016/j.jebdp.2017.06.004 [DOI] [PubMed] [Google Scholar]
12.Hasuike A, Iguchi S, Suzuki D, Kawano E, Sato S. Systematic review and assessment of systematic reviews examining the effect of periodontal treatment on glycemic control in patients with diabetes. Medicina oral, patologia oral y cirugia bucal. 2017Mar;22(2):e167. doi: 10.4317/medoral.21555 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Layton D. A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: are systematic reviews actually systematic?. International Journal of Prosthodontics. 2017Jan1;30(1). doi: 10.11607/ijp.5193 [DOI] [PubMed] [Google Scholar]
14.Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Errors in search strategies used in systematic reviews and their effects on information retrieval. Journal of the Medical Library Association: JMLA. 2019Apr;107(2):210. doi: 10.5195/jmla.2019.567 [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. Journal of clinical epidemiology. 2006Oct1;59(10):1057–e1. doi: 10.1016/j.jclinepi.2006.01.007 [DOI] [PubMed] [Google Scholar]
16.Morton S, Berg A, Levit L, Eden J, editors. Finding what works in health care: standards for systematic reviews. [PubMed]
17.Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. editors. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2019Sep23. [DOI] [PMC free article] [PubMed] [Google Scholar]
18.US Department of Health and Human Services. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication; No. 10 (14)-EHC063-EF. [Google Scholar]
19.Akers J, Aguiar-Ibáñez R, Baba-Akbari A. Systematic reviews: CRD’s guidance for undertaking reviews in health care.
20.Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Jama. 2000Apr19;283(15):2008–12. doi: 10.1001/jama.283.15.2008 [DOI] [PubMed] [Google Scholar]
21.Batten J, Brackett A. Ensuring the rigor in systematic reviews: Part 3, the value of the search. Heart & Lung. 2021Mar1;50(2):220–2. doi: 10.1016/j.hrtlng.2020.08.005 [DOI] [PubMed] [Google Scholar]
22.Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology. 2008May1;61(5):440–8. doi: 10.1016/j.jclinepi.2007.06.005 [DOI] [PubMed] [Google Scholar]
23.Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PloS one. 2015May4;10(5):e0125931. doi: 10.1371/journal.pone.0125931 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016Sep26;11(9):e0163309. doi: 10.1371/journal.pone.0163309 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association: JMLA. 2016Oct;104(4):267. doi: 10.3163/1536-5050.104.4.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Rethlefsen ML, Farrell AM, Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of clinical epidemiology. 2015Jun1;68(6):617–26. doi: 10.1016/j.jclinepi.2014.11.025 [DOI] [PubMed] [Google Scholar]
27.McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of clinical epidemiology. 2016Jul1;75:40–6. doi: 10.1016/j.jclinepi.2016.01.021 [DOI] [PubMed] [Google Scholar]
28.Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Information & Libraries Journal. 2003Jun;20(2):65–74. doi: 10.1046/j.1471-1842.2003.00411.x [DOI] [PubMed] [Google Scholar]
29.International Committee of Medical Journal Editors [Internet]. Defining the Role of Authors and Contributors. 2021. Available from: http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html
30.Schvaneveldt N, Stellrecht EM. Assessing the roles and challenges of librarians in dental systematic and scoping reviews. Journal of the Medical Library Association: JMLA. 2021Jan1;109(1):52. doi: 10.5195/jmla.2021.1031 [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10(1):39. Published 2021 Jan 26. doi: 10.1186/s13643-020-01542-z [DOI] [PMC free article] [PubMed] [Google Scholar]
32.American Dental Association. Education [Internet]. ADA Health Policy Institute FAQ–Dental Schools and Students. 2021. Available from: https://www.ada.org/en/science-research/health-policy-institute/dental-statistics/education#:~:text=Education-,Education,U.S.%20and%20ten%20in%20Canada
33.Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):e1003583. Published 2021 Mar 29. doi: 10.1371/journal.pmed.1003583 [DOI] [PMC free article] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0256833.r001

Decision Letter 0

Lisa Susan Wieland

22 Jun 2021

PONE-D-21-15877

The effect of librarian involvement on the quality of systematic reviews in dental medicine

PLOS ONE

Dear Dr. Bloss,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 06 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Lisa Susan Wieland

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please ensure that you include a title page within your main document. We do appreciate that you have a title page document uploaded as a separate file, however, as per our author guidelines (http://journals.plos.org/plosone/s/submission-guidelines#loc-title-page) we do require this to be part of the manuscript file itself and not uploaded separately.

Could you therefore please include the title page into the beginning of your manuscript file itself, listing all authors and affiliations

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting and well-researched paper, illustrating the often poor reporting of literature search methods in dental systematic reviews. The writing is clear and the data is well-presented, with limitations acknowledged and neatly described.

I have some minor comments for the authors:

1. Lines 43-44: the statement that: "More limited studies nonetheless indicate that shortcomings in reporting and methodological quality extend to reviews published in dental medicine" - I would expect to see some referencing to support this (I don't doubt that is true!). For example, El-Rabbany M, Li S, Bui S, Muir JM, Bhandari M, Azarpazhooh A. A Quality Analysis of Systematic Reviews in Dentistry, Part 1: Meta-Analyses of Randomized Controlled Trials. J Evid Based Dent Pract. 2017 Dec;17(4):389-398. doi: 10.1016/j.jebdp.2017.06.004. Epub 2017 Jun 28. PMID: 29197440. This paper reports some improvement in dental SRs, but also that "there is considerable room for improvement" still. It would support this statement I think.

2. How did the team determine that the SR authors were/were not librarians or information scientists? Was this simply down to self-reporting of the job titles in the author line? I'm guessing so, but it would be good to have this spelled out explicitly.

3. Is there any data to support the assertion in lines 84-85 that there is a correlation between the number of databases searched and the quality of the search? It may be reasonable to search only selected databases depending on the topic and there may be reduced efficiency / some search redundancy involved in searching a large number of databases if this is not warranted. It may be interesting to look at a subset of reviews which searched more than four databases and check that wider searching was justified by the topic. I believe that information scientists are generally moving away from the assumption that more databases searched is equal to a higher quality search, see the paper by Bramer et al, which instead considers optimal databases to search: Bramer, W.M., Rethlefsen, M.L., Kleijnen, J. et al. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev 6, 245 (2017). https://doi.org/10.1186/s13643-017-0644-y.

4. Lines 151-154: This paragraph is a little confusing. What do you mean by "significance" in this context? What exactly was being measured?

Reviewer #2: Thank you for the opportunity to review this paper. This topic is of enormous interest to me, and I am happy to see that further research is being conducted in this area. In this article, the authors examine systematic reviews published in dental journals to assess the role and impact of librarians, the reproducibility of searches, and the quality of searches. This work builds upon prior work by others in clinical medical fields. The sheer volume of assessed manuscripts and searches in this work is impressive; having conducted similar research myself, I know that assessing search quality, for example, can take multiple hours for one search.

After reading this article, I have several questions and comments that I hope can help the authors strengthen this impactful manuscript.

1. Based on the content of the manuscript, I was surprised not to see several key papers cited and referenced, including the two major papers on errors in search strategies (Salvador-Olivan et al 2019 and Sampson & McGowan 2006) and PRESS (McGowan et al 2016). It seems that these might bring possibilities for direct comparisons, if not just references.

2. Regarding the text, Finding What Works in Health Care, the authors (correctly) note that it was published by the Institute of Medicine. Since the name of that organization has changed, it may be beneficial to readers to note the new name as well.

3. Line 75: The authors note that previous studies focused on internal medicine, yet they also cite relevant work looking at pediatrics, surgery, and cardiology. Perhaps this should be “clinical medicine” instead of “internal medicine”?

4. The authors do not discuss the role of librarian authorship in the introduction specifically, yet it is one of the primary research questions. Perhaps it would be helpful to add a brief sentence describing why this would be important in the intro?

5. Lines 95-99: The authors do not say when the search took place, and neither do they comment anywhere in the article on the time period included in their search and/or results. It would be helpful to state the dates of the searches as well as that there was no limit in date (if that is the case).

6. Line 97: The authors clearly conducted their search prior to the new revision of the PubMed systematic reviews subset search. It might be helpful to note that this was the active query at the date of the search; it is completely different today.

7. Lines 129-36: It was not clear to me; did the authors actually copy and paste in the searches to test reproducibility? It appears they may have, but this is not spelled out. If they did, this would be the first study to do so, which should be noted.

8. Lines 137-143: It was helpful that the study instrument was linked here, but I admit that I did not review the instrument immediately; I was therefore greatly surprised by many of the results, which did not appear to be mentioned in the manuscript. I would suggest that the main data elements, especially those reported in the results, should be listed and/or briefly described in the manuscript.

9. Lines 137-43: When assessing searches for quality, were the authors blinded to: the authors, the presence of a librarian, the findings from the first review, etc? The practice of duplicate assessment would reduce some of the inherent bias, certainly, but when librarians are assessing other librarians’ work, there may unconscious bias at play (both in that we would expect more from each other, so we might be pickier, and the opposite, that we would rate quality higher because it would match our expectations).

10. Lines 146-48: How did the authors conduct their analysis? What statistics were planned to be used? What were the planned analyses (i.e., multivariate regression? Linear regression?)? I see one location (line 173) where the authors mention one statistical analysis used, but this is the only place. For example, see the primary comparative articles that are similar topics for their analytical methods: Koffel et al, Meert et al, Rethlefsen et al.

11. It would be helpful to have other information about the data set in the results; information about years included, number of reviews per journal title, etc, could potentially help readers get a better grasp on what is actually being studied.

12. Lines 151-154: I appreciate that the authors are being up front that there is no statistical significance here, but I think that this should be located further down in the results—and the type of analysis used should be described in the methods.

13. Line 163 (and Lines 129-43): The authors do not make it explicitly clear whether they assessed whether ALL searches are available, or whether having AT LEAST ONE search available counts for the measure, “included the full search strategy.” This is a very different metric, and while I suspect it was “at least one” and not “all” (based on line 168, e.g., saying “a full search strategy”), this would be helpful to comment upon, as would the change in requirements for reporting between PRISMA 2009 and PRISMA 2020/PRISMA-S.

14. Lines 172-6: There is nothing in the appendices or the methods that I can see that discusses using an overall scored quality assessment. Am I missing it? This would definitely need to be discussed, along with details about how reviewer concordance/interrater reliability was assessed.

15. Lines 224-33: Because the authors’ data collection instrument uses binary responses to challenging questions, I think it is really difficult to say whether degree of librarian involvement led to anything. For example, one article without a librarian mentioned might use a single keyword for a search, missing potentially multiple synonyms. An article with a librarian author might have 10 keyword synonyms for a concept, but could miss one. They would have the same results in this study, however, because there is not a measurement of degree. I think it would be informative for the authors to discuss this as a limitation. I also think it would be really difficult to assess these for over 400 articles, in duplicate—pretty impressive! For example, Salvador-Olivan et al only examined 137 articles. It’s clear the scale of this work necessitated this simpler method.

16. Lines 238-46: I’m not entirely convinced that the authors’ conclusion matches their findings; to me, I can’t see any effect of librarian co-authorship on search quality (though clearly there is an association in the inclusion of a full search). Are the authors weighing some of their findings higher than others? It seems to be about evenly split between those areas where librarian co-authors are better and worse.

17. Discussion section: There is no comparison with other research anywhere in the Discussion section, which makes it hard to understand the context of these findings. Do they differ from other disciplines? Do they match other (non-librarians’) studies done in dental medicine? I would really like to see an expanded discussion section here, as I think this is an important study that would benefit from contextualization.

18. Line 269-70: The authors could cite Koffel here (#21) as evidence that others have shown that librarians are not always mentioned, even when they played a significant role.

19. Line 272: The authors suggest that bias was “limited where possible,” but I’m not clear on how that happened—was it just reviewing in duplicate?

20. Lines 277-8. I would suggest restating this sentence as, “Overall findings suggest that involvement of a librarian is associated with more reproducible searches and improved methodological reporting in dental medicine systematic reviews, though the association is non-significant.”

21. The Appendices lack titles, which makes it difficult to quickly identify.

22. Appendix D: What is meant by a grey lit search? How did the authors define it (this should really be in the Methods section)?

23. Figures: I may be missing something, but do the figures have captions? It was really difficult to figure out what I was looking at just by looking at the figure.

24. Fig 1: I’m not sold on how this figure was composed. I find it really confusing, personally. I think it would be more effective to use all of the search data here (including no search), as that would more starkly show the differences—and wouldn’t necessitate readers figuring out that the 100% only refers to 2 of the 3 collected data points.

25. Figures: Fig 2 and 3 use black for “no mention of a librarian.” The remaining figures use black for “librarian as co-author.” This is incredibly confusing.

26. Fig. 3: Why did the authors analyze librarian participation prior to deduplication? Seems odd.

27. Fig 5: Are the numbers here referring to number of articles with errors in line numbers? Or it looks like percentages of articles? Did most of the articles even use line numbers? How did you account for not applicable responses? From this graph, it appears pretty clearly that librarians make lots and lots of errors, except with brackets and Boolean, compared to non-specialists.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Anne Littlewood

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Sep 1;16(9):e0256833. doi: 10.1371/journal.pone.0256833.r002

Author response to Decision Letter 0

25 Jun 2021

AUTHOR RESPONSE: Thank you sincerely to the two reviewers of our manuscript entitled The effect of librarian involvement on the quality of systematic reviews in dental medicine. We appreciate your valuable feedback and the extensive amount of time and thought you put into it. We have attempted to address your questions and concerns as follows (all page numbers refer to the Revised Manuscript with Track Changes):

I have some minor comments for the authors:

AUTHOR RESPONSE: Citation added - #11

AUTHOR RESPONSE: Added details in lines 146-153

AUTHOR RESPONSE: Added reasoning to lines 99-101. Also added new Figure 3 based off new data.

4. Lines 151-154: This paragraph is a little confusing. What do you mean by "significance" in this context? What exactly was being measured?

AUTHOR RESPONSE: Relocated this section and clarified in lines 357-363

After reading this article, I have several questions and comments that I hope can help the authors strengthen this impactful manuscript.

AUTHOR RESPONSE: Added citations and added comparisons in discussion section 367-392

AUTHOR RESPONSE: Noted new name in lines 71-72

AUTHOR RESPONSE: Changed wording in line 83

AUTHOR RESPONSE: Included in lines 68-86

AUTHOR RESPONSE: Inserted search date in line 112. Statement about no date restrictions is in line 114.

AUTHOR RESPONSE: Inserted search date in line 112. Added wording about old version of PubMed in lines 166-168.

AUTHOR RESPONSE: Added wording in lines 164-165.

AUTHOR RESPONSE: Added wording in lines 153-157 and 176-180.

AUTHOR RESPONSE: Added wording in lines 158-159, 175-176, 184-186, 423-428.

AUTHOR RESPONSE: Adjusted language in lines 191-197.

AUTHOR RESPONSE: Unfortunately, we do not have data about number of reviews per journal. Added language to data analysis section for more clarity.

AUTHOR RESPONSE: Relocated this section and clarified in lines 354-361. Data analysis further described in lines 191-197.

AUTHOR RESPONSE: Adjusted language in lines 210-219 and line 302

AUTHOR RESPONSE: Adjusted language in lines 139-141 to assist in clarity about the modification of the Rethlefsen instrument and language to lines 225-227 for clarity.

AUTHOR RESPONSE: Thank you for your recognition of the scope of this work. Language was added to lines 422-428 in limitations to address lack of scale.

AUTHOR RESPONSE: Adjusted language in the Principal findings section in lines 367-392.

AUTHOR RESPONSE: Added language to lines 367-392.

18. Line 269-70: The authors could cite Koffel here (#21) as evidence that others have shown that librarians are not always mentioned, even when they played a significant role.

AUTHOR RESPONSE: Added citation for Koffel article in line 418.

19. Line 272: The authors suggest that bias was “limited where possible,” but I’m not clear on how that happened—was it just reviewing in duplicate?

AUTHOR RESPONSE: Added wording about bias for clarity in lines 158-159, 175-176, 184-186, 423-428.

AUTHOR RESPONSE: Adjusted language to lines 438-440.

21. The Appendices lack titles, which makes it difficult to quickly identify.

AUTHOR RESPONSE: Added appendix titles

22. Appendix D: What is meant by a grey lit search? How did the authors define it (this should really be in the Methods section)?

AUTHOR RESPONSE: Added information about grey literature in lines 93-96.

23. Figures: I may be missing something, but do the figures have captions? It was really difficult to figure out what I was looking at just by looking at the figure.

AUTHOR RESPONSE: Added legends for figures. Titles and legends are contained in article instead of in figure files as per PlosOne Instructions: https://journals.plos.org/plosone/s/figures#loc-captions

AUTHOR RESPONSE: Adjusted Figure 1. See figure file.

25. Figures: Fig 2 and 3 use black for “no mention of a librarian.” The remaining figures use black for “librarian as co-author.” This is incredibly confusing.

AUTHOR RESPONSE: Adjusted figures for consistency.

26. Fig. 3: Why did the authors analyze librarian participation prior to deduplication? Seems odd.

AUTHOR RESPONSE: Figure 3 has become Figure 4. It is purely about reporting of numbers and not about analysis. The use of the PRISMA Diagram format was due to familiarity of the document. Added wording to the figure legend for clarity.

AUTHOR RESPONSE: Figure 5 has become figure 6. Speculation about reasons for errors was added in lines 380-389.

________________________________________

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

AUTHOR RESPONSE: No thank you.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Anne Littlewood

Reviewer #2: No

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(22.6KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0256833.r003

Decision Letter 1

Lisa Susan Wieland

22 Jul 2021

PONE-D-21-15877R1

The effect of librarian involvement on the quality of systematic reviews in dental medicine

PLOS ONE

Dear Dr. Bloss,

Please address both the reviewer clarifications of previous comments and the new comments regarding reorganizing text and providing citations.

Please submit your revised manuscript by Sep 05 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Lisa Susan Wieland

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #2: Yes

**********

6. Review Comments to the Author

Reviewer #2: I thank the authors for their excellent work addressing reviewer comments, and I appreciate the chance to review their revised version. This version addresses nearly all of the comments, and the manuscript now offers more context to readers and offers clarity on a number of areas. I have a few remaining questions and comments.

1. The authors do not discuss the role of librarian authorship in the introduction specifically, yet it is one of the primary research questions. Perhaps it would be helpful to add a brief sentence describing why this would be important in the intro?

The authors have added additional content to the introduction, but I am not seeing any addition of information specifically about librarian co-authorship. I still think it would be helpful to mention something specifically about authorship prior to the research questions. The same is true with grey literature, which is a research question, but is not specifically addressed in the intro.

2. Line 97: The authors clearly conducted their search prior to the new revision of the PubMed systematic reviews subset search. It might be helpful to note that this was the active query at the date of the search; it is completely different today.

This item was unfortunately written poorly by me. I meant that the systematic review subset search has changed, not PubMed (though that is also true). If the authors compare the current PubMed systematic reviews search with the old one, they will see it has significantly changed. However, with the date included, it will help readers understand that this was the subset search at the time of that search. Unfortunately, NLM doesn’t seem to track versions of this search anywhere that I’ve seen.

3. Lines 146-48: How did the authors conduct their analysis? What statistics were planned to be used? What were the planned analyses (i.e., multivariate regression? Linear regression?)? I see one location (line 173) where the authors mention one statistical analysis used, but this is the only place. For example, see the primary comparative articles that are similar topics for their analytical methods: Koffel et al, Meert et al, Rethlefsen et al.

I appreciate the additional detail provided by the authors about their analyses. I think there is still more work to be done here. “Mostly” should be avoided. What variables were tested using chi squares, e.g.

4. It would be helpful to have other information about the data set in the results; information about years included, number of reviews per journal title, etc, could potentially help readers get a better grasp on what is actually being studied. AUTHOR RESPONSE: Unfortunately, we do not have data about number of reviews per journal. Added language to data analysis section for more clarity.

I think the authors do have this information, or I would hope so? Do you not have the citations for the included studies?

5. Discussion section: There is no comparison with other research anywhere in the Discussion section, which makes it hard to understand the context of these findings. Do they differ from other disciplines? Do they match other (non-librarians’) studies done in dental medicine? I would really like to see an expanded discussion section here, as I think this is an important study that would benefit from contextualization.

A few new comments:

6. The authors very nicely lay out new research questions. The associated detail with those questions, however, should be in the methods section. Lines 93-6; 99-101

7. PRISMA 2009, PRISMA 2020, and PRISMA-S are mentioned in several places in the text, but the latter two are not cited. It would be appropriate to add citations. (Lines 397; 432-3)

8. I could be missing something, but it looks to me like the information in lines 272-5 is redundant with the information in the paragraph above—with one major caveat. The caveat is that, in line 270, I think 71% is supposed to be 74%.

9. Lines 307-310: The authors may want to consider moving this up to be their first data point in the results, as the analyses do all seem to depend upon it.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

PLoS One. 2021 Sep 1;16(9):e0256833. doi: 10.1371/journal.pone.0256833.r004

Author response to Decision Letter 1

5 Aug 2021

Plos ONE Rebuttal Document – Version 2

We again appreciate the thoughtful comments made by Reviewer 2. We appreciate the time and effort taken to help us improve this paper for publication. We believe these adjustments have significantly improved the paper. Thank you.

Added wording about Gray Literature to lines 76-77 and wording about co-authors to lines 82-87 (including a reference [29] to the ICMJE recommendations for authorship).

Added wording to lines 113-114 about the use of the Clinical Queries search string prior to the PubMed update. Also, complete search is included in Appendix A, with the Clinical Queries search contained in the full search.

Added wording to Data Analysis section (lines 193-204) and Table 1

I think the authors do have this information, or I would hope so? Do you not have the citations for the included studies?

As all studies were identified with PMID numbers, this was not a data point we originally ran. Re-ran the journal title search compared to all included PMID numbers in response to this comment. Added number of examined articles from each journal to Appendix A.

A few new comments:

6. The authors very nicely lay out new research questions. The associated detail with those questions, however, should be in the methods section. Lines 93-6; 99-101

Moved associated detail to the methods section.

7. PRISMA 2009, PRISMA 2020, and PRISMA-S are mentioned in several places in the text, but the latter two are not cited. It would be appropriate to add citations. (Lines 397; 432-3)

Added references for PRISMA 2020 and PRISMA-S

Adjusted language for clarity. Now lines 273-284

9. Lines 307-310: The authors may want to consider moving this up to be their first data point in the results, as the analyses do all seem to depend upon it.

We have arranged the paper with two main Results sections: Reporting in All Reviews; Analysis of Reproducible Searches. We have arranged the data point as the first in the section labeled: Analysis of Reproducible Searches.

Attachment

Submitted filename: Response to Reviewers 2.docx

Click here for additional data file.^{(14.7KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0256833.r005

Decision Letter 2

Lisa Susan Wieland

17 Aug 2021

The effect of librarian involvement on the quality of systematic reviews in dental medicine

PONE-D-21-15877R2

Dear Dr. Bloss,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Lisa Susan Wieland

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #2: Yes

**********

6. Review Comments to the Author

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

PLoS One. doi: 10.1371/journal.pone.0256833.r006

Acceptance letter

Lisa Susan Wieland

23 Aug 2021

PONE-D-21-15877R2

The effect of librarian involvement on the quality of systematic reviews in dental medicine

Dear Dr. Bloss:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Lisa Susan Wieland

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Appendix

(DOCX)

Click here for additional data file.^{(12.3KB, docx)}

S2 Appendix

(DOCX)

Click here for additional data file.^{(13.5KB, docx)}

S3 Appendix

(DOCX)

Click here for additional data file.^{(13.8KB, docx)}

S4 Appendix

(DOCX)

Click here for additional data file.^{(14KB, docx)}

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(22.6KB, docx)}

Attachment

Submitted filename: Response to Reviewers 2.docx

Click here for additional data file.^{(14.7KB, docx)}

Data Availability Statement

All data is available via OSF: https://osf.io/qsua8/files/?view_only=ea7a550400f4403eb280b1d383ec11fa.

[pone.0256833.ref001] 1.Faggion CM Jr, Huivin R, Aranda L, Pandis N, Alarcon M. The search and selection for primary studies in systematic reviews published in dental journals indexed in MEDLINE was not fully reproducible. Journal of clinical epidemiology. 2018Jun1;98:53–61. doi: 10.1016/j.jclinepi.2018.02.011 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref002] 2.Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Systematic reviews. 2017Dec;6(1):1–4. doi: 10.1186/s13643-016-0385-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref003] 3.Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 2009Jul21;6(7):e1000097. doi: 10.1371/journal.pmed.1000097 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref004] 4.Nawijn F, Ham WH, Houwert RM, Groenwold RH, Hietbrink F, Smeeing DP. Quality of reporting of systematic reviews and meta-analyses in emergency medicine based on the PRISMA statement. BMC emergency medicine. 2019Dec;19(1):1–8. doi: 10.1186/s12873-018-0218-x [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref005] 5.Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS medicine. 2016May24;13(5):e1002028. doi: 10.1371/journal.pmed.1002028 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref006] 6.Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007Mar27;4(3):e78. doi: 10.1371/journal.pmed.0040078 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref007] 7.Hua F, Walsh T, Glenny AM, Worthington H. Surveys on reporting guideline usage in dental journals. Journal of Dental Research. 2016Oct;95(11):1207–13. doi: 10.1177/0022034516657803 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref008] 8.Sarkis-Onofre R, Cenci MS, Moher D, Pereira-Cenci T. Research reporting guidelines in dentistry: A survey of editors. Brazilian dental journal. 2017Feb;28(1):3–8. doi: 10.1590/0103-6440201601426 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref009] 9.Bassani R, Pereira GK, Page MJ, Tricco AC, Moher D, Sarkis-Onofre R. Systematic reviews in dentistry: current status, epidemiological and reporting characteristics. Journal of dentistry. 2019Mar1;82:71–84. doi: 10.1016/j.jdent.2019.01.014 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref010] 10.Sideri S, Papageorgiou SN, Eliades T. Are orthodontic systematic reviews registered a priori in PROSPERO?. Journal of orthodontics. 2017Dec;44(4):249–55. doi: 10.1080/14653125.2017.1370773 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref011] 11.El-Rabbany M, Li S, Bui S, Muir JM, Bhandari M, Azarpazhooh A. A quality analysis of systematic reviews in dentistry, part 1: meta-analyses of randomized controlled trials. Journal of Evidence Based Dental Practice. 2017Dec1;17(4):389–98. doi: 10.1016/j.jebdp.2017.06.004 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref012] 12.Hasuike A, Iguchi S, Suzuki D, Kawano E, Sato S. Systematic review and assessment of systematic reviews examining the effect of periodontal treatment on glycemic control in patients with diabetes. Medicina oral, patologia oral y cirugia bucal. 2017Mar;22(2):e167. doi: 10.4317/medoral.21555 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref013] 13.Layton D. A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: are systematic reviews actually systematic?. International Journal of Prosthodontics. 2017Jan1;30(1). doi: 10.11607/ijp.5193 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref014] 14.Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Errors in search strategies used in systematic reviews and their effects on information retrieval. Journal of the Medical Library Association: JMLA. 2019Apr;107(2):210. doi: 10.5195/jmla.2019.567 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref015] 15.Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. Journal of clinical epidemiology. 2006Oct1;59(10):1057–e1. doi: 10.1016/j.jclinepi.2006.01.007 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref016] 16.Morton S, Berg A, Levit L, Eden J, editors. Finding what works in health care: standards for systematic reviews. [PubMed]

[pone.0256833.ref017] 17.Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. editors. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2019Sep23. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref018] 18.US Department of Health and Human Services. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication; No. 10 (14)-EHC063-EF. [Google Scholar]

[pone.0256833.ref019] 19.Akers J, Aguiar-Ibáñez R, Baba-Akbari A. Systematic reviews: CRD’s guidance for undertaking reviews in health care.

[pone.0256833.ref020] 20.Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Jama. 2000Apr19;283(15):2008–12. doi: 10.1001/jama.283.15.2008 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref021] 21.Batten J, Brackett A. Ensuring the rigor in systematic reviews: Part 3, the value of the search. Heart & Lung. 2021Mar1;50(2):220–2. doi: 10.1016/j.hrtlng.2020.08.005 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref022] 22.Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology. 2008May1;61(5):440–8. doi: 10.1016/j.jclinepi.2007.06.005 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref023] 23.Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PloS one. 2015May4;10(5):e0125931. doi: 10.1371/journal.pone.0125931 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref024] 24.Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016Sep26;11(9):e0163309. doi: 10.1371/journal.pone.0163309 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref025] 25.Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association: JMLA. 2016Oct;104(4):267. doi: 10.3163/1536-5050.104.4.004 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref026] 26.Rethlefsen ML, Farrell AM, Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of clinical epidemiology. 2015Jun1;68(6):617–26. doi: 10.1016/j.jclinepi.2014.11.025 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref027] 27.McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of clinical epidemiology. 2016Jul1;75:40–6. doi: 10.1016/j.jclinepi.2016.01.021 [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref028] 28.Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Information & Libraries Journal. 2003Jun;20(2):65–74. doi: 10.1046/j.1471-1842.2003.00411.x [DOI] [PubMed] [Google Scholar]

[pone.0256833.ref029] 29.International Committee of Medical Journal Editors [Internet]. Defining the Role of Authors and Contributors. 2021. Available from: http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html

[pone.0256833.ref030] 30.Schvaneveldt N, Stellrecht EM. Assessing the roles and challenges of librarians in dental systematic and scoping reviews. Journal of the Medical Library Association: JMLA. 2021Jan1;109(1):52. doi: 10.5195/jmla.2021.1031 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref031] 31.Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10(1):39. Published 2021 Jan 26. doi: 10.1186/s13643-020-01542-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0256833.ref032] 32.American Dental Association. Education [Internet]. ADA Health Policy Institute FAQ–Dental Schools and Students. 2021. Available from: https://www.ada.org/en/science-research/health-policy-institute/dental-statistics/education#:~:text=Education-,Education,U.S.%20and%20ten%20in%20Canada

[pone.0256833.ref033] 33.Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):e1003583. Published 2021 Mar 29. doi: 10.1371/journal.pmed.1003583 [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

The effect of librarian involvement on the quality of systematic reviews in dental medicine

Jana Schellinger

Kerry Sewell

Jamie E Bloss

Tristan Ebron

Carrie Forbes

Roles

Abstract

Objectives

Methods

Results

Conclusion

Introduction

Methods

Database search and selection of SRs

Method for reproducibility assessment and data extraction

Data analysis

Table 1. Data analysis.

Results

Librarian participation

Inclusion of search strategy

Fig 1. Inclusion of search strategy.

Search quality

Reporting in all reviews

Information reported in reviews

Fig 2. Information reported in reviews.

Databases

Reporting the numbers

Fig 3. Numbers as reported in reviews.

Reproducibility

Analysis of reproducible searches

Librarian assistance

Information reported in reproducible searches

Fig 4. Information reported in reproducible searches.

Search details

Fig 5. Search syntax mistakes.

Fig 6. Search language mistakes.

Significance

Discussion

Principal findings

Implications

Limitations

Conclusion

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Lisa Susan Wieland

Roles

Author response to Decision Letter 0

Decision Letter 1

Lisa Susan Wieland

Roles

Author response to Decision Letter 1

Decision Letter 2

Lisa Susan Wieland

Roles

Acceptance letter

Lisa Susan Wieland

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases