Skip to main content
PMC home page
BMC Medical Research Methodology logoLink to BMC Medical Research Methodology
. 2020 Jun 1;20:138. doi: 10.1186/s12874-020-00983-3

Database combinations to retrieve systematic reviews in overviews of reviews: a methodological study

Käthe Goossen 1,✉, Simone Hess 1, Carole Lunny 2, Dawid Pieper 1
PMCID: PMC7268249  PMID: 32487023

Abstract

Background

When conducting an Overviews of Reviews on health-related topics, it is unclear which combination of bibliographic databases authors should use for searching for SRs. Our goal was to determine which databases included the most systematic reviews and identify an optimal database combination for searching systematic reviews.

Methods

A set of 86 Overviews of Reviews with 1219 included systematic reviews was extracted from a previous study. Inclusion of the systematic reviews was assessed in MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. The mean inclusion rate (% of included systematic reviews) and corresponding 95% confidence interval were calculated for each database individually, as well as for combinations of MEDLINE with each other database and reference checking.

Results

Inclusion of systematic reviews was higher in MEDLINE than in any other single database (mean inclusion rate 89.7%; 95% confidence interval [89.0–90.3%]). Combined with reference checking, this value increased to 93.7% [93.2–94.2%]. The best combination of two databases plus reference checking consisted of MEDLINE and Epistemonikos (99.2% [99.0–99.3%]). Stratification by Health Technology Assessment reports (97.7% [96.5–98.9%]) vs. Cochrane Overviews (100.0%) vs. non-Cochrane Overviews (99.3% [99.1–99.4%]) showed that inclusion was only slightly lower for Health Technology Assessment reports. However, MEDLINE, Epistemonikos, and reference checking remained the best combination. Among the 10/1219 systematic reviews not identified by this combination, five were published as websites rather than journals, two were included in CINAHL and Embase, and one was included in the database ERIC.

Conclusions

MEDLINE and Epistemonikos, complemented by reference checking of included studies, is the best database combination to identify systematic reviews on health-related topics.

Keywords: Review methods, Overview of reviews, Umbrella review, Search strategy, Databases, Systematic reviews

Highlights

What is already known

  • Overviews of Reviews summarize data from several systematic reviews, thus providing a broad picture of relevant evidence.

  • Few empirical studies exist that support methodological approaches to producing Overviews of Reviews, including the search process for systematic reviews.

What is new

  • The present empirical study has shown that in a sample of 1219 systematic reviews, 99.2% of records are included in a combination of the electronic databases MEDLINE via PubMed and Epistemonikos, supplemented by checking reference lists of eligible studies retrieved by these databases.

Potential impact

  • A resource-efficient search process for systematic reviews should be based on the free-of-charge electronic databases MEDLINE via PubMed and Epistemonikos, supplemented by reference checking.

Background

Overviews of Reviews (Overviews) are used to summarize data from across multiple systematic reviews (SRs) on the same topic, thus providing a broader picture of evidence relevant for decision-making [1]. Generally, they aim to provide a comprehensive assessment of multiple interventions, populations, or outcomes for a defined health problem using SR-level evidence.

Some methodological components of preparing Overviews are analogous to SRs, such as defining the review question structured into a Population, Intervention, Comparator, Outcomes, and Study types (PICOS) format, setting eligibility criteria, and defining a search strategy, though nuances clearly exist. Other methodological components such as the critical appraisal of included SRs, extraction and analysis of data, and presentation of findings differ more markedly between Overviews and SRs [2–4].

Many specific steps for conducting an Overview are not yet based on empirical research [5], and some steps contain few empirical studies which may guide future recommendations including selection of electronic databases to use when searching for SRs.

Current guidance recommends searching the Cochrane Database of Systematic Reviews (CDSR) to locate Cochrane SRs [1, 6], and a structured search process to locate and select non-Cochrane reviews, (e.g. in MEDLINE [7] and Embase [8]). Guidance also recommends searching additional regional and subject-specific databases (e.g. LILACS [9], CINAHL [10], PsycINFO [11]), major repositories of SRs, and the PROSPERO register [1, 2, 12]. In practice, the majority of Overviews in healthcare-related fields published to date searched at least two databases [13, 14], in addition to reference checking (i.e. looking through the reference list of included studies), and consultation with the relevant Cochrane group [14].

Rathbone et al. have evaluated retrieval of a set of 400 SRs of interventions for hypertension in seven electronic databases [15]. They searched for SRs in the Cochrane Library, DARE, Embase, Epistemonikos, MEDLINE, PubMed Health, and TRIP. For each database, sensitivity, precision, numbers missed, and unique records identified were calculated. Embase had the highest sensitivity (169/400, or 69% of SRs) and retrieved the largest number of unique SR records, i.e. records found by their search in only one database (34/400, or 8.5% of SRs). The authors conclude that searching should not be restricted to one or two major databases, but that all databases should be searched.

In a situation where resources are limited, it remains unclear which combination of electronic resources Overview authors should use for searching for SRs. Therefore, the aims of the present methodological study were: (1) to assess the inclusion of SRs in six electronic databases: MEDLINE [7], CINAHL [10], Embase [8], Epistemonikos [16], PsycINFO [11], and TRIP [17] without limiting the SR topic; (2) to evaluate whether a single database combined with manual reference checking is sufficient to comprehensively retrieve SRs in the context of preparing Overviews; (3) to identify a combination of databases plus reference checking that may be recommended for searching SRs; and (4) to explore possible contexts (for example, for certain Overview types, or for healthcare interventions compared to other fields of healthcare research) in which the recommendation result of objective #3 produces inadequate retrieval of SRs, defined as lower than 95% retrieval [18]. It was not our aim to evaluate search strategies.

Methods

Study design

A methodological study was conducted to analyse the rate of inclusion of SRs in six selected electronic resources, and in combinations of two databases with reference checking. We did not write a protocol for this study.

Search methods

A set of Overviews, including Health Technology Assessment (HTA) reports, Cochrane Overviews, and non-Cochrane Overviews, was obtained from a preceding methodological study, as described in detail in Pieper et al. (2014) [19]. Briefly, a search for overviews was conducted in MEDLINE via PubMed, Embase via Embase.com [8], CINAHL via EBSCOhost [10], PEDro [20], CDSR, DARE, CENTRAL, CMR, HTA, and NHS-EED via the Cochrane Library, and the websites of 127 HTA agencies from inception to May 2012. Search terms were text words related to Overviews and SRs and the search algorithms can be found in Pieper et al. (2014) [19]. The search was limited to Overviews published in English or German between 2009 and 2011.

Eligibility criteria

Overviews were defined as systematic reviews for which the unit of searching, inclusion and data analysis is the systematic review rather than the primary study [1]. Thus, we included all Overviews that had searched explicitly and systematically for SRs in at least one electronic database, included at least one SR (Overviews including both SRs and primary studies were eligible if the evidence synthesis relied at least in part on SRs, e.g., by combining primary studies and SRs in the evidence synthesis), and critically appraised all included SRs and additional primary studies. A HTA report was defined as “a method of evidence synthesis that considers evidence regarding clinical effectiveness, safety, cost-effectiveness and, when broadly applied, includes social, ethical, and legal aspects of the use of health technologies” [21].

Inclusion criteria

  • Searched for SRs in at least one electronic database;

  • Included at least one SR in their evidence synthesis;

  • Critically appraised included SRs and primary studies; and

  • Full text publication was available.

Exclusion criteria

  • Overviews with a methodological focus; and

  • Published in a language other than English or German.

The set of included Overviews will be henceforth called the “reference set”.

Description of electronic databases selected in this study

Six databases were selected to assess inclusion of systematic reviews as described in the section “data collection”, below, namely MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. The key features of these databases are described in Table 1.

Table 1.

Description of electronic databases and resources

Publisher Access Type Coverage
CINAHL [10] EBSCO by subscription indexed database nursing, biomedicine, health sciences librarianship, alternative medicine, and allied health topics
Embase [8] Elsevier by subscription indexed database biomedical literature, 1947 to present
Epistemonikos [16] Epistemonikos foundation (non-profit) free of charge citations database, data scraped from other databases and the web health evidence, nine supported languages [22, 23]
MEDLINE [7] U.S. National Library of Medicine (non-profit) free of charge indexed database biomedicine and health literature, 1966 to present [24]
PsycINFO [11] EBSCO, American Psychological Association by subscription indexed database behavioural science and mental health
TRIP [17] Trip Inc. free of charge clinical search engine health care [25]
Open in a new tab

Psyc. topic = mental health- or psychology-related topic

The sources scraped by Epistemonikos include, or have included, CDSR, PubMed, Embase, CINAHL, PsycINFO, LILACS, DARE, HTA database, The Campbell Collaboration online library, JBI Database of Systematic Reviews and Implementation Reports, EPPI-Centre Evidence Library. Updates of algorithms in February and April 2019 have led to an expansion in the dataset by more than a factor of 1.5. TRIP widely collects references from sources of SRs (including Cochrane Library and DARE), guidelines, regulatory agencies (FDA, EMA, NICE, IQWIG), HTA databases, NHS EED, literature databases (PubMed), journals, as well as PROSPERO and clinical trial registries.

Data collection

From the full text of each included Overview, the following data were extracted into MS Excel (2016): citation, publication title, number of databases searched, name of each database searched, searched in social science/economics databases (yes/no; i.e. EconLit, HEED, NHS EED, IBSS, Social Sciences Citation Index, Social SciSearch, the Campbell Collaboration Database, Social Sciences Abstracts, Social Services Abstracts, Applied Social Science Index and Abstracts, Social Service Information Gateway), searched in additional sources (‘other sources’ yes/no; i.e. reference lists of included studies, queries to experts, Google, Google Scholar, internal departmental files, hand-searching or electronically searching journals, clinical trial or study registries (e.g. clinicaltrials.gov, PROSPERO), publishers’ databases (e.g. Springer, ScienceDirect, Thieme, Wolters Kluwer), HTA agencies’ websites (e.g. https://www.iqwig.de, https://www.dimdi.de, http://www.msac.gov.au)), number of SRs included, Overview type (Cochrane Overview, HTA report, or non-Cochrane Overview), intervention/non-intervention Overview, and mental health- or psychology-related topic (yes/no).

For each Overview, the included SRs were extracted and tagged with the Overview from which they originated. Primary studies were not extracted. HTA reports are usually structured into sections entitled clinical effectiveness, safety, cost-effectiveness, social, ethical, or legal. For HTAs, we only included SRs from the clinical effectiveness section of the report.

The database searches for SRs were performed in April 2019. A stepwise process was followed to identify whether a SR was included in an electronic database, found by reference checking, or included in a database combination:

  • (A)

    From the sample of SRs extracted from the Overviews, we determined which of six databases each SR was included in, namely MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, or TRIP.

  • (B)

    We then determined which database contained the largest overall number of included SRs. This database was identified as the ‘reference database’. We set the reference database to MEDLINE as it had the highest inclusion rate. The SRs included in MEDLINE will be henceforth called the “MEDLINE-included SRs”.

  • (C)

    A list of all SRs not included in MEDLINE was compiled. These SRs are called the ‘MEDLINE-non-included SRs’.

  • (D)

    For each ‘MEDLINE-non-included SR’ obtained in step C, we then manually checked the reference lists of the ‘MEDLINE-included SRs’ that were cited in the same Overview as the ‘MEDLINE-non-included SR’. The purpose of this step was to find out if each SR not included in MEDLINE could have been identified by reference checking of SRs identified in MEDLINE on the same topic, rather than by additional database searching. The SRs found in the reference lists/bibliographies are henceforth called ‘biblio SRs’.

Finally, we constructed five combined sets of SRs by merging the ‘MEDLINE-included SRs’, ‘biblio-SRs’, and SRs obtained in step A for CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. For each of these five combined sets, we calculated a combined mean inclusion rate (see statistical analysis). This was done to evaluate whether searching more than one database would expand the study pool.

Statistical analysis

The dataset generated and analysed during the current study is made available in Additional file 1. For each Overview, we calculated:

  • (A)

    the mean inclusion rate (% of included SRs) and corresponding 95% confidence interval (95% CI) separately for each database.

  • (B)

    the mean inclusion rate for the reference database (as defined above) combined with reference checking (as described above).

  • (C)

    the mean inclusion rates for combinations of the reference database, reference checking, and each of the other five databases.

The Overview-level inclusion rates obtained in statistical analysis steps A to C were then aggregated for the entire dataset by calculating weighted mean inclusion rates and corresponding 95% confidence intervals (95% CI). Weighting was based on the number of SRs included in each Overview.

Stratification

The goals of stratification were to generate hypotheses on contexts where the results-based recommendations would apply, and to identify situations where retrieval would be inadequate and further searches may be necessary. Inadequate retrieval of SRs was defined as lower than 95% retrieval. Thus, we: (1) investigated whether searching large numbers of databases offers added value, (2) gauged the magnitude of effect when using ‘other sources’ (as defined above in section ‘data collection’), (3) examined whether different Overview types require searching different electronic resources, (4) explored differences in database inclusion between healthcare interventions and other fields of healthcare research, and (5) evaluated the role of specialist databases when such databases exist in the area of the Overview topic, using PsycINFO as an example.

To answer the above objectives (1) to (5), respectively, exploratory analyses were performed for the following strata: (1) number of databases searched (1–3 / ≥4), (2) other sources searched (yes / no), (3) Overview type (Cochrane Overview, HTA report, or non-Cochrane Overview), (4) intervention/non-intervention Overviews, and (5) mental health- or psychology-related topic (yes/no).

Stratification analysis was performed only for strata containing ≥3 Overviews for analysis. For each stratification analysis, the weighted mean inclusion rate with 95% CI was calculated for combinations of the reference database and reference checking with each of the other databases. For analyses with two strata, the weighted difference in means and corresponding p-value were calculated using a two-sample weighted t-test (Welch) computed in R version 3.5.1 (2018-07-02) using the R package ‘weights’ [26, 27]. The significance level for each individual test αi was adjusted for multiple testing using the Bonferroni correction, i.e. αi = αg/n = 0.0017 for a global significance level of αg = 0.05 and n = 30 tests (6 databases and 5 stratified analyses).

Qualitative analysis of missed SRs

All SRs that were not included in a combination of the reference database, reference checking, and the best additional database were analysed qualitatively. Features investigated were the topics of these SRs, whether they were located on websites, included in the other five databases that were investigated in this study, listed in a publisher’s database (e.g. ScienceDirect, Wiley Online Library, Springerlink, De Gruyter), or in Google Scholar.

Results

Overviews included in the study

The literature search yielded a reference set of 86 Overviews [28–113], of which 73 had been identified in electronic databases and 13 on HTA websites [19]. The characteristics of the included Overviews are summarised in Table 2, and detailed in Table S1 (see Additional file 2).

Table 2.

Summary characteristics of N = 86 included Overviews

Median IQR Range
Databases searched 6 4–8 1–25
SRs included per Overview 8 5–17 1–103
n % (of 86 Overviews)
Type:
 Cochrane Overview 3 3%
 HTA report 14 16%
 non-Cochrane Overview 69 80%
n % (of 1219 SRs)
Databases:
 MEDLINE/PubMeda 82 95%
 Embase 61 71%
 DARE or Cochrane Library b 59 69%
 CINAHL 36 42%
 PsycINFO 30 35%
 HTA databases (any) 20 23%
 TRIP 3 3%
 Epistemonikos 0 0%
 Any other social science/economics database 18 21%
Other sources searched 64 74%
Intervention Overviews 61 69%
Mental health/psychology-related topic 9 10%
Open in a new tab

a MEDLINE also contains all CDSR content [6]. b DARE was searchable via the Cochrane Library until August 2018, when it was removed along with the other CRD databases NHS EED and HTA, as all three are no longer updated by CRD [114]. HTA is now produced by INAHTA and can be searched via the Canadian Agency for Drugs and Technologies in Health [115]. IQR interquartile range

A total of 1219 SRs were included in the 86 Overviews overall, with a median of eight SRs in an Overview. Between one and 25 (median 6) databases had been searched. Among them, the most widely used were MEDLINE and Embase. More than half of the Overviews had searched DARE or the Cochrane Library, which up to 2018 also contained all DARE records. None had searched the international SR-focused database Epistemonikos, which was first launched internationally in August 2012, and was therefore not available at the time of publication of the included Overviews [22].

A wide range of subjects were covered by the included Overviews [13]. Sixty-nine percent were intervention Overviews, defined as Overviews of health-related interventions, i.e. actions taken with the intent of modifying a defined outcome, usually to treat or cure a health condition or change behaviour. Non-intervention Overviews focused, for example, on diagnosis, incidence/prevalence, or risk factors for certain outcomes.

Thirty-five percent (n = 30/86) of Overviews searched PsycINFO, but of these, only 30% (n = 9/30) had topics related to mental health (i.e. autism, depression, psychosis, or schizophrenia), or psychology (i.e. psychological or psychosocial interventions (such as behavioural therapy)).

Inclusion in individual databases and database combinations

Among the 1219 SRs retrieved, 90% (n = 1093/1219) were included in MEDLINE. The inclusion rates in other databases were lower (Table 3, Fig. 1). Therefore, MEDLINE was used as the reference database.

Table 3.

Proportion of SRs included in individual databases and their combination with MEDLINE and reference checking

Database single database % [95% CI] database + references % [95% CI] MEDLINE + one database + references; % [95% CI]
MEDLINE 89.7% [89.0 to 90.3%] 93.7% [93.2 to 94.2%]
CINAHL 44.7% [43.7 to 45.7%] 94.6% [94.1 to 94.2%]
Embase 83.7% [82.9 to 84.5%] 94.8% [94.4 to 95.3%]
Epistemonikos 85.6% [84.7 to 86.5%] 99.2% [99.0 to 99.3%]
PsycINFO 24.5% [23.1 to 26.0%] 95.1% [94.7 to 95.5%]
TRIP 52.6% [51.0 to 54.1%] 96.3% [95.9 to 96.7%]
Open in a new tab

Best results in italics

Fig. 1.

Fig. 1

Open in a new tab

Inclusion rates in individual databases and their combination with MEDLINE and reference checking. MEDL = MEDLINE, CIN = CINAHL, EMB = Embase, PSI = PsycINFO, EPIS = Epistemonikos, DB = database

In addition to the SRs included in MEDLINE, a further 4% of SRs (n = 49/1219) were identified by reference checking (see Methods, step B). Combinations of MEDLINE and reference checking with the addition of one database (Epistemonikos, PsycINFO, and TRIP) led to inclusion rates above 95% (n = 1209/1219, 1159/1219, and 1174/1219, respectively). Among these databases, the inclusion rate was highest in the combination with Epistemonikos (Table 3, Fig. 1).

Stratification

Table 4 presents the results obtained for each of the strata investigated. For the combination of MEDLINE, Epistemonikos and reference checking, inclusion rates and their 95% confidence intervals were above the 95% retrieval threshold for all individual strata. Therefore, no context was identified in which this combination produced inadequate retrieval of SRs.

Table 4.

Stratified inclusion rates in database combinations with MEDLINE and reference checking (mean [95% CI])

Stratum CINAHL Embase Epistemonikos PsycINFO TRIP
≥4 dB 94.4% [93.8 to 94.9%] 94.5% [93.9 to 95.1%] 99.3% [99.2 to 99.5%] 94.8% [94.3 to 95.4%] 96.4% [95.9 to 96.9%]
1–3 dB 95.1% [94.5 to 95.8%] 95.7% [95.1 to 96.4%] 98.8% [98.4 to 99.2%] 95.7% [95.2 to 96.3%] 96.0% [95.4 to 96.7%]
Δ in means 0.8% 1.3% −0.5% 0.9% −0.4%
p-value 0.078 0.004 0.009 0.020 0.370
Other sources 93.5% [92.8 to 94.2%] 93.9% [93.2 to 94.6%] 98.9% [98.7 to 99.2%] 94.3% [93.7 to 95.0%] 95.3% [94.7 to 95.9%]
No other sources 96.3% [96.0 to 96.6%] 96.3% [95.9 to 96.6%] 99.6% [99.5 to 99.7%] 96.3% [96.0 to 96.5%] 97.9% [97.8 to 98.1%]
Δ in means 2.8% 2.4% 0.7% 1.9% 2.7%
p-value < 0.001* < 0.001* < 0.001* < 0.001* < 0.001*
HTA report 86.2% [82.8 to 89.6%] 86.2% [82.8 to 89.6%] 97.7% [96.5 to 98.9%] 83.9% [80.7 to 87.1%] 95.4% [93.7 to 97.1%]
Cochrane Overview 97.8% [97.5 to 98.1%] 100.0% 100.0% 97.8% [97.5 to 98.1%] 100.0%
Non-Cochrane Overview 95.1% [94.7 to 95.5%] 95.3% [94.9 to 95.7%] 99.3% [99.1 to 99.4%] 95.9% [95.5 to 96.2%] 96.2% [95.8 to 96.6%]
Δ in means for HTA vs. non-Cochrane 8.9% 9.1% 1.6% 12.0% 0.8%
p-value for HTA vs. non-Cochrane < 0.001* < 0.001* 0.012 < 0.001* 0.274
Intervention Overview 95.0% [94.6 to 95.5%] 95.0% [94.6 to 95.5%] 99.3% [99.1 to 99.5%] 95.5% [95.1 to 95.9%] 96.0% [95.6 to 96.5%]
Non-intervention Overview 93.2% [92.1 to 94.4%] 94.2% [93.1 to 95.3%] 98.7% [98.5 to 99.0%] 93.9% [92.8 to 95.0%] 97.1% [96.7 to 97.6%]
Δ in means −1.8% −0.8% −0.6% −1.6% 1.1%
p-value 0.004 0.182 < 0.001* 0.010 0.001*
Psyc. topic 91.8% [89.8 to 93.8%] 95.9% [94.6 to 97.2%] 100.0% 95.9% [94.5 to 97.4%] 93.9% [92.0 to 95.8%]
No psyc. Topic 94.8% [94.3 to 95.4%] 94.7% [94.1 to 95.3%] 99.1% [98.9 to 99.3%] 95.0% [94.5 to 95.5%] 96.5% [96.1 to 97.0%]
Δ in means 3.0% −1.2% −0.9% −0.9% 2.6%
p-value 0.005 0.093 0.000* 0.243 0.010
Open in a new tab

Best results in italics; all data are weighted. *Individual comparisons are significant if p < 0.0017. Psyc. topic = mental health- or psychology-related topic

Databases searched

The Overviews in the reference set had searched for SRs in 1–25 databases. The inclusion rates calculated for Overviews searching 1–3 databases was roughly equal to that calculated for Overviews searching 4 or more databases.

Other sources searched

Other sources that were searched by Overview authors include reference lists of included studies, queries to experts, Google, Google Scholar, internal, non-public departmental files from the institutions of Overview authors, hand-searching journals, trial registries, publishers’ databases, journals, institutions, and HTA agencies. Because this represents a larger dataset containing SRs that may not all be included in databases, one may expect that database inclusion rates would be smaller in this stratum.

A total of 74% of Overviews (n = 64/86) reported searching at least one other source. In any database combination, the mean inclusion of SRs for Overviews that had searched other sources was only slightly lower than for Overviews that had not searched other sources. For example, the difference was 0.7% for the combination of MEDLINE, Epistemonikos and reference checking.

Overview types

Overviews were classified into three types, i.e. Cochrane Overviews (n = 3), HTA reports (n = 14), and non-Cochrane Overviews (n = 69). Inclusion of SRs from Cochrane Overviews was complete for combinations of MEDLINE plus reference checking and each of the databases Embase, TRIP, and Epistemonikos (Fig. 2). In contrast, inclusion rates were lower for SRs originating from HTA reports in most database combinations. MEDLINE, Epistemonikos, and reference checking remained the best combination, reaching a mean rate of 97.7% included SRs [96.5 to 98.9]. For non-Cochrane Overviews, which made up 80% (n = 69/86) of the total set and contributed 89% (n = 1087/1219) of SRs, the results were similar to the results for the total set. Thus, the SR inclusion rate for MEDLINE, Epistemonikos, and reference checking was 99.3% [99.1 to 99.4%].

Fig. 2.

Fig. 2

Open in a new tab

Inclusion rates in database combinations with MEDLINE and reference checking, stratified by Overview type

Intervention/non-intervention overviews

Intervention Overviews represented 69% of the overall sample. The difference between SRs originating from intervention vs. non-intervention Overviews was minimal for all databases.

Mental health- or psychology-related topics

The present dataset included only 10% (n = 9/86) Overviews in the stratum with a mental health- or psychology-related topic, corresponding to 8% (n = 98/1219) of included SRs. In this stratum, the inclusion rate in PsycINFO combined with MEDLINE and reference checking was slightly higher than in the stratum without psychology-related topics. However, the combination of the non-specialist databases Epistemonikos and MEDLINE with reference checking was superior, because 100% of SRs with a mental health- or psychology-related topic were included in this combination.

Qualitative analysis of missed SRs

Ten SRs (n = 10/1219, 0.8%) were not identified by the best database combination (Table 5). Of these, one was incorrectly cited in the original Overview [116]. The correct citation [126] was included in the databases CINAHL, Epistemonikos, and Embase. Two were evidence syntheses published on the web only, for which alternative versions were found by database searching [124, 125]. Thus, the evidence for 3/10 missed records would have been included in alternative form, resulting in a total coverage of 99.43% (n = 1212/1219) of SRs.

Table 5.

Qualitative analysis of SRs not included in MEDLINE, Epistemonikos and reference checking

Ref. Topic Website DB Inclusion Publisher’s DB Google Scholar
Brunton 2003 [116] Promoting physical activity amongst children outside of physical education classes – – –
Büssing 2009 [117] Impact of yoga on chronic pain – – Y (ScienceDirect) Y
Christopher 1995 [118] Adolescent Pregnancy Prevention – ERIC –
Clamp 2005 [119] The value of electronic health records Y – Y1
Erny-Albrecht 2007 [120] Sublingual immunotherapy in allergic rhinitis and asthma – CINAHL, Embase Y
Evans 2008 [121] Yoga as treatment for chronic pain conditions – CINAHL, Embase Y (De Gruyter) Y
Horta 2007 [122] Evidence on the long-term effects of breastfeeding Y – –
ISMP 2010 [123] Principles of Designing a Medication Label for Injectable Syringes Y – –
Nelson 2004 [124] Screening for ovarian cancer Y –- a Y
Shekelle 2009 [125] Costs and benefits of health information technology Y –- a Y
Open in a new tab

DB database, ERIC Education Resources Information Center. aAlternative versions in several databases including Epistemonikos and MEDLINE. Website = Availability of the Overview available on the internet

Three missed references (n = 3/1219, 0.24%) were websites not included in any of the databases investigated in the present study [119, 122, 123]. Two missed SRs (n = 2/1219, 0.16%) were included in both CINAHL and Embase [120, 121]. One (n = 1/1219, 0.08%) was listed on the publisher’s website (ScienceDirect) and found via Google Scholar [117]. Finally, one was not found via any of the routes explored in this study [118]. However, it was included in the Education Resources Information Center (ERIC) database, sponsored by the Institute of Education Sciences of the U.S. Department of Education [127], which had been searched by the Overview authors [93].

Discussion

We identified MEDLINE/Epistemonikos, complemented by reference checking, as the best combination to retrieve SRs for an Overview of Reviews on health-related topics. This study fill a gap in the evidence around the evaluation of methods in overviews as identified in the MoOR framework by Lunny et al. [5, 128]. Our investigation mapped to the option ‘select the types of databases to search’ in the MoOR framework of all methods used in overviews, and provides clear guidance for authors around which databases to choose when planning an overview.

No single database was found to be sufficiently comprehensive by itself for a systematic search for SRs in the context of preparing Overviews. MEDLINE emerged as the best single source for retrieval of SRs (inclusion rate 89.7%). MEDLINE using the PubMed interface has a good inclusion rate for SRs, and is free of charge and open access.

Among databases assessed in combination with MEDLINE and reference checking (i.e. checking the citation lists of SRs identified in MEDLINE), Epistemonikos gave the highest pooled inclusion rate with 99.2%, followed by TRIP (96.3%). Both Epistemonikos and TRIP are resources specialising in the retrieval of SRs. In spite of the fact that the inclusion rate of SRs was relatively high in Embase (inclusion rate 83.7%), the pooled inclusion rate of MEDLINE, Embase and reference checking was below 95%, possibly due to greater overlap with MEDLINE. The subject-specific databases CINAHL (nursing) and PsycINFO (psychology) had pooled inclusion rates similar to Embase (94.6 and 95.1%, respectively).

Analysis of the different strata did not identify any context in which the combination of MEDLINE, Epistemonikos and reference checking was inadequate.

The difference between the number of SRs retrieved from Overviews that had searched in 4 or more databases was minimal compared to Overviews that had searched in 1–3 databases, casting doubt on the value of searching large numbers of databases without specific justification. When looking at the SRs not covered by our optimal database combination, an additional 0.16% of SRs were included in both CINAHL and Embase, and 0.08% in ERIC. The effect of searching other sources beyond electronic databases (e.g. Google Scholar and publishers’ databases) was found to be similarly small (0.5 and 0.16%, respectively, of studies found in addition to the recommended combination). Nevertheless, a limited number of SRs are published on the web only and not identified by searching the databases investigated in this study (0.24% in this dataset).

The recommended database combination included slightly more SRs from Cochrane and non-Cochrane Overviews than from HTA reports. The better rate of SRs included in Cochrane Overviews (100% of included SRs) is likely a result of guidance for Cochrane Overview authors to search for, and include, only Cochrane SRs [1]. SRs from Cochrane Overviews are more frequently included than from non-Cochrane Overviews (99.3% of included SRs), because all Cochrane Reviews are included in PubMed and other electronic databases [6]. The inclusion rate using the recommended combination was slightly lower for SRs from HTA reports (97.7% of included SRs). This may be because authors of HTA reports typically search more widely, and might thus identify more studies, than authors of Cochrane or non-Cochrane Overviews (median 8 databases searched, compared to 2 databases for Cochrane Overviews and 5 for non-Cochrane Overview; all searches in HTA reports complemented by other methods).

Only one empirical study was found that investigated the types of databases to search [15]. The recall rates in the Rathbone study (88% recall for Cochrane Library and Embase; and 83% for Cochrane Library, Epistemonikos and MEDLINE) are substantially lower than in our study (99.2% indexing in MEDLINE, Epistemonikos and reference checking). Rathbone et al. examined bibliographic database performance in identifying SRs using an approach that was different to ours in several key aspects. First, the study was topic specific and related to interventions for hypertension. Second, the reference set of SRs was generated by searching in the same electronic databases that were later used to analyse retrieval of SRs, and was thus a relative recall analysis based on databases alone [129]. They performed systematic literature searches in seven electronic databases and used search filters for SRs to increase precision. Therefore, their approach primarily evaluated the relative performance of search strategies in bibliographic databases compared to each other, in terms of precision and sensitivity.

In contrast to the Rathbone study [15], our sample was not topic-specific, and included both intervention and non-intervention SRs. We evaluated database coverage independently of the search. Our sample of SRs was larger (1219 vs. 400), and was obtained from published Overviews which had used various methods to identify relevant literature. Our study investigating database inclusion of SRs, as opposed to evaluating the performance of search strategies in different databases.

These methodological differences may explain the different results obtained. In addition, our last search was performed after a major update in the Epistemonikos machine learning algorithms. As a result, a larger number of SRs were included in this resource and retrieved by our search.

Rathbone et al. concluded that “a search of all databases should be performed”. This conclusion is not supported by our results. First, the high database inclusion rate of over 99% that we observed for a combination of two databases and reference checking shows that when a sensitive search strategy is employed, this combination is sufficient for retrieving SRs. Second, in the context of limited resources, it may not be efficient to search additional databases as we found no relevant differences when we stratified inclusion of SRs by Overviews that had searched in 4 or more databases compare with 1 to 3 databases (99.3% vs. 98.8% inclusion, respectively).

Strengths and limitations of the study

The present methodological study is based on a large sample of 1219 SRs. The SRs were extracted from published Overviews, and were found in diverse electronic databases, often complemented by reference checking, queries to experts, searches of the web (e.g. Google, Google Scholar, government reports), or hand-searching journals, thus making our sample representative of searches in a variety of overviews. We analysed the inclusion rate of SR in databases, and have not used any search strategies for the retrieval of SRs. Therefore, our results are independent of the effectiveness of any search strategy employed.

The results of this study apply only to database coverage. Real-world retrieval of SRs will usually be lower because search strategies can be unreliable and less than 100% sensitive. Another limitation relates to the choice of sample. The lack of recency of the dataset of SRs dated from 1982 to 2011 may be limiting, and future efforts should be made to validate this research in a more up-to-date set of SRs. The reference set of SRs was not generated by hand-searching all relevant resources, which may be considered the gold standard, but relies on relative recall [129]. Some Overview authors may have searched for SRs less extensively or less effectively than others, resulting in a possible bias for studies included in databases. This may affect the absolute values for means and confidence intervals generated within this study. In the worst case, if some Overview authors have missed harder-to-find SRs in their searches, this may tend to even out observable differences between databases and bias the results towards higher overall inclusion rates.

The inclusion of SRs in the six databases is likely to change over time, as the terminology used to describe systematic reviews changes, and as tagging of SRs and algorithms for retrieval of SRs improves. For example, the National Library of Medicine has established a new MeSH publication type “Systematic Reviews” in January 2019, and SRs have retrospectively been re-indexed in MEDLINE [130]. A more general search filter, designed to include SRs that had not yet been indexed, is also available [131]. Other databases may decline in inclusion rates, such as DARE and the NHS EED database (not updated since 2015). The reference set of Overviews was limited to those published in English or German, so that the results are not applicable to SRs published in other languages. In addition, some SRs may have been included in more than one Overview, the effect of which was not assessed. The ‘biblio-SRs’, i.e. those obtained by reference checking in this study, may also have been retrievable by other means. Overview authors were not contacted to clarify the origin of these ‘biblio-SRs’. Also, the results may depend on factors not studied, for example, specific diseases, drugs, or other interventions that are more likely to be included in one database than in another. Furthermore, the present study was not designed primarily to quantify differences between strata, so that the results of the stratification analysis should be considered exploratory. Finally, inclusion in databases was not assessed in duplicate, leading to potential errors in extractions and calculations.

Future research

Future research should be directed at validating our findings using an up-to-date reference set, assessing whether it is applicable to languages not evaluated in the present study, and exploring the relative contribution of other methods of study retrieval.

The Cochrane Handbook [1] and the Joanna Briggs Institute [132] currently recommend that a structured search process using multiple databases (e.g. MEDLINE and Embase, and additional regional and subject-specific databases, such as LILACS, CINAHL, PsycINFO) be used to retrieve systematic reviews. This guidance will need to be re-evaluated based on our findings that MEDLINE and Epistemonikos, with reference checking of included studies, is the best strategy for retrieval of systematic reviews.

Conclusion

The present study shows that the literature search for Overviews should be performed in MEDLINE and Epistemonikos, complemented by reference checking.

Supplementary information

12874_2020_983_MOESM1_ESM.xlsx (313.9KB, xlsx)

Additional file 1. Dataset generated and analysed during the current study.

12874_2020_983_MOESM2_ESM.docx (94.1KB, docx)

Additional file 2: Table S1. Characteristics of included Overviews.

Acknowledgments

Zemzem Cinar and Jelena Heitz are gratefully acknowledged for contributing to the assessment of database inclusion. We thank Camilo Vergara for insights into the Epistemonikos search and classification methods.

Abbreviations

CI

Confidence interval

DB

Database

HTA

Health Technology Assessment

IQR

Interquartile range

PICOS

Population, Intervention, Comparator, Outcomes, and Study types

SR

Systematic review

Authors’ contributions

SH performed database searches and collected data. KG designed the analysis methods, analysed and interpreted the data. KG and CL co-wrote the manuscript. DP conceived and supervised the research, and revised the manuscript. All authors read and approved the final manuscript.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Käthe Goossen, Email: kaethe.goossen@uni-wh.de.

Simone Hess, Email: simone.hess@uni-wh.de.

Carole Lunny, Email: carole.lunny@ti.ubc.ca.

Dawid Pieper, Email: dawid.pieper@uni-wh.de.

Supplementary information

Supplementary information accompanies this paper at 10.1186/s12874-020-00983-3.

References

  • 1.Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: Overviews of Reviews. In: JPT H, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated March 2020): Cochrane; 2020. Available from www.training.cochrane.org/handbook.
  • 2.Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Int J Evid Based Healthc. 2015;13(3):132–140. doi: 10.1097/XEB.0000000000000055. [DOI] [PubMed] [Google Scholar]
  • 3.Fusar-Poli P, Radua J. Ten simple rules for conducting umbrella reviews. Evid Based Mental Health. 2018;21(3):95–100. doi: 10.1136/ebmental-2018-300014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Pollock M, Fernandes RM, Becker LA, Featherstone R, Hartling L. What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Syst Rev. 2016;5(1):190. doi: 10.1186/s13643-016-0367-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):231. doi: 10.1186/s13643-017-0617-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Cochrane Library: About the Cochrane Database of Systematic Reviews. https://www.cochranelibrary.com/de/cdsr/about-cdsr. Accessed 6 Sept 2019.
  • 7.NCBI PubMed. https://www.ncbi.nlm.nih.gov/pubmed/. Accessed 23 Mar 2019.
  • 8.Embase. https://www.embase.com. Accessed 23 Mar 2019.
  • 9.Literatura Latino-Americana e do Caribe em Ciências da Saúde [Latin American and Caribbean Center on Health Sciencies Information]: LILACS. http://lilacs.bvsalud.org/en/. Accessed 23 Mar 2019.
  • 10.Cumulative Index for Nursing and Allied Health Literature (CINAHL). https://www.ebscohost.com/nursing/products/cinahl-databases/cinahl-complete. Accessed 23 Mar 2019.
  • 11.EBSCO PsycINFO. https://www.ebsco.com/products/research-databases/psycinfo. Accessed 23 Mar 2019.
  • 12.International prospective register of systematic reviews (PROSPERO). https://www.crd.york.ac.uk/prospero/. Accessed 23 Mar 2019.
  • 13.Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65(12):1267–1273. doi: 10.1016/j.jclinepi.2012.06.015. [DOI] [PubMed] [Google Scholar]
  • 14.Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One. 2012;7(11):e49667. doi: 10.1371/journal.pone.0049667. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Rathbone J, Carter M, Hoffmann T, Glasziou P, et al. Syst Rev. 2016;5:27. doi: 10.1186/s13643-016-0197-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Epistemonikos: Database of the best of Evidence-Based Health Care. https://www.epistemonikos.org/. Accessed 23 Mar 2019.
  • 17.Trip medical database. https://www.tripdatabase.com/. Accessed 23 Mar 2019.
  • 18.Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):245. doi: 10.1186/s13643-017-0644-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Pieper D, Antoine SL, Mathes T, Neugebauer EA, Eikermann M. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol. 2014;67(4):368–375. doi: 10.1016/j.jclinepi.2013.11.007. [DOI] [PubMed] [Google Scholar]
  • 20.Physiotherapy Evidence Database (PEDro). https://search.pedro.org.au/search. Accessed 23 Mar 2019.
  • 21.Luce BR, Drummond M, Jonsson B, Neumann PJ, Schwartz JS, Siebert U, et al. EBM, HTA, and CER: clearing the confusion. Milbank Q. 2010;88(2):256–276. doi: 10.1111/j.1468-0009.2010.00598.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Rada G, Perez D, Capurro D. Epistemonikos: a free, relational, collaborative, multilingual database of health evidence. Stud Health Technol Inform. 2013;192:486–490. [PubMed] [Google Scholar]
  • 23.El-Khayat YM. Epistemonikos (Resource Review) J Med Libr Assoc. 2017;105(4):431–432. [Google Scholar]
  • 24.MEDLINE (R): Description of the database. https://www.nlm.nih.gov/bsd/medline.html. Accessed 27 Aug 2019.
  • 25.Sources searched by Trip. In: Trip database blog. https://blog.tripdatabase.com/2017/10/27/sources-searched-by-trip/. Accessed 7 Mar 2019.
  • 26.R Core Team . R: a language and environment for statistical computing. Vienna: R Foundation for statistical computing; 2018. [Google Scholar]
  • 27.Josh Pasek, with some assistance from Alex Tahk, some code modified from R-core; Additional contributions by Gene Culter and Marcus Schwemmle . weights: Weighting and Weighted Statistics. R package version 1.0. 2018. [Google Scholar]
  • 28.Andersen JH, Fallentin N, Thomsen JF, Mikkelsen S. Risk factors for neck and upper extremity disorders among computers users and the effect of interventions: an overview of systematic reviews. PLoS One. 2011;6(5):e19691. doi: 10.1371/journal.pone.0019691. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Australia Medical Services Advisory Committee . Positron emission tomography for lymphoma: assessment report: Medical Services Advisory Committee. 2009. [Google Scholar]
  • 30.Australia Medical Services Advisory Committee . Human papillomavirus triage test for women with possible or definite low-grade squamous intra-epithelial lesions. 2009. [Google Scholar]
  • 31.Australia Medical Services Advisory Committee . Positron emission tomography for cervical cancer : assessment report: Medical Services Advisory Committee. 2010. [Google Scholar]
  • 32.Australia Medical Services Advisory Committee . Positron emission tomography for glioma: Medical Services Advisory Committee. 2010. [Google Scholar]
  • 33.Australia Medical Services Advisory Committee. Khafagi F, Atherton J, Gracanin A, Kearney B, Pitman A. Positron emission tomography for myocardial viability: assessment report: Medical Services Advisory Committee. 2010. [Google Scholar]
  • 34.Beattie M, Taylor J. Silver alloy vs. uncoated urinary catheters: a systematic review of the literature. J Clin Nurs. 2011;20(15–16):2098–2108. doi: 10.1111/j.1365-2702.2010.03561.x. [DOI] [PubMed] [Google Scholar]
  • 35.Berkhof M, van Rijssen HJ, Schellart AJ, Anema JR, van der Beek AJ. Effective training strategies for teaching communication skills to physicians: an overview of systematic reviews. Patient Educ Couns. 2011;84(2):152–162. doi: 10.1016/j.pec.2010.06.010. [DOI] [PubMed] [Google Scholar]
  • 36.Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011;8(1):e1000387. doi: 10.1371/journal.pmed.1000387. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Brouwers MC, Garcia K, Makarski J, Daraz L. The landscape of knowledge translation interventions in cancer control: what do we know and where to next? A review of systematic reviews. Implement Sci. 2011;6:130. doi: 10.1186/1748-5908-6-130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Clar CBK, Cummins E, Royle P, Waugh N. Self-monitoring of blood glucose in type 2 diabetes: systematic review. Health Technol Assess (Winchester, England). 2010;14(12). 10.3310/hta14120. [DOI] [PubMed]
  • 39.Damen A, Remmen R, Wens J, Paulus D. Evidence based post graduate training. A systematic review of reviews based on the WFME quality framework. BMC Med Educ. 2011;11:80. doi: 10.1186/1472-6920-11-80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.De Niet GJ, Tiemens BG, Kloos MW, Hutschemaekers GJ. Review of systematic reviews about the efficacy of non-pharmacological interventions to improve sleep quality in insomnia. Int J Evid Based Healthc. 2009;7(4):233–242. doi: 10.1111/j.1744-1609.2009.00142.x. [DOI] [PubMed] [Google Scholar]
  • 41.de Vet E, de Ridder DT, de Wit JB. Environmental correlates of physical activity and dietary behaviours among young people: a systematic review of reviews. Obes Rev. 2011;12(5):e130–e142. doi: 10.1111/j.1467-789X.2010.00784.x. [DOI] [PubMed] [Google Scholar]
  • 42.Diener MK, Voss S, Jensen K, Buchler MW, Seiler CM. Elective midline laparotomy closure: the INLINE systematic review and meta-analysis. Ann Surg. 2010;251(5):843–856. doi: 10.1097/SLA.0b013e3181d973e4. [DOI] [PubMed] [Google Scholar]
  • 43.Hagen A, Gorenoi V, Schönermark MP. Spezifische Immuntherapie (SIT) zur Behandlung der allergischen Rhinitis. Schriftenreihe Health Technology Assessment. 2010. p. Bd. 96. [Google Scholar]
  • 44.Gorenoi V, Schönermark MP, Hagen A. Perkutane Koronarinterventionen zusätzlich zur optimalen medikamentösen Therapie bei stabiler Angina Pectoris Schriftenreihe Health Technology Assessment. 2011. p. Bd. 115. [Google Scholar]
  • 45.Schnell-Inderst P, Hunger T, Hintringer K, Schwarzer R, Seifert-Klauss V, Gothe H, et al. Individuelle Gesundheitsleistungen. Schriftenreihe Health Technology Assessment. 2011. p. Bd. 113. [Google Scholar]
  • 46.Ekeland AG, Bowes A, Flottorp S. Effectiveness of telemedicine: a systematic review of reviews. Int J Med Inform. 2010;79(11):736–771. doi: 10.1016/j.ijmedinf.2010.08.006. [DOI] [PubMed] [Google Scholar]
  • 47.Ernst E, Lee M. Ayurvedic medicine: an overview of systematic reviews. Perfusion. 2010;23:132–140. [Google Scholar]
  • 48.Ernst E, Lee MS. Acupressure: an overview of systematic reviews. J Pain Symptom Manag. 2010;40(4):e3–e7. doi: 10.1016/j.jpainsymman.2010.07.003. [DOI] [PubMed] [Google Scholar]
  • 49.Ernst E, Lee MS. Acupuncture for rheumatic conditions: an overview of systematic reviews. Rheumatology (Oxford, England) 2010;49(10):1957–1961. doi: 10.1093/rheumatology/keq180. [DOI] [PubMed] [Google Scholar]
  • 50.Ernst E, Lee MS. How effective is yoga? A concise overview of systematic reviews. Focus Altern Complement Ther. 2010;15(4). 10.1111/j.2042-7166.2010.01049.x.
  • 51.Ernst E, Lee MS, Choi TY. Acupuncture for depression?: a systematic review of systematic reviews. Eval Health Prof. 2011;34(4):403–412. doi: 10.1177/0163278710386109. [DOI] [PubMed] [Google Scholar]
  • 52.Ernst E, Lee MS, Choi TY. Acupuncture for insomnia? An overview of systematic reviews. Eur J Gen Pract. 2011;17(2):116–123. doi: 10.3109/13814788.2011.568475. [DOI] [PubMed] [Google Scholar]
  • 53.Ernst E, Posadzki P. Complementary and alternative medicine for rheumatoid arthritis and osteoarthritis: an overview of systematic reviews. Curr Pain Headache Rep. 2011;15(6):431–437. doi: 10.1007/s11916-011-0227-x. [DOI] [PubMed] [Google Scholar]
  • 54.Ernst E, Posadzki P, Lee MS. Complementary and alternative medicine (CAM) for sexual dysfunction and erectile dysfunction in older men and women: an overview of systematic reviews. Maturitas. 2011;70(1):37–41. doi: 10.1016/j.maturitas.2011.06.011. [DOI] [PubMed] [Google Scholar]
  • 55.Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev. 2011;(7):Cd009255. 10.1002/14651858.cd009255. Epub 2011/07/08. PubMed PMID: 21735443; PubMed Central PMCID: PMCPmc4204491. [DOI] [PMC free article] [PubMed]
  • 56.Gibson M, Petticrew M, Bambra C, Sowden AJ, Wright KE, Whitehead M. Housing and health inequalities: a synthesis of systematic reviews of interventions aimed at different pathways linking housing and health. Health Place. 2011;17(1):175–184. doi: 10.1016/j.healthplace.2010.09.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.Greaves CJ, Sheppard KE, Abraham C, Hardeman W, Roden M, Evans PH, et al. Systematic review of reviews of intervention components associated with increased effectiveness in dietary and physical activity interventions. BMC Public Health. 2011;11:119. doi: 10.1186/1471-2458-11-119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Heighes PT, Doig GS, Sweetman EA, Simpson F. An overview of evidence from systematic reviews evaluating early enteral nutrition in critically ill patients: more convincing evidence is needed. Anaesth Intensive Care. 2010;38(1):167–174. doi: 10.1177/0310057X1003800126. [DOI] [PubMed] [Google Scholar]
  • 59.Hillberg T, Hamilton-Giachritsis C, Dixon L. Review of meta-analyses on the association between child sexual abuse and adult mental health difficulties: a systematic approach. Trauma Violence Abuse. 2011;12(1):38–49. doi: 10.1177/1524838010386812. [DOI] [PubMed] [Google Scholar]
  • 60.Hopton A, MacPherson H. Acupuncture for chronic pain: is acupuncture more than an effective placebo? A systematic review of pooled data from meta-analyses. Pain Practi. 2010;10(2):94–102. doi: 10.1111/j.1533-2500.2009.00337.x. [DOI] [PubMed] [Google Scholar]
  • 61.Huguet F, Girard N, Guerche CS, Hennequin C, Mornex F, Azria D. Chemoradiotherapy in the management of locally advanced pancreatic carcinoma: a qualitative systematic review. J Clin Oncol. 2009;27(13):2269–2277. doi: 10.1200/JCO.2008.19.7921. [DOI] [PubMed] [Google Scholar]
  • 62.Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Nutzenbewertung nichtmedikamentöser Behandlungsstrategien bei Patienten mit essenzieller Hypertonie: Kochsalzreduktion. Rapid Rep. 2009:A05–21B. Available from: https://www.iqwig.de/download/A05-21B_Rapid_Report_Nichtmedikamentoese_Behandlungsstrategien_bei_Hypertonie_Kochsalzreduktion.pdf.
  • 63.Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen . Positronenemissionstomographie (PET und PET/CT) bei malignen Lymphomen. IQWiG-Berichte. 2009. p. Nr. 47. [Google Scholar]
  • 64.Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen . Positronenemissionstomographie (PET) und PET/CT bei malignem Melanom. IQWiG-Berichte. 2011. p. Nr. 86. [Google Scholar]
  • 65.Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen . Positronenemissionstomographie (PET) und PET/CT bei Kopf- und Halstumoren. IQWiG-Berichte. 2011. p. Nr. 82. [Google Scholar]
  • 66.Jepson RG, Harris FM, Platt S, Tannahill C. The effectiveness of interventions to change six health behaviours: a review of reviews. BMC Public Health. 2010;10:538. doi: 10.1186/1471-2458-10-538. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Kamioka H, Tsutani K, Okuizumi H, Mutoh Y, Ohta M, Handa S, et al. Effectiveness of aquatic exercise and balneotherapy: a summary of systematic reviews based on randomized controlled trials of water immersion therapies. J Epidemiol. 2010;20(1):2–12. doi: 10.2188/jea.JE20090030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Kang HS, Jeong D, Kim DI, Lee MS. The use of acupuncture for managing gynaecologic conditions: an overview of systematic reviews. Maturitas. 2011;68(4):346–354. doi: 10.1016/j.maturitas.2011.02.001. [DOI] [PubMed] [Google Scholar]
  • 69.Keus F, Gooszen HG, van Laarhoven CJ. Open, small-incision, or laparoscopic cholecystectomy for patients with symptomatic cholecystolithiasis. An overview of Cochrane Hepato-Biliary Group reviews. Cochrane Database Syst Rev. 2010;(1):Cd008318. 10.1002/14651858.cd008318. Epub 2010/01/22. PubMed PMID: 20091665. [DOI] [PMC free article] [PubMed]
  • 70.Khambalia AZ, Dickinson S, Hardy LL, Gill T, Baur LA. A synthesis of existing systematic reviews and meta-analyses of school-based behavioural interventions for controlling and preventing obesity. Obe Rev:214–33. 10.1111/j.1467-789X.2011.00947.x. Epub 2011/11/11. PubMed PMID: 22070186. [DOI] [PubMed]
  • 71.Kumar A, Galeb S, Djulbegovic B. Treatment of patients with multiple myeloma: an overview of systematic reviews. Acta Haematol. 2011;125(1–2):8–22. doi: 10.1159/000318880. [DOI] [PubMed] [Google Scholar]
  • 72.Lakke SE, Soer R, Takken T, Reneman MF. Risk and prognostic factors for non-specific musculoskeletal pain: a synthesis of evidence from systematic reviews classified into ICF dimensions. Pain. 2009;147(1–3):153–64. 10.1016/j.pain.2009.08.032. Epub 2009/10/06. PubMed PMID: 19800735. [DOI] [PubMed]
  • 73.Fischer S, Zechmeister I. Sakralnervenstimulation bei fäkaler Inkontinenz. Rapid Assessment. Wien: Ludwig Boltzmann Institut für Health Technology Assessment; 2011. p. Nr. 004. [Google Scholar]
  • 74.Lee MS, Kim JI, Ernst E. Is cupping an effective treatment? An overview of systematic reviews. J Acupunct Meridian Studies. 2011;4(1):1–4. doi: 10.1016/S2005-2901(11)60001-0. [DOI] [PubMed] [Google Scholar]
  • 75.Lenz M, Richter T, Mühlhauser I. Morbidität und Mortalität bei Übergewicht und Adipositas im Erwachsenenalter. Dtsch Arztebl Int. 2009;106(40):641–648. doi: 10.3238/arztebl.2009.0641. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 76.List T, Axelsson S. Management of TMD: evidence from systematic reviews and meta-analyses. J Oral Rehabil. 2010;37(6):430–451. doi: 10.1111/j.1365-2842.2010.02089.x. [DOI] [PubMed] [Google Scholar]
  • 77.Maniglio R. The impact of child sexual abuse on health: a systematic review of reviews. Clin Psychol Rev. 2009;29(7):647–657. doi: 10.1016/j.cpr.2009.08.003. [DOI] [PubMed] [Google Scholar]
  • 78.Maniglio R. Child sexual abuse in the etiology of depression: a systematic review of reviews. Depress Anxiety. 2010;27(7):631–642. doi: 10.1002/da.20687. [DOI] [PubMed] [Google Scholar]
  • 79.Matheson SL, Green MJ, Loo C, Carr VJ. Quality assessment and comparison of evidence for electroconvulsive therapy and repetitive transcranial magnetic stimulation for schizophrenia: a systematic meta-review. Schizophr Res. 2010;118(1–3):201–210. doi: 10.1016/j.schres.2010.01.002. [DOI] [PubMed] [Google Scholar]
  • 80.Matheson SL, Shepherd AM, Laurens KR, Carr VJ. A systematic meta-review grading the evidence for non-genetic risk factors and putative antecedents of schizophrenia. Schizophr Res. 2011;133(1–3):133–42. 10.1016/j.schres.2011.09.020. Epub 2011/10/18. PubMed PMID: 21999904. [DOI] [PubMed]
  • 81.Matyas E, Jeitler K, Horvath K, Semlitsch T, Hemkens LG, Pignitter N, et al. Benefit assessment of salt reduction in patients with hypertension: systematic overview. J Hypertens. 2011;29(5):821–828. doi: 10.1097/HJH.0b013e3283442840. [DOI] [PubMed] [Google Scholar]
  • 82.Melnik T, Soares BG, Puga ME, Atallah AN. Efficacy and safety of atypical antipsychotic drugs (quetiapine, risperidone, aripiprazole and paliperidone) compared with placebo or typical antipsychotic drugs for treating refractory schizophrenia: overview of systematic reviews. Sao Paulo Med J. 2010;128(3):141–166. doi: 10.1590/S1516-31802010000300007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 83.Mikton C, Butchart A. Child maltreatment prevention: a systematic review of reviews. Bull World Health Organ. 2009;87(5):353–361. doi: 10.2471/BLT.08.057075. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84.Minozzi S, Davoli M, Bargagli AM, Amato L, Vecchi S, Perucci CA. An overview of systematic reviews on cannabis and psychosis: discussing apparently conflicting results. Drug Alcohol Rev. 2010;29(3):304–317. doi: 10.1111/j.1465-3362.2009.00132.x. [DOI] [PubMed] [Google Scholar]
  • 85.Moe RH, Kjeken I, Uhlig T, Hagen KB. There is inadequate evidence to determine the effectiveness of nonpharmacological and nonsurgical interventions for hand osteoarthritis: an overview of high-quality systematic reviews. Phys Ther. 2009;89(12):1363–1370. doi: 10.2522/ptj.20080398. [DOI] [PubMed] [Google Scholar]
  • 86.Monasta L, Batty GD, Cattaneo A, Lutje V, Ronfani L, Van Lenthe FJ, et al. Early-life determinants of overweight and obesity: a review of systematic reviews. Obes Rev. 2010;11(10):695–708. doi: 10.1111/j.1467-789X.2010.00735.x. [DOI] [PubMed] [Google Scholar]
  • 87.Muangpaisan W, Brayne C. Systematic review of statins for the prevention of vascular dementia or dementia. Geriatr Gerontol Int. 2010;10(2):199–208. doi: 10.1111/j.1447-0594.2009.00579.x. [DOI] [PubMed] [Google Scholar]
  • 88.O'Connor EA, Whitlock EP, Gaynes B, Beil TL. U.S. Preventive Services Task Force Evidence Syntheses, formerly Systematic Evidence Reviews. Screening for Depression in Adults and Older Adults in Primary Care: An Updated Systematic Review. Rockville: Agency for Healthcare Research and Quality (US); 2009. [PubMed] [Google Scholar]
  • 89.Oestergaard S, Moldrup C. Improving outcomes for patients with depression by enhancing antidepressant therapy with non-pharmacological interventions: a systematic review of reviews. Public Health. 2011;125(6):357–367. doi: 10.1016/j.puhe.2011.02.001. [DOI] [PubMed] [Google Scholar]
  • 90.Onakpoya IJ, Wider B, Pittler MH, Ernst E. Food supplements for body weight reduction: a systematic review of systematic reviews. Obesity (Silver Spring, Md) 2011;19(2):239–244. doi: 10.1038/oby.2010.185. [DOI] [PubMed] [Google Scholar]
  • 91.Oxman AD, Fretheim A. Can paying for results help to achieve the millennium development goals? Overview of the effectiveness of results-based financing. J Evid Based Med. 2009;2(2):70–83. doi: 10.1111/j.1756-5391.2009.01020.x. [DOI] [PubMed] [Google Scholar]
  • 92.Palmateer N, Kimber J, Hickman M, Hutchinson S, Rhodes T, Goldberg D. Evidence for the effectiveness of sterile injecting equipment provision in preventing hepatitis C and human immunodeficiency virus transmission among injecting drug users: a review of reviews. Addiction (Abingdon, England) 2010;105(5):844–859. doi: 10.1111/j.1360-0443.2009.02888.x. [DOI] [PubMed] [Google Scholar]
  • 93.Peters LW, Kok G, Ten Dam GT, Buijs GJ, Paulussen TG. Effective elements of school health promotion across behavioral domains: a systematic review of reviews. BMC Public Health. 2009;9:182. doi: 10.1186/1471-2458-9-182. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 94.Posadzki P, Ernst E. Spinal manipulation: an update of a systematic review of systematic reviews. N Z Med J. 2011;124(1340):55–71. [PubMed] [Google Scholar]
  • 95.Posadzki P, Ernst E. Systematic reviews of spinal manipulations for headaches: an attempt to clear up the confusion. Headache. 2011;51(9):1419–1425. doi: 10.1111/j.1526-4610.2011.01888.x. [DOI] [PubMed] [Google Scholar]
  • 96.Ryan R, Santesso N, Hill S, Lowe D, Kaufman C, Grimshaw J. Consumer-oriented interventions for evidence-based prescribing and medicines use: an overview of systematic reviews. Cochrane Database Syst Rev. 2011;(5):Cd007768 10.1002/14651858.CD007768.pub2. Epub 2011/05/13. PubMed PMID: 21563160. [DOI] [PubMed]
  • 97.Safron M, Cislak A, Gaspar T, Luszczynska A. Micro-environmental characteristics related to body weight, diet, and physical activity of children and adolescents: a systematic umbrella review. Int J Environ Health Res. 2011;21(5):317–330. doi: 10.1080/09603123.2011.552713. [DOI] [PubMed] [Google Scholar]
  • 98.Safron M, Cislak A, Gaspar T, Luszczynska A. Effects of school-based interventions targeting obesity-related behaviors and body weight change: a systematic umbrella review. Behav Med (Washington, DC) 2011;37(1):15–25. doi: 10.1080/08964289.2010.543194. [DOI] [PubMed] [Google Scholar]
  • 99.Salvo F, Fourrier-Reglat A, Bazin F, Robinson P, Riera-Guardia N, Haag M, et al. Cardiovascular and gastrointestinal safety of NSAIDs: a systematic review of meta-analyses of randomized clinical trials. Clin Pharmacol Ther. 2011;89(6):855–866. doi: 10.1038/clpt.2011.45. [DOI] [PubMed] [Google Scholar]
  • 100.Savard LA, Thompson DR, Clark AM. A meta-review of evidence on heart failure disease management programs: the challenges of describing and synthesizing evidence on complex interventions. Trials. 2011;12:194. doi: 10.1186/1745-6215-12-194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 101.Schneider Chafen JJ, Newberry SJ, Riedl MA, Bravata DM, Maglione M, Suttorp MJ, et al. Diagnosing and managing common food allergies: a systematic review. JAMA. 2010;303(18):1848–1856. doi: 10.1001/jama.2010.582. [DOI] [PubMed] [Google Scholar]
  • 102.Seida JK, Ospina MB, Karkhaneh M, Hartling L, Smith V, Clark B. Systematic reviews of psychosocial interventions for autism: an umbrella review. Dev Med Child Neurol. 2009;51(2):95–104. doi: 10.1111/j.1469-8749.2008.03211.x. [DOI] [PubMed] [Google Scholar]
  • 103.Smith CA, Cochrane S. Does acupuncture have a place as an adjunct treatment during pregnancy? A review of randomized controlled trials and systematic reviews. Birth (Berkeley, Calif) 2009;36(3):246–253. doi: 10.1111/j.1523-536X.2009.00329.x. [DOI] [PubMed] [Google Scholar]
  • 104.Smith V, Devane D, Begley CM, Clarke M, Higgins S. A systematic review and quality assessment of systematic reviews of randomised trials of interventions for preventing and treating preterm birth. Eur J Obstet Gynecol Reprod Biol. 2009;142(1):3–11. doi: 10.1016/j.ejogrb.2008.09.008. [DOI] [PubMed] [Google Scholar]
  • 105.Smith CA, Carmady B. Acupuncture to treat common reproductive health complaints: an overview of the evidence. Auton Neurosci. 2010;157(1–2):52–56. doi: 10.1016/j.autneu.2010.03.013. [DOI] [PubMed] [Google Scholar]
  • 106.Spearing NM, Connelly LB. Is compensation “bad for health”? A systematic meta-review. Injury. 2011;42(1):15–24. doi: 10.1016/j.injury.2009.12.009. [DOI] [PubMed] [Google Scholar]
  • 107.Swinkels A, Cochrane K, Burt A, Johnson L, Lunn T, Rees AS. Exercise interventions for non-specific low back pain: an overview of systematic reviews. Phys Ther Rev. 2009;14(4):247–259. [Google Scholar]
  • 108.Tamayo-Velazquez MI, Simon-Lorda P, Villegas-Portero R, Higueras-Callejon C, Garcia-Gutierrez JF, Martinez-Pecino F, et al. Interventions to promote the use of advance directives: an overview of systematic reviews. Patient Educ Couns. 2010;80(1):10–20. doi: 10.1016/j.pec.2009.09.027. [DOI] [PubMed] [Google Scholar]
  • 109.Terry R, Perry R, Ernst E. An overview of systematic reviews of complementary and alternative medicine for fibromyalgia. Clin Rheumatol. 2012;31(1):55–66. doi: 10.1007/s10067-011-1783-5. [DOI] [PubMed] [Google Scholar]
  • 110.Trudeau M, Green E, Cosby R, Charbonneau F, Easty T, Ko Y, et al. Key components of intravenous chemotherapy labeling: a systematic review and practice guideline. J Oncol Pharm Pract. 2011;17(4):409–424. doi: 10.1177/1078155210385160. [DOI] [PubMed] [Google Scholar]
  • 111.van Achterberg T, Huisman-de Waal GG, Ketelaar NA, Oostendorp RA, Jacobs JE, Wollersheim HC. How to promote healthy behaviours in patients? An overview of evidence for behaviour change techniques. Health Promot Int. 2011;26(2):148–162. doi: 10.1093/heapro/daq050. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 112.van der Feltz-Cornelis CM, Sarchiapone M, Postuvan V, Volker D, Roskar S, Grum AT, et al. Best practice elements of multilevel suicide prevention strategies: a review of systematic reviews. Crisis. 2011;32(6):319–333. doi: 10.1027/0227-5910/a000109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 113.Zwicker JG, Mayson TA. Effectiveness of treadmill training in children with motor impairments: an overview of systematic reviews. Pediatr Physical Ther. 2010;22(4):361–377. doi: 10.1097/PEP.0b013e3181f92e54. [DOI] [PubMed] [Google Scholar]
  • 114.NIHR Centre for Reviews and Dissemination (CRD): Database of Abstracts of Reviews and Effects (DARE). https://www.crd.york.ac.uk/CRDWeb/. .
  • 115.Canadian Agency for Drugs and Technologies in Health. https://www.cadth.ca/. Accessed 25 Feb 2020.
  • 116.Brunton G, Harden A, Rees R, Kavanagh J, Oliver S, Oakley A. Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention and qualitative studies. London: EPPI Centre; 2003. [Google Scholar]
  • 117.Büssing A, Schnepp W, Ostermann T, Neugebauer E. Impact of yoga on chronic pain – a review. Eur J Integr Med. 2009;1:236–237. [Google Scholar]
  • 118.Christopher FS. Adolescent Pregnancy Prevention. Family Relations, Vol. 44, No. 4, Helping Contemporary Families (Oct., 1995), pp. 384–391. https://www.jstor.org/stable/pdf/584994.pdf. Accessed 25 Apr 2019.
  • 119.Clamp S, Keen J. The value of electronic health records: A literature review. Final report, December 2005. Yorkshire Centre for Health Informatics, University of Leeds. Available from: https://www.researchgate.net/publication/240702496. Accessed 25 Apr 2019.
  • 120.Erny-Albrecht K, Valentine WJ, Christensen J, Vestenbaek U, Palmer AJ. Sublingual immunotherapy in allergic rhinitis and asthma: a review of recent clinical evidence. J Appl Res. 2007;7(1):17–31. [Google Scholar]
  • 121.Evans S, Subramanian S, Sternlieb D. Yoga as treatment for chronic pain conditions: a literature review. Int J Disabil Hum Dev. 2008;7:25–32. [Google Scholar]
  • 122.Horta BL, Bahl R, Martinés JC, Victora CG. Evidence on the long-term effects of breastfeeding : systematic review and meta-analyses: World Health Organization. 2007. [Google Scholar]
  • 123.Institute for Safe Medication Practices US. ‘Principles of Designing a Medication Label for Injectable Syringes for Patient Specific, Inpatient Use’, original website http://www.ismp.org/Tools/guidelines/labelFormats/injectionSyringe.asp (accessed by the Overview authors in March 2008; no longer accessible); updated website https://www.ismp.org/recommendations/designing-medication-label-injectable-syringes. Accessed 27 Feb 2018.
  • 124.Nelson H, Westhoff C, Piepert J, Berg A. Screening for ovarian cancer: brief evidence update: AHRQ; 2004. https://www.uspreventiveservicestaskforce.org/Home/GetFile/1/900/ovcanup/pdf. Accessed 25 Apr 2019.
  • 125.Shekelle PG, Goldzweig CL. Costs and benefits of health information technology: an updated systematic review. Southern California Evidence-based Practice Centre RAND Corporation. https://www.health.org.uk/publications/costs-and-benefits-of-health-technology-information. Accessed 25 Apr 2019.
  • 126.Brunton G, Thomas J, Harden A, Rees R, Kavanagh J, Oliver S, et al. Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention studies and qualitative studies. Health Educ J. 2005;64(4):323–338. [Google Scholar]
  • 127.ERIC - Education Resources Information Center. https://www.eric.ed.gov/.
  • 128.Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev. 2018;7(1):159. doi: 10.1186/s13643-018-0784-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 129.Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, et al. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Med Res Methodol. 2006;6:33. doi: 10.1186/1471-2288-6-33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 130.MEDLINE Data Changes - 2019. NLM Technical Bulletin, 2018 Nov-Dec. https://www.nlm.nih.gov/pubs/techbull/nd18/nd18_medline_data_changes_2019.html. Accessed 6 Sept2019.
  • 131.National Library of Medicine: Search Strategy Used to Create the PubMed Systematic Reviews Filter. www.nlm.nih.gov/bsd/pubmed_subsets/sysreviews_strategy.html. Accessed 20 Feb 2020.
  • 132.Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P. Chapter 10: Umbrella Reviews. In: Aromataris E, Munn Z, editors. Joanna Briggs Institute Reviewer's Manual: The Joanna Briggs Institute. 2017. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

12874_2020_983_MOESM1_ESM.xlsx (313.9KB, xlsx)

Additional file 1. Dataset generated and analysed during the current study.

12874_2020_983_MOESM2_ESM.docx (94.1KB, docx)

Additional file 2: Table S1. Characteristics of included Overviews.

Data Availability Statement

All data generated or analysed during this study are included in this published article and its supplementary information files.

Articles from BMC Medical Research Methodology are provided here courtesy of BMC

RESOURCES