Introduction to Umbrella Reviews as a Useful Evidence-Based Practice

Abstract

The umbrella review (UR) is a novel methodological approach that has been established to address the ever-expanding research volume of published systematic reviews. In this study, we examined the rationale underlying URs, the concepts and goals of URs, and their applicability in clinical settings. Additionally, we briefly assessed the process of conducting URs and discussed the current challenges in this regard. URs are used to integrate, evaluate, and synthesize the findings of related systematic reviews. By organizing and summarizing the abundant information in accordance with the level of evidence, URs can serve as a useful methodological tool and provide appropriate entry points to clinicians or decision-makers in the medical field. Considering the availability of many suitable interventions for specific conditions in a broad field, URs can enable evidence-based decision-making and offer a broad perspective for the resolution of issues in healthcare by summarizing the evidence and providing directions on a variety of topics. URs are clearly contributing to the management of the deluge of evidence in evidence-based medicine. However, despite the availability of several directions for conducting URs, some points of confusion persist, especially when determining the certainty of evidence. Therefore, advanced guidelines for the appropriate performance of URs are required to provide more reliable evidence through URs.

INTRODUCTION

The surge in published research in recent decades has led to an abundance of data aiming to provide objective information on a variety of topics.¹,² As a result of this overwhelming volume of evidence, decision-making on specific topics and the application of appropriate methodologies have grown increasingly challenging. Furthermore, the utilization of high-quality evidence that can be practically used or applied from a plethora of evidence has become difficult. This awareness has driven the need for systematic reviews that provide high-quality evidence collected and improved using a rigorous process to provide decision-makers access to the most useful information when making decisions on a health-related topic.³,⁴ In this situation, evidence synthesis via systematic reviews has become more significant, and the volume of evidence produced by systematic reviews has been rapidly expanding. Numerous research groups use various strategies to perform systematic reviews on certain issues, which are then published in multiple journals around the same time.¹,² One recent report suggested that more than 10 systematic reviews are published every day.¹

The solution for logically and appropriately dealing with overflowing evidence is to systematically review the previously published systematic reviews. Thus, there is a need for a methodology to objectively collect and summarize published evidence on given topics as well as to compare and evaluate them. These methodologies have been referred to as umbrella reviews (URs), overviews of reviews, reviews of reviews, summaries of systematic reviews, and syntheses of reviews.³,⁴,⁵ In the present study, we will refer to them as URs, since, metaphorically, these reviews allow researchers to synthesize and assess the evidence while peering at the water streaking under the umbrella under the pouring rain of evidence.⁶ While a systematic review involves collection and synthesis of data from several primary studies that is presented statistically, an UR suggests that data are collected, summarized, and presented comprehensively.⁷ URs can thus provide a broad perspective on topics in health-related fields. Although many researchers have recently conducted and reported URs, this methodology is still regarded as novel and unfamiliar.⁸ Therefore, in this review, we aim to provide a general overview of URs and to address issues related to the conduct of URs (Table 1).

Table 1. Checklist for UR⁵,²²,²⁴ .

Section	Preferred reporting items of UR
Title	Declare the report to be an UR.
Abstract	Describe the purpose, methods, and results of UR.
Introduction	Explain the justification for performing the UR and indicate clearly what objective(s) and question(s) the UR aims to address.
Methods	1)	Eligibility criteria and information source: specify the UR’s inclusion and exclusion criteria and all databases or the other information sources. Indicate a date when each source was last searched.
	2)	Study selection: display all search strategies for all information sources including databases so that they may be reproduced.
	3)	Data collection: describe the methods used to collect data from studies. List and specify all variables and outcomes for which data were collected.
	4)	Risk of bias assessment: describe the methods used to evaluate risk of bias or methodological quality.
	5)	Data synthesis: describe the methods used to summarize or synthesize the results; investigate potential sources of heterogeneity in results; identify sensitivity analyses to evaluate the robustness of results.
	6)	Reporting bias assessment: describe the methods used to evaluate the risk of bias due to missing results.
	7)	Certainty assessment: describe the methods used to evaluate certainty or confidence in the body of evidence for an outcome.
Results	1)	Report the results of study search and selection.
	2)	List the specifics of each systematic review and supplemental primary study and display its characteristics.
	3)	Describe the extent of which the primary studies in the included systematic reviews overlap.
	4)	Present the evaluation of risk of bias or methodological quality of each included systematic review.
	5)	Provide summarization of the data from included systematic reviews.
		If meta-analyses were performed, display each summary estimate and measure of its precision, heterogeneity measure, and results of sensitivity analyses.
	6)	Present assessment of the risk of bias due to missing primary studies.
	7)	Present assessment of evidence certainty in the body of evidence for each outcome.
Discussion	1)	Summarize the main findings and offer an overall interpretation of the findings.
	2)	Discuss the limitations on the data from systematic reviews and their primary studies.
	3)	Discuss the limitations on any UR methodology applied.
	4)	Discuss the implications for future study, policy, and practice.
	5)	Consider the findings’ significance for healthcare providers, policymakers, and patients.

UR, umbrella review.

WHY TO USE AN UR

1. Purpose of URs

The UR is a relatively new methodological concept for synthesizing the evidence derived from systematic reviews. Using this approach, researchers can evaluate widely distributed evidence systematically and rapidly by summarizing the data reported in systematic reviews. URs are conducted to provide a comprehensive overview of the knowledge available on a specific topic, as well as to compare and contrast the findings of published systematic reviews.³ Systematic reviews are the fundamental unit of analysis for URs and serve as their foundation. From this perspective, URs can be regarded as one of the highest levels of evidence summarization in the body of biomedical literature (Fig. 1).⁹ URs are most useful when numerous systematic reviews exist on a related topic, and they can be used for systematically integrating, evaluating, and aggregating the results of systematic reviews. URs should not reveal the evidence of the included reviews sequentially or narratively, and cannot, like systematic reviews, be conducted using only one systematic review.³,⁷

Fig. 1. Hierarchy of evidence synthesis methods.

The purpose of URs is not to repeat study eligibility, assessment of risk of bias, or meta-analysis of the included studies, but rather to present a comprehensive picture of the results for a certain topic.¹⁰ For example, a UR may provide a larger overview of various therapies by collecting and summarizing the data from multiple systematic reviews for the same condition, where systematic reviews present a limited perspective by comparing treatments for a single intervention. The UR will thus determine whether to evaluate different interventions for the same condition or to investigate the same intervention and condition. Furthermore, researchers who conduct URs may desire to combine summaries for one or more domains for various conditions or population groups.

2. Usefulness of URs in clinical realty

The development of the review methodology to answer the question of which treatments or interventions actually work contributed to the rise of systematic reviews.¹¹ There are questions that can be used to assess systematic reviews by concentrating on intrinsically qualitative aspects, such as the effectiveness of the intervention, the safety of the intervention, and the participants’ experience of the intervention, without focusing on the effectiveness of the intervention.¹² Thus, URs are appropriate for showing whether the evidence base for a topic is consistent or conflicting, as well as for investigating the causes for certain outcomes. URs have evolved as a result of a growing demand in the medical field to filter information overload, increase access to the desired information, and support decision-making. URs can thus provide useful decision-making guidance for clinicians, healthcare workers, and policy-makers.¹³ Nevertheless, several factors should be considered before deciding to conduct a UR, such as identifying areas where the methodology is unclear and the evidence is ambiguous. This aspect will be discussed at the end of the present review.

URs can be used to combine and summarize the results of various systematic reviews for a given query. URs aim to connect systematic reviews that provide high-quality evidence in order to supply general consistency.³ Unlike systematic reviews, which are limited to comparing one treatment or one intervention, URs can be used when an overall picture is needed for a disease or health-related issue that requires treatment or modification, such as a therapeutic approach or a lifestyle change.¹⁴ URs are conducted to gain access to a wide range of therapy and disease-related topics, such as the influence of statins on cancer incidence.¹⁵ Another study utilized a UR to approach health-related issues related to everyday behaviors such as caffeine or coffee consumption and health outcomes.¹⁶

URs offer a practical, evidence-based avenue for healthcare decision-makers to comprehend a wide range of topics. Consequently, URs can be considered when a wide range of questions are raised, such as when examinations involving various interventions for particular diseases are necessary and when high-quality evidence is required quickly to formulate a new policy or guideline.¹³ During the coronavirus disease 2019 (COVID-19) pandemic, a substantial amount of evidence was published in a short period of time.¹⁷ In this situation, qualitative analysis is crucial for determining the validity of the evidence, which can be used to establish appropriate assessments. However, because of the unprecedented nature of the COVID-19 pandemic, policy makers were forced to swiftly announce COVID-19-related proposals or policies based on a flood of evidence. URs can be a very helpful methodological approach in these scenarios. In 2020, we conducted a UR with just 2 systematic reviews to examine the relationship between dyslipidemia and the severity of COVID-19.¹⁸ In an urgent catastrophic situation, this study represented an effort to swiftly produce the necessary evidence, and URs have since been shown to be a very useful tool in this process.

HOW TO PERFORM AN UR

Although various approaches have been described for performing URs, researchers still face issues related to the conduct of URs, and readers and reviewers also have questions about interpreting the results of URs and the expectations from URs. Since systematic reviews are the basic unit for data extraction in URs, researchers conducting URs frequently encounter methodological challenges for which there is no clear direction.¹⁹,²⁰ Therefore, despite the availability of much guidance for performing URs, these reviews are typically conducted on the basis of the researchers’ experience, resulting in substantial variations in the methodologies used and reported in published URs. Pollock et al.²¹ evaluated 52 guidance reports on URs produced by 19 research groups recognizing the methodological issues inherent in conducting URs. The term “umbrella review” was coined by the Joanna Briggs Institute (JBI) group,⁷ and it was used to refer to an overview of reviews with a similar premise by the Cochrane group.²² The Cochrane guideline research team reported a protocol based on the Priority Reporting Item for Overviews of Reviews, which became a consensus checklist for URs²³ and has been further updated in 2022.²⁴ The updated version is planned to be made available soon and will be used similarly to checklists such as Priority Reporting Items for Systematic Reviews and Meta-analyses,²⁵ providing a framework for journal editorial boards.

The Cochrane Collaboration and JBI have recently produced guidelines for conducting URs,⁵,²² which are the 2 most widely suggested and cited. Since all researchers have easy access to these evidence-based recommendations, it appears somewhat unnecessary to describe every step of URs in detail. Therefore, in this review, we would like to briefly discuss the key processes in performing URs based on these 2 guidelines.

1. Justification and design of the UR

To conduct a UR, a reason must be established. Topics that are covered by numerous systematic reviews are more likely to be controversial or interesting overall. Consequently, the need for conducting URs on these topics should be investigated. Like other forms of research, URs are undertaken to answer certain research questions, and the necessity of conducting URs for these topics should be explored.¹⁰ Prior to performing a UR, the researchers should clearly describe the type of reviews that will be included.⁷ Formalizing the research topics and scope of the review can facilitate subsequent methodological decision-making.¹³ Reviews that rely solely on theoretical studies and published opinions as the primary source of evidence should be excluded from URs and mentioned in the exclusion criteria in the a priori protocol. The Population, Intervention, Comparator, Outcome variables should be defined in the presentation of inclusion criteria for URs to evaluate the effectiveness of interventions. Similarly, the elements of Population, Phenomena of Interest, and Context are specified for URs that incorporates qualitative systematic reviews.⁷

2. Preparation of search strategies and literature search

URs, like systematic reviews, require an organized, transparent, and reproducible search strategy. The goal of a UR’s search strategy is to identify all study aggregates related to the review topic. The Cochrane Database of Systematic Reviews, the JBI Database of Systematic Reviews and Implementation Reports, the Database of Abstracts of Reviews of Effects, and the PROSPERO registry, as well as databases such as MEDLINE, Embase, and CINAHL, which are used when undertaking conventional systematic reviews, are also utilized for URs.⁷

3. Assessment of methodological quality

Methodological quality should be assessed for the systematic reviews included in URs. This approach is strongly encouraged in the methodological standards for URs and is critical for ensuring that the methodological quality of the systematic reviews contained in a UR is appropriately examined and incorporated into the results and conclusions. A Measurement Tool to Assess Systematic Reviews (AMSTAR) has been mainly used for assessing methodological quality. The AMSTAR tool has been recommended since 2007,²⁶ and an updated version, AMSTAR 2, was released in 2016.²⁷ Given the newly revised tools, the methodologies advocated in the published recommendations have evolved over time, and the changes in the tools utilized for methodological quality evaluation identified in this study are compatible with those of the current guidelines for performing URs.

4. Assessment of certainty in evidence

At present, there are no official standards for determining the certainty of evidence in performing URs.²⁸ The certainty of evidence acquired from URs is a crucial element supporting decisions on the course of treatment or policy since it shows the confidence in the conclusions drawn from numerous studies.²⁹ Therefore, explicit directions for the process of assessing certainty in evidence in UR are expected to be provided after various stages of debate. Grading of Recommendations, Assessment, Development and Evaluation (GRADE) is currently the most widely used method for determining the certainty of evidence.³⁰ GRADE was initially created to evaluate the degree of confidence in the evidence from primary studies included in systematic reviews.³⁰ As a result, several published URs without further analysis report the certainty of systematic reviews as initially reported in the previous studies. However, more recently, rigorous criteria have been applied to stratify the strength of the evidence using a number of statistical factors. These criteria are occasionally recommended as useful advice for performing URs.¹⁴

5. Summary of evidence

URs draw conclusions purely to facilitate the reader’s acceptance of a summary of the findings. The evidence is supposed to be summarized and expressed using graphs or figures in various ways.⁷,³¹ Researchers should make an effort to provide contextual elements of the outcomes and results presented in URs in order to provide the reader with a clear and comprehensive view. URs can have a significant impact on research, policy, and practice by allowing decision-makers to abstract evidence to broader scales when assessing the strengths and limitations of current knowledge. This is especially true when the research question’s scope is broad or when evidence is required rapidly to inform a new policy.¹³

LIMITATIONS OF URs

URs emerged from systematic reviews that followed well-established guidelines to ensure the integrity, validity, and reliability of outcomes. Thus, URs attempt to use explicit, reproducible, and systematic procedures to search, identify, and extract data from systematic reviews. However, because the basic unit of search is a systematic review, researchers performing URs frequently encounter novel methodological issues with no defined rules.²¹ Thus, the conduct of URs is mostly based on personal experience and trial and error, and published URs show notable variations in methodologies and reporting.¹⁹,²¹ Systematic reviews represent a synthesis of the available data, and the UR’s effectiveness may be constrained by the methods and reporting of systematic reports. Thus, guidance is required for cases in which data in the included systematic reviews may be missing or incorrectly reported, and on whether the researchers performing URs should depend exclusively on the conducted and published systematic reviews or refer back to the individual primary studies included in the systematic reviews.¹⁹ These aspects are currently not clear. To address these problems, we reviewed primary papers during URs and conducted an updated meta-analysis, including the primary studies that were recently published and were not included in currently published systematic reviews.³² Many studies reporting updated meta-analyses with URs have recently been published, and while this cannot be a comprehensive answer to the limitations of URs, it can be a partial solution.

URs may also show overlap across studies that were included.³³ However, unlike typical meta-analyses, URs do not integrate the findings of the retrieved meta-analyses and do not estimate combined effect sizes, thereby reducing the possibility of duplication. Additionally, as mentioned earlier, there are limitations related to the certainty of the evidence. Although GRADE provides a comprehensive framework for evaluating evidence, the majority of the essential information is not provided in the original study, which represents a restriction.²⁸ Reaserchers are not universally following the guidelines that have been published so far, and there remain issues that need to be solved, such as those that conflict with the guidelines or prevent their usage because of their omission when conducting URs. To ensure an accurate and rigorous assessment of evidence in URs, more sophisticated guidelines for the conduct of URs are required.

CONCLUSIONS

URs are clearly contributing to evidence-based medicine in light of the increasing volume of systematic reviews, and they can serve as an entrance point for clinicians and decision-makers in the healthcare field. More specific guidelines for performing URs are required to ensure rigorous methodology and thereby provide more reliable evidence through URs.