What Can We Expect from an Umbrella Review?

Dear Editor:

We read with great interest the article entitled “The Dietary Inflammatory Index and Human Health: An Umbrella Review of Meta-Analyses of Observational Studies” recently published in Advances in Nutrition by Marx et al. (1). It is a very stimulating article and, unlike many other umbrella reviews, of high methodological quality. Indeed, although the methodology for conducting an umbrella review was published >5 y ago (2), there is still great confusion among authors about how to perform an umbrella review, and among reviewers and readers about what to expect from an umbrella review and how to interpret its results. According to its definition, umbrella review is a methodological approach used to review systematic reviews and meta-analyses on a shared topic (2). Its purpose is to collect, summarize, and assess the quality and strength of the available evidence, providing an overall picture of the results for specific research questions or phenomena that look at different exposures and outcomes, or different exposures for the same outcome, or even 1 exposure for many outcomes (3). Authors generally tabulate the main characteristics and results of included systematic reviews and meta-analyses, assess their methodological quality, and evaluate the strength of the evidence that each included meta-analysis provides.

In our experience, we have found that during the peer review of umbrella reviews, some reviewers have questioned the risk of overlap between included studies. However, an umbrella review does not combine the results of retrieved meta-analyses, and does not estimate new pooled effect sizes, as is usually expected from meta-analyses. Because the results are not statistically pooled, there is no risk of overlap between studies.

There is also confusion about the differences between assessing the methodological quality of included studies and assessing the strength of evidence. As regards methodological quality, one of the most used validated tools is A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR-2) (4). It is based on 16 items, used to critically appraise systematic reviews that include randomized or nonrandomized studies of health care interventions, or both. The working group that developed AMSTAR-2 identified critical domains that are more likely than others to critically affect the validity of a review. However, the list provided is a suggestion, and appraisers may add or substitute critical domains if detailed in the manuscript. In this respect, it may happen that some reviewers request that meta-analyses with low quality be excluded from an umbrella review, as is usually advisable in sensitivity meta-analysis to improve the quality of the pooled results. However, in an umbrella review, all available systematic reviews and meta-analyses should be included so that their methodological quality can be analyzed, discussed, and possibly criticized. Studies cannot be included or excluded based on methodological quality; in fact, quality assessment is downstream of the selection process (5). Excluding meta-analyses a priori because of their low quality would result in loss of information.

Regarding the strength of evidence, several approaches have been proposed to assess the credibility of each association. Some are based on predefined credibility grading criteria that consider summary effect sizes, P values, sample size, number of the events of interest, heterogeneity, 95% prediction intervals, and tests of bias (e.g., small-study effects and excessive significance), as proposed by Ioannidis and colleagues (6). However, these criteria are categorized according to arbitrary cutoffs, therefore they can be combined with other assessment tools such as the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach (7). GRADE provides a comprehensive method for evidence evaluation, including all factors capable of influencing its quality and the overall balance between benefits and risks, patients’ values and preferences, and the appropriate use of resources. However, GRADE also suffers from some limitations, mainly because much of the required information often is not reported in the original studies (8). In this regard, pushing for high-quality peer review and requiring the submission of papers with validated checklists would help to improve the quality of published articles.

Umbrella reviews also have limitations. Firstly, as for systematic reviews, an umbrella review is dependent on the reporting of the included meta-analyses and does not account for potential omissions or overlap of original studies. For instance, meta-analyses of observational studies are affected by potential selection bias, or lost-to-follow-up bias (in the case of meta-analyses of prospective cohort studies), ultimately undermining the representativeness of the sample. Moreover, heterogeneity in exposure and outcome assessment in the original studies is reflected first in meta-analyses and then in umbrella reviews. Although that might seem to be an unsolvable issue, it can be addressed through a strict definition of inclusion/exclusion criteria for systematic reviews and meta-analyses. Clearly, encouraging the publication of only high-quality primary research is the only way to truly have confidence in the results, of both primary and secondary studies. Another limitation of an umbrella review is related to the fact that only systematic reviews and meta-analyses are included. As a result, the newest evidence from studies that have not yet been included in meta-analyses or reviews is not included in the assessment. Last but not least, umbrella reviews suffer from a limitation intrinsically related to the novelty of their approach. In fact, despite the availability of well-described and standardized procedures, many authors do not adhere to these guidelines, increasing the noise and uncertainty around the knowledge and interpretability of an umbrella review (9).

In conclusion, when approaching an umbrella review, readers cannot expect what is normally found in a meta-analysis, in terms of both results and limitations. The added value of umbrella reviews is that they offer an overview of systematic reviews and meta-analyses providing a comprehensive assessment of the quality and credibility of existing evidence. According to 1 article (10), >10 systematic reviews and meta-analyses are published daily, often with redundancy and poor quality. Therefore, the comprehensive and critical assessment provided by umbrella reviews can positively affect both clinical practice and public health.

Notes

The authors’ responsibilities were as follows—VG: wrote the first version of the manuscript; DN: contributed in collecting material and revised the first version; MD: critically read and revised the manuscript; and all authors: read and approved the final manuscript.

The authors reported no funding received for this work.

Author disclosures: The authors report no conflicts of interest.