ABSTRACT
BACKGROUND:
Numerous systematic reviews on coronavirus disease-19 (COVID-19) treatment have been developed to provide syntheses of the large volume of primary studies. However, the methodological quality of most of these reviews is questionable and the results provided may therefore present bias.
OBJECTIVE:
To investigate how many systematic reviews on the therapeutic or preventive options for COVID-19 assessed the certainty of the evidence through the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.
METHODS:
We conducted a sensitive search in MEDLINE (via PubMed) and included all systematic reviews that assessed any intervention for COVID-19. The systematic reviews included were examined to identify any planned and/or actual assessment using the GRADE approach (or absence thereof) regarding the certainty of the evidence.
RESULTS:
We included 177 systematic reviews and found that only 37 (21%; 37/177) assessed and reported the certainty of the evidence using the GRADE approach. This number reduced to 27 (16.2%; 27/167) when Cochrane reviews (n = 10), in which an evaluation using GRADE is mandatory, were excluded.
CONCLUSION:
Most of the systematic reviews on interventions relating to COVID-19 omitted assessment of the certainty of the evidence. This is a critical methodological omission that must not be overlooked in further research, so as to improve the impact and usefulness of syntheses relating to COVID-19.
KEY WORDS (MeSH terms): COVID-19, Systematic review [publication type], Evidence-based practice, GRADE approach, Evidence-based medicine, Public health
AUTHORS’ KEY WORDS: Methodology, Certainty of evidence, Strength of recommendations, Evidence-based healthcare
INTRODUCTION
Since the beginning of the coronavirus disease 2019 (COVID-19) pandemic, large numbers of studies have been published in an attempt to find an effective treatment for this disease. Consequently, many systematic reviews have been developed on this topic, to provide syntheses of the large volume of primary studies. Healthcare professionals and policymakers commonly use systematic reviews to formulate recommendations and make practical decisions.1
However, the methodological quality of most of these systematic reviews is questionable. Hence, the results provided through these reviews may present bias.
Assessing the certainty of the evidence is an indispensable step in a systematic review. This is especially true within the current context, in which information is often misleading yet has been widely disseminated, both by scientific journals and by the traditional media. Thus, efforts need to be made by the authors of syntheses of the evidence on a given topic, to ensure that the degree of certainty that can be placed on the estimates of effect and clinical recommendations can be established.1,2
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) is a transparent approach to rating the certainty of the body of evidence in systematic reviews and other forms of synthesis, as a guide to making decisions.3 This approach should be beneficial during the COVID-19 pandemic. It is entirely possible to perform GRADE assessments, even (or even more so) within an emergency context.1,4
OBJECTIVE
We carried out a critical appraisal study with the aim of investigating how many systematic reviews that have been published in relation to therapeutic or preventive options for COVID-19 made assessments of the certainty of the evidence through the GRADE approach.
METHODS
We conducted a sensitive search in MEDLINE (via PubMed) on January 20, 2021, using the MeSH term “Coronavirus” and its synonyms combined with the PubMed clinical queries filter for systematic reviews (Annex 1). Two authors independently screened all titles and abstracts through the Rayyan platform,5 in order to include systematic reviews that assessed any intervention for COVID-19.
RESULTS
The systematic reviews included were analyzed in full text, to identify whether there was any planned and/or actual assessment of the certainty of the evidence using the GRADE approach (or absence thereof). The search strategy found 1,075 references, and 177 fulfilled the inclusion criteria. Of these, only 37 reviews (21%; 37/177) assessed the certainty of the evidence using the GRADE approach. This number reduced to 27 (16.2%; 27/167) when Cochrane reviews (n = 10), in which an evaluation using GRADE is mandatory, were excluded.
CONCLUSION
This result highlights the fact that most of the systematic reviews on interventions conducted in relation to COVID-19 omitted assessment of the certainty of the evidence. This is a critical methodological omission that must not be overlooked in further research, so as to improve the impact and usefulness of syntheses relating to COVID-19.
Annex 1.
Search strategy.
| Database | Search strategy |
|---|---|
| #1 “Coronavirus”[Mesh] OR “Covid-19” OR (COVID) OR (Coronavirus) OR (SARS-CoV-2) OR (Coronaviruses) OR (Deltacoronavirus) OR (Deltacoronaviruses) OR “Munia coronavirus HKU13” OR (Coronavirus HKU15) OR (Coronavirus, Rabbit) OR (Rabbit Coronavirus) OR (Coronaviruses, Rabbit) OR (Rabbit Coronaviruses) OR “Bulbul coronavirus HKU11” OR “Thrush coronavirus HKU12” | |
| MEDLINE (via PubMed) |
#2 (((systematic review[ti] OR systematic literature review[ti] OR systematic scoping review[ti] OR systematic narrative review[ti] OR systematic qualitative review[ti] OR systematic evidence review[ti] OR systematic quantitative review[ti] OR systematic meta-review[ti] OR systematic critical review[ti] OR systematic mixed studies review[ti] OR systematic mapping review[ti] OR systematic cochrane review[ti] OR systematic search and review[ti] OR systematic integrative review[ti]) NOT comment[pt] NOT (protocol[ti] OR protocols[ti])) NOT MEDLINE [subset]) OR (Cochrane Database Syst Rev[ta] AND review[pt]) OR systematic review[pt] #3 #1 AND #2 |
Footnotes
Centro Universitário São Camilo, São Paulo (SP), Brazil
Sources of funding: This research did not receive any specific grant from any funding agency in the public, commercial or not-for-profit sectors
REFERENCES
- 1.Schünemann HJ, Santesso N, Vist GE, et al. Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol. 2020;127:202–207. doi: 10.1016/j.jclinepi.2020.05.030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kumar A, Taggarsi M. GRADEing quality of evidence and its importance in evidence-based practice. BMJ Evid Based Med. 2020. bmjebm-2020-111525. [DOI] [PubMed]
- 3.Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–394. doi: 10.1016/j.jclinepi.2010.04.026. [DOI] [PubMed] [Google Scholar]
- 4.Munn Z, Twaddle S, Service D, et al. Developing guidelines before, during, and after the COVID-19 pandemic. Ann Intern Med. 2020;173(12):1012–1014. doi: 10.7326/M20-4907. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210. doi: 10.1186/s13643-016-0384-4. [DOI] [PMC free article] [PubMed] [Google Scholar]