Scoping review of COVID-19-related systematic reviews and meta-analyses: can we really have confidence in their results?

Abstract

Aim

The aim of this study was to systematically appraise the quality of a sample of COVID-19-related systematic reviews (SRs) and discuss internal validity threats affecting the COVID-19 body of evidence.

Design

We conducted a scoping review of the literature. SRs with or without meta-analysis (MA) that evaluated clinical data, outcomes or treatments for patients with COVID-19 were included.

Main outcome measures

We extracted quality characteristics guided by A Measurement Tool to Assess Systematic Reviews-2 to calculate a qualitative score. Complementary evaluation of the most prominent published limitations affecting the COVID-19 body of evidence was performed.

Results

A total of 63 SRs were included. The majority were judged as a critically low methodological quality. Most of the studies were not guided by a pre-established protocol (39, 62%). More than half (39, 62%) failed to address risk of bias when interpreting their results. A comprehensive literature search strategy was reported in most SRs (54, 86%). Appropriate use of statistical methods was evident in nearly all SRs with MAs (39, 95%). Only 16 (33%) studies recognised heterogeneity in the definition of severe COVID-19 as a limitation of the study, and 15 (24%) recognised repeated patient populations as a limitation.

Conclusion

The methodological and reporting quality of current COVID-19 SR is far from optimal. In addition, most of the current SRs fail to address relevant threats to their internal validity, including repeated patients and heterogeneity in the definition of severe COVID-19. Adherence to proper study design and peer-review practices must remain to mitigate current limitations.

Introduction

Since the first report of SARS-CoV-2 in December 2019, the betacoronavirus responsible for COVID-19, there has been an exponential increase in the published literature on the topic; more than 2400 articles on COVID-19 were published in a single day alone, 5 June 2020,1 2 and since December, 56 534 full-text articles related to COVID-19 have been documented on the WHO Global literature on coronavirus disease database.3

The deluge of manuscripts represents the largest explosion of scientific evidence in history, and there has been increasing concern that high publication volumes, including expedited peer-review processes and increased use of preprint servers, may be compromising the scientific quality of current research.4 5 Concerns are not limited to quality; duplicate and incomplete reporting of patient data has been recognised as a significant threat to the accuracy of subsequent prevalence and effect estimates.6–9 In addition, inconsistent clinical definitions, particularly for classifying severe COVID-19, make the synthesis of information a problematic task.7 9 10

The large volume and variable quality of published work on COVID-19 highlight an overwhelming need to organise and summarise findings so that the most current and accurate information can be easily accessed.11 Several groups, including our own, have conducted systematic reviews (SRs) with or without meta-analyses (MAs) to address this need.7 Using our experience and a scoping review of the literature, we will discuss the limitations of the current COVID-19-related SRs and provide suggestions for improving future research.

Methods

Search strategy and selection criteria

A search strategy was executed in MEDLINE via PubMed from 1 December 2019 until 17 May 2020. The search strategy was limited to peer-reviewed SRs with or without MAs published in English that evaluated clinical data, outcomes or treatments for patients with COVID-19. The search terms for PubMed were “coronavirus disease 2019,” “COVID-19” and “SARS-CoV-2.” This scoping review was not registered with PROSPERO.

Study selection

A title-and-abstract and a full-text screening phase was performed by experienced reviewers in an independent and duplicated manner. Each phase was prepiloted to ensure basic understanding of the selection criteria. Substantial agreement had to be achieved to perform each phase (kappa >0.70).

Data extraction and quality assessment

For each eligible study, two reviewers independently extracted the primary outcome, country(s) of the primary studies, journal, associated impact factor for 2019 (per Journal Citation Reports, https://jcr.clarivate.com/) and methodological quality indicators.

Methodological appraisal using AMSTAR-2

The included studies’ methodological quality was appraised independently and in duplicate by experienced reviewers using the critical domains of A Measurement Tool to Assess Systematic Reviews-2 (AMSTAR-2).12 A qualitative score of critically low, low, moderate or high quality was assigned to directly reflect the number of critical flaws present across each of the domains. A quantitative score was calculated by giving 1 point for ‘yes’ and 0.5 points for ‘partial yes’ and 0 points for ‘no’ for a total of 7 points for SRs with MAs and 5 points for SRs without MAs.12 13

Supplementary methodological appraisal

Complementary evaluation of internal validity threats to the COVID-19 body of evidence was performed based on several concerns with current COVID-19 reports. This evaluation aimed to ascertain the prevalence of SRs which included primary studies that repeated patient populations, provided clinical definitions for comorbidities (eg, hypertension was defined using specific blood pressure values) and were preprint. In addition, the articles were assessed for the presence of methods to manage the absence of a universal definition for severe COVID-19.6–8 10 Discrepancies between reviewers in the screening and data extraction phases were resolved by consensus. If consensus could not be reached, a third senior investigator was consulted (FHS, RRG or JPB).

Statistical analysis

Categorical data are summarised in frequencies and percentages, and numerical data in means and SD. Student’s t-test and Pearson’s χ² test were performed to seek an association between the methodological quality as a quantitative or qualitative score, respectively, and the inclusion of preprint primary studies, or single/multinational primary studies, or a clearly defined primary outcome. Statistical analysis was performed using SPSS V.25.0 for Mac (IBM).

Results

Our search strategy yielded a total of 105 studies, of which 63 met the inclusion criteria (figure 1, online supplemental appendix 1). The majority of the SRs included primary studies from more than one country (34, 56%) and 23 (37%) included data from a single country, China. The mean±SD for impact factor was 4.36±3.37 (range, 1.42–17.37; table 1) for SRs with MAs and 4.30±3.19 (range, 0.86–13.95; table 1) for SRs without MAs.

Figure 1.

Flow chart of the selection process.

Table 1.

Quantitative AMSTAR-2 score for systematic reviews with or without meta-analyses was not influenced by primary study characteristics or journal impact factor

Characteristics	SR with MA (n=41)		Characteristics	SR without MA (n=22)
	AMSTAR-2 score	P value		AMSTAR-2 score	P value
Average score	4.49±1.47	–	Average score	1.98±1.52	–
Single country (n=16)	4.46±1.47	0.95	Single country (n=7)	2.00±1.32	0.88
Multinational (n=21)	4.5±1.63		Multinational (n=13)	1.88±1.66
Primary outcome present (n=16)	4.65±1.74	0.57	Primary outcome present (n=7)	2.28±1.60	0.53
Primary outcome absent (n=25)	4.38±1.29		Primary outcome absent (n=15)	1.83±1.51
Pre-print studies included (n=11)	3.90±0.83	0.067	Pre-print studies included (n=3)	1.16±0.28	0.54
Pre-print studies excluded (n=13)	5.00±1.70		Pre-print studies excluded (n=13)	1.73±1.49
Journal impact factor vs. quantitative score (Spearman’s rho) (n=34)	−0.018	0.92	Journal impact factor vs. quantitative score (Spearman’s rho) (n=19)	0.152	0.53
Average impact factor (n=34)	4.36±3.37	–	Average impact factor (n=19)	4.30±3.19	–

Methodological appraisal using AMSTAR-2

The methodological quality was qualitatively judged as critically low in 27 (66%) and 16 (73%) of the included studies for SRs with MAs and SRs without MAs, respectively; only 6 (15%) and 2 (9%) were judged as high quality (table 2). The mean±SD AMSTAR-2 score for SRs with MAs was 4.49±1.47 (range, 1–7) and for SRs without MAs was 1.98±1.52 (range, 0–5) (table 1). For both SRs with and without MAs, the inclusion of multinational primary studies, preprint primary studies or a clearly defined primary outcome did not appear to influence the qualitative score (table 1).

Table 2.

Characteristics of included systematic reviews with and without meta-analyses

ID	Author	Journal	Journal impact factor	Qualitative score
Systematic reviews with meta-analysis
2	Sarma et al48	Journal of Medical Virology	2.021	Critically low
5	Di Mascio et al38	American Journal of Obstetrics & Gynecology MFM	–	High
7	Yang et al49	Journal of Infection	4.842	Low
15	Cao et al50	Journal of Medical Virology	2.021	Critically low
16	Wang et al51	Research	–	Critically low
17	Mantovani et al52	Liver International	5.175	Critically low
19	Kumar et al18	Diabetes & Metabolic Syndrome: Clinical Research & Reviews	–	Low
20	Parohan et al53	Hepatology Research	3.165	Low
21	Farsalinos et al39	Internal and Emergency Medicine	2.322	Critically low
22	Tong et al54	Otolaryngology– Head and Neck Surgery	2.341	Low
24	Huang et al14	Diabetes & Metabolic Syndrome: Clinical Research & Reviews	–	Critically low
25	Zheng et al55	Journal of Infection	4.842	Critically low
27	Hu et al19	Journal of Clinical Virology	2.777	Critically low
28	Chang et al15	Journal of the Formosan Medical Association	3.008	Critically low
30	Henry et al20	Clinical Chemistry and Laboratory Medicine	3.595	Critically low
32	Fu et al21	Journal of Infection	4.842	Critically low
33	Cheung et al22	Gastroenterology	17.373	Critically low
34	Lippi et al23	Polish Archives of Internal Medicine	3.007	Critically low
35	Wang et al56	Aging	4.831	Low
36	Emami et al57	Archives of Academic Emergency Medicine	–	Critically low
37	Zhao et al24	International Journal of Infectious Diseases	3.202	High
38	Zhu et al58	Family Medicine and Community Health	–	Low
39	Santoso et al59	American Journal of Emergency Medicine	1.911	Critically low
40	Li et al25	Progress in Cardiovascular Diseases	6.763	Critically low
41	Zhao et al26	Journal of Medical Virology	2.021	Moderate
42	Zhu et al40	Journal of Medical Virology	2.021	Low
43	Borges do Nascimento et al60	Journal of Clinical Medicine	3.303	High
44	Wang et al61	Diabetes Research and Clinical Practice	4.234	Critically low
45	Aggarwal et al41	Current Problems in Cardiology	2.966	High
46	Pranata et al27	Journal of the Renin-Angiotensin-Aldosterone System	1.417	Critically low
47	Pranata et al16	Journal of Stroke & Cerebrovascular Diseases	1.787	Critically low
48	Rodriguez-Morales et al62	Travel Medicine and Infectious Disease	4.589	Critically low
49	Yang et al63	International Journal of Infectious Diseases	3.202	Critically low
50	Li et al64	Clinical Research in Cardiology	5.268	Critically low
63	Alqahtani et al65	PLoS One	2.74	Critically low
65	Zhang et al17	Clinical Infectious Diseases	8.313	Moderate
66	Singh et al66	Diabetes & Metabolic Syndrome: Clinical Research & Reviews	–	Critically low
67	Mao et al28	Lancet Gastroenterology & Hepatology	14.789	High
68	Zhang et al67	Pharmacological Research	5.893	Critically low
71	Wang et al68	Clinics and Research in Hepatology and Gastroenterology	2.718	Critically low
73	Gao et al69	Journal of Infection	4.842	Critically low
Systematic reviews without meta-analysis
1	Yang et al70	Journal of Maternal-Fetal & Neonatal Medicine	1.737	Low
4	Zaigham et al71	Acta Obstetricia et Gynecologica Scandinavica	2.77	Critically low
6	Yousefifard et al72	Archives of Academic Emergency Medicine	–	Critically low
8	Ford et al73	Journal of the International AIDS Society	5.553	Critically low
9	Ludvigsson et al74	Acta Paediatrica	2.111	Critically low
11	Balla et al75	Journal of Clinical Medicine Research	–	Critically low
12	Moujaess et al76	Critical Reviews in Oncology/ Hematology	5.833	Critically low
13	AminJafari et al77	International Immunopharmacology	3.943	Critically low
14	Singh et al78	Diabetes & Metabolic Syndrome: Clinical Research & Reviews	–	Critically low
23	Minotti et al79	Journal of Infection	4.842	Critically low
26	Castagnoli et al80	JAMA Pediatrics	13.946	High
31	Lovato et al81	Ear, Nose & Throat Journal	0.859	Low
51	Cortegiani et al82	Journal of Critical Care	2.685	Critically low
52	Vardavas et al83	Tobacco Induced Diseases	1.434	Critically low
55	Rajendran et al84	Journal of Medical Virology	2.021	Critically low
56	Della Gatta et al85	American Journal of Obstetrics and Gynecology	6.502	Low
57	Elshafeey et al86	International Journal of Gynecology & Obstetrics	2.216	Critically low
58	Mehta et al87	Clinical Infectious Diseases	8.313	Critically low
61	Alzghari et al88	Journal of Clinical Virology	2.777	Critically low
64	Veronese et al89	Frontiers in Medicine	3.900	Critically low
70	Aiello et al90	Eye	2.455	Low
72	Valk et al91	Cochrane Database of Systematic Reviews	7.89	High

The complete performance of the SRs with and without MAs for the critical domains of AMSTAR-2 can be found in table 3. Across both groups of included studies (SRs with MAs and SRs without MAs), the most critical methodological flaws were lack of or inadequate pre-established study protocol (39, 62%) and discussion of risk of bias when interpreting the results (39, 62%), respectively. In addition, the majority of SRs without MAs suffered from deficient techniques for assessing the risk of bias for the included studies (15, 68%; table 3). The most prominent strength of SRs with MAs was the use of appropriate statistical methods to synthesise results (39, 95%). In addition, the use of a comprehensive literature search strategy was reported in most of the included studies (54, 86%).

Table 3.

Performance of systematic reviews with and without meta-analyses for the critical domains of AMSTAR-2

Critical domain question	n (%) of studies judged as ‘no’
Systematic review with meta-analysis
Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?	22 (54)
Did the review authors use a comprehensive literature search strategy?	4 (10)
Did the review authors provide a list of excluded studies and justify the exclusion?	3 (7)
Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?	21 (51)
If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?	2 (5)
Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review?	24 (59)
If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?	12 (29)
Systematic review without meta-analysis
Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?	17 (77)
Did the review authors use a comprehensive literature search strategy?	5 (23)
Did the review authors provide a list of excluded studies and justify the exclusion?	9 (41)
Did the review authors use a satisfactory technique for assessing the RoB in individual studies that were included in the review?	15 (68)
If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?	NA
Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review?	15 (68)
If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?	NA

Supplementary methodological appraisal

Of the 49 SRs that evaluated comorbidities, only 15 (30%) provided a clinical definition of included comorbidities. Severe COVID-19 was evaluated as an outcome in 48 SRs, although only 16 (33%) recognised heterogeneity in the definition of severe COVID-19 as a limitation of the study. Of these 16 studies, 4 stated that only one definition of severe COVID-19 was used.14–17 Almost all (15, 94%) included an MA; however, the vast majority (11, 73%) did not perform a sensitivity analysis for each definition of severe COVID-19 used in the primary studies.14 15 18–28 Only 15 (24%) of the included SRs recognised repeated patient populations as a limitation of their study, and the majority (11, 73%) of these SRs implemented a method to mitigate the risk of including repeated patients in their analysis. Finally, 15 (24%) SRs included preprint primary studies.

Discussion

Main findings

In this scoping review, we aimed to appraise the methodological quality of COVID-19 SRs with and without MAs using a validated SR appraisal tool, AMSTAR-2, and complementary criteria evaluating limitations pertinent to the current pandemic literature. Overall, the quality of included SRs was judged as critically low. A small number of studies recognised limitations affecting COVID-19-related primary literature, namely, the inclusion of primary studies that repeat patient populations and heterogeneity in the definition of severe COVID-19. Ultimately, the SRs evaluated in this study suffered from several major limitations and the reported effect estimates, and conclusions should be interpreted cautiously.

Methodological appraisal using AMSTAR-2

The majority of COVID-19-related SRs evaluated in this study suffered from significant limitations, with two-thirds of the included SRs with MAs and 7 of 10 SRs without MAs judged as a critically low methodological quality. The average quantitative score was 4.49 of 7 and 1.98 of 5 for SRs with and without MAs, respectively. Although AMSTAR-2 does not recommend the use of a quantitative score to evaluate SRs, previous studies have used both qualitative and quantitative scores.13 29 30 We did not find any correlation between qualitative score and the inclusion of multinational primary studies, preprint primary studies or a clearly defined primary outcome.

One of the most prominent flaws of both SRs with and without MAs was lack of or inadequate pre-established study protocol. Interestingly, previous studies evaluating the methodological quality of SRs across various disciplines also reported insufficient pre-established study protocols as a predominant limitation.29–31 Therefore, this limitation appears to impact SRs holistically. Brito et al reported that SRs including randomised controlled trials (RCTs) received a higher AMSTAR score. Inclusion of RCT primary studies was extremely limited in our body of evidence, likely due to the recent emergence of COVID-19. Future appraisals of the methodological quality of COVID-19-related SRs should explore associations between primary study design type and AMSTAR score. Ultimately, unambiguous eligibility criteria for included primary studies and a structure for quantitative and qualitative synthesis are critical components to an SR.32 SRs conducted without a prespecified protocol may be subject to selection bias.33

We also found that the majority of SRs with and without MAs failed to adequately discuss risk of bias when interpreting the results of the study. In the SR/MA recently published by the authors, the primary studies were of critical risk of bias, and therefore it is imperative to recognise this limitation to prevent any conclusions from being overstated.7 While retrospective studies are an initial source of information at the onset of the pandemic, failure to consider potential biases affecting retrospective studies, including, but not limited to, confounding, and collider bias creates methodological flaws.34–37 Strengths of the included studies were use of appropriate statistical methods for combining results, such as justifying the use of a random-effects or fixed-effects model and providing pre-established methods to investigate heterogeneity, as well as the use of a comprehensive literature search. Adherence to both these practices will minimise bias and help achieve more representative and reliable effect estimates.33

Supplementary methodological appraisal

Of the 63 SRs evaluated in our study, less than one-quarter either identified repeat patient populations as a limitation of their study or considered repeat patient populations to be a factor when selecting studies for MA. To avoid examination of repeat populations, some MAs excluded studies that appeared to have overlap.20 38–41 In some cases, authors of the primary publications may have been contacted to explain the overlaps. In one study,40 the authors assessed information from the facilities to which patients were admitted, as well as the ‘epidemiological week’ to avoid any overlap. However, the vast majority of publications examined did not recognise repeat patient populations as a limitation in performing an MA, whereas others recognised duplicate patient populations as a limitation of the study but did not specify if or how those repeat patient populations were addressed.

In our experience, to prevent analysis of repeated patient populations, we evaluated all included studies for overlap in both hospital and time frame of enrolment, selecting the study with the largest sample size when overlap was suspected.7 We encourage authors to implement similar methods to prevent the introduction of bias and inflation of results. In the case of interventional studies, reporting populations more than once increases the chance that the CI around the pooled effect size will be lower, altering interpretations of significance values.42

One of the larger concerns in performing SRs of COVID-19 is the lack of a universal definition of severe COVID-19.9 Of the 63 SRs examined, 48 evaluated severe COVID-19 as an outcome. However, half of the SRs that recognised heterogeneity in these definitions did not address the issue in their analysis. Of the SRs that addressed heterogeneity, many outlined severe disease according to specific organisations (such as the Chinese National Health Committee, WHO guidelines, American Thoracic Society Guidelines), whereas others constructed their own definition of severe COVID-19.20 These self-constructed definitions included presentation of acute respiratory distress syndrome, use of ventilation or life support, or admission to the intensive care unit (ICU). Inconsistencies in the definition of severe COVID-19 used by primary studies creates a source of bias known as information bias, where exposure and/or outcome are incorrectly determined.43 Future SRs should be cognisant of this type of bias and consider establishing selection criteria for primary studies and conducting sensitivity analyses to determine whether effect estimates vary.44 Ultimately, until a universal definition of severe COVID-19 is established, the clinical significance of severe COVID-19 as an outcome will remain unclear.

Implications for future research

SRs related to COVID-19 suffer from significant limitations as reflected by the poor methodological quality of the majority of SRs included in our study. In table 4, we provide a brief overview of additional, highly contested limitations affecting the COVID-19 body of evidence and possible solutions to mitigate their deleterious consequences. Future such analyses should establish methods to eliminate duplicate patient populations, including evaluation of overlapping hospitals and study duration, to prevent artificial inflation of outcomes.45 In addition, heterogeneity in the definition of severe COVID-19 prevents establishing reliable associations between risk factors and this outcome. Authors should consider using surrogate quantifiable definitions of severity, such as ICU admission, to mitigate this concern.46 Ultimately, addressing these limitations will help reduce bias and establish a more accurate estimation of risks associated with COVID-19 outcomes of interest.

Table 4.

Additional limitations affecting the COVID-19 body of evidence and suggestions to mitigate their deleterious consequences

Limitation	Definition and example	Recommendations to address limitation
Confounding	Definition: Extraneous factors influence the effect of interest and can lead to misrepresentation of a causal relationship.35 92	DAGs can be used to assess confounders in exposure–outcome relationships.36 93 94 Comprehensive reporting of patient characteristics.7
	Example: Age, diabetes mellitus and obesity are likely confounders for mortality risk in patients with diabetes mellitus and hypertension.7 These variables have not always been reported in the primary literature, limiting the rigour of sensitivity analyses in MAs.
Collider bias
	Definition: Two variables, exposure and outcome, influence a third variable, the collider, resulting in a false association between exposure and outcome.95	Weighted regression analysis to account for over-representation or under-representation of certain individuals.37 DAGs can be used to assess confounders in exposure–outcome relationships.36 93 94
	Example: Testing for COVID-19, as currently conducted, may over-represent patients who are symptomatic, hospitalised and have better access to healthcare. Artificial associations may be established when using non-generalisable sample populations.34 95
Publishing demands raise concern regarding scientific quality	Definition: The peer-review system is designed to mitigate the possibility that critically flawed articles reach circulation.	No compromise should be made to the rigour of the peer-review process.4
	Example: From 1 January to 1 June 2020, JAMA has received more than 11 000 articles, compared with 4000 in the same time frame the previous year.4 Lancet Global Health has seen a 185% increase in submissions for June 2020 compared with June 2019.5
Preprint servers
	Definition: Repositories for studies that have not yet undergone the formal peer-review process.	AI-powered literature reviews summarise key findings in articles to aid in identifying high-quality studies.1 11 Databases, such as the Novel Coronavirus Research Compendium, feature high-quality articles each with their own appraisal.11
	Examples: Since 14 July 2020, 25 publications have been retracted, 10 of which have been preprint.96

AI, artificial intelligence; DAGs, directed acyclic graphs; MAs, meta-analyses.

As new studies continue to be published, a living SR model (LSR) may serve as a valuable mechanism for representing the dynamic COVID-19 literature. LSRs are continuously updated, with new searches conducted at pre-established time frames to synthesise the most up-to-date information. LSRs are justified for research questions deemed important to clinical decision-making and in the setting of rapidly evolving or emerging health issues or disease.47 Therefore, authors should consider implementing this model to establish the most current risks and clinical guidance.

Ultimately, the COVID-19 pandemic has resulted in rapid collaborations between academia, government and industry, in some cases at a multinational level, to produce an astronomical amount of data on virtually every aspect of SARS-CoV-2. Although rapid dissemination of findings essential to human health is invaluable, long-standing practices of proper study design and peer review cannot be compromised if we are to establish optimal public health policies.

Strengths and Limitations

Limitations of our study include only searching in MEDLINE. It is likely that additional SRs that met our inclusion criteria were missed, and therefore our conclusions may not be generalisable to COVID-19-related SRs. However, the main intention of this scoping review was to obtain a representative sample of the total SRs available in the literature and not to provide a comprehensive overview and appraisal of the totality. Second, our appraisal was limited to the critical domains of AMSTAR-2, and the additional methodological qualities assessed through the non-critical domains of AMSTAR-2 were not evaluated in this study. Our study is strengthened by inclusion of a secondary evaluation of limitations pertinent to the COVID-19 body of evidence, inclusion of repeated patient populations and heterogeneity in the definition of severe COVID-19 and by a systematic approach to the screening and data extraction of the studies included.

Conclusion

Current SRs suffer from important methodological limitations according to our systematic evaluation using the AMSTAR-2 critical domains and additional concerns pertinent to the COVID-19 current literature. The methodological flaws place these articles at high risk of bias that, if existent, could influence their results and lead to misleading conclusions. Therefore, the findings of the majority of the studies should be interpreted with caution. We encourage future SRs to take into consideration these particularities in the COVID-19 literature to obtain more reliable results and lead to a better understanding of the current pandemic.

Main messages.

• The quality of COVID-19-related systematic reviews (SRs) included in our study was judged as critically low, and only a small number of studies recognised limitations affecting COVID-19 related primary literature.

• The most prominent methodological flaws of the included SRs were lack of or inadequate pre-established study protocol and discussion of risk of bias.

• The majority of included SRs used a comprehensive literature search strategy and appropriate statistical methods to synthesise results (when a meta-analysis was performed).

Current research questions.

• What limitations are affecting COVID-19-related systematic reviews?

• Are COVID-19-related SRs accounting for current limitations affecting the COVID-19 literature?

What is already known about this subject?

• The exponential increase in COVID-19-related literature has raised concerns about the quality of current research.

• High publication demand has overwhelmed publishers, leading to increased use of preprint servers and in some cases expedited peer-review processes.

• Limitations pertinent to the COVID-19 body of evidence, including a lack of a universal definition for severe COVID-19, may introduce biases that are not currently accounted for.