Reporting and methodological quality of COVID-19 systematic reviews needs to be improved: an evidence mapping

Abstract

Objectives

To assess the reporting and methodological quality of COVID-19 systematic reviews, and to analyze trends and gaps in the quality, clinical topics, author countries, and populations of the reviews using an evidence mapping approach.

Study Design and Setting

A structured search for systematic reviews concerning COVID-19 was performed using PubMed, Embase, Cochrane Library, Campbell Library, Web of Science, CBM, WanFang Data, CNKI, and CQVIP from inception until June 2020. The quality of each review was assessed using the Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) checklist and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist.

Results

In total, 243 systematic reviews met the inclusion criteria, over 50% of which (128, 52.7%) were from 14 developing countries, with China contributing the most reviews (76, 31.3%). In terms of methodological quality of the studies, 30 (12.3%) were of moderate quality, 63 (25.9%) were of low quality, and 150 (61.7%) were of critically low quality. In terms of reporting quality, the median (interquartile range) PRISMA score was 14 (10–18). Regarding the topics of the reviews, 24 (9.9%) focused on the prevalence of COVID-19, 69 (28.4%) focused on the clinical manifestations, 30 (12.3%) focused on etiology, 43 (17.7%) focused on diagnosis, 65 (26.7%) focused on treatment, 104 (42.8%) focused on prognosis, and 25 (10.3%) focused on prevention. These studies mainly focused on general patients with COVID-19 (161, 66.3%), followed by children (22, 9.1%) and pregnant patients (18, 7.4%).

Conclusion

This study systematically evaluated the methodological and reporting quality of systematic reviews of COVID-19, summarizing and analyzing trends in their clinical topics, author countries, and study populations.

What is new?

Key findings

• •Overall, the reporting and methodological quality of the included SRs were low, especially the methodological quality.
• •The methodological quality of almost all SRs related to prevalence, etiology, prognosis, prevention, and special populations (such as children and pregnant women) were assessed as being low or critically low quality. The reporting quality of SRs related to treatment, prevention, and special populations was also relatively low.
• •Several evidence gaps were identified and a trend analysis of the clinical topics, countries, and study populations was performed.

What this adds to what is known?

• •This is the first study to assess the reporting and methodological quality of COVID-19 systematic reviews.

What is the implication and what should change now?

• •Considering the significance of systematic reviews of COVID-19, the reporting and methodological quality of these studies should be strengthened, especially the methodological quality. It is also necessary to discuss whether conventional quality assessment tools for systematic reviews are suitable during a public health emergency.

1. Introduction

A novel coronavirus was reported in Wuhan, China, for the first time on December 31, 2019. This virus was named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by the World Health Organization (WHO). On March 11, 2020, the WHO characterized the coronavirus disease 2019 (COVID-19) outbreak as a pandemic [1]. To rapidly increase our understanding of this new virus, scientists all over the world are making unprecedented research efforts. Because the outbreak of COVID-19, many relevant studies have been registered and published each day around the world [2,3]. However, due to challenges in systematically evaluating and summarizing large amounts of data, the findings cannot be actively used in practice. Furthermore, if many similar studies are being carried out, this may waste academic resources [3].

For decision-making in healthcare, systematic reviews (SRs) hold a unique place [4]. SRs serve as the basis for developing practice guidelines and provide information on gaps in knowledge, thus informing future research efforts [5]. The rigor and reliability of SRs are largely related to their methodology and reporting quality [5,6]. Therefore, in the context of COVID-19, it is necessary to conduct systematic analysis and quality assessment, as well as a tentative trend analysis, of SRs to help readers quickly and comprehensively understand this epidemic and to provide references for prevention and control of similar viruses in the future. Evidence mapping (EM) is a comprehensive evidence-based research method that systematically and rapidly collects, evaluates, organizes, and presents existing evidence to clarify research status and address gaps, thereby promoting scientific research and decision-making [7,8]. Compared with SRs [9], [10], [11], [12], [13] or bibliometric analysis [14], EM can more intuitively and deeply present and organize evidence. Especially in the face of urgent public health concerns, EM can provide a meaningful reference for quick decision-making, avoiding wastage of academic resources [3,7,8].

Several preliminary reviews of evidence for COVID-19 have been performed. For example, one study discussed current evidence regarding the pathophysiology, transmission, diagnosis, and management of COVID-19 [15]. Another study evaluated the methodological quality of 17 SRs at the beginning of the COVID-19 outbreak [16]. Nevertheless, a systematic EM study that presents, assesses, and analyzes the existing SRs of COVID-19 is lacking. Thus, the present study aimed to assess the reporting and methodological quality of COVID-19 SRs, and to analyze trends and gaps in the quality, clinical topics, author countries, and populations of the reviews using an EM approach.

2. Methods

2.1. Literature search

Nine electronic databases (PubMed, Embase, Cochrane Library, Campbell Library, Web of Science, CBM, WanFang Data, CNKI, and CQVIP) were searched from inception to June 15, 2020. The major search terms and strategies (Appendix Table S1) were as follows: ("Meta-Analysis" [Mesh] OR "Meta-Analysis as Topic" [Mesh] OR "Systematic Review" [Mesh] OR "Systematic Reviews as Topic" [Mesh]) OR systematic* OR “meta-analysis” or metaanalysis OR “meta analysis” OR “meta analyses” OR “meta-analysis” OR “meta-analyses” OR metaanalysis OR metanalysis OR metaanalyse OR metanalyses) AND (coronavirus* OR coronovirus* OR coronavirinae* OR "2019‐nCoV" OR 2019nCoV OR “2019‐CoV” OR nCoV2019 OR "nCoV‐2019" OR "COVID‐19" OR COVID19 OR "CORVID‐19" OR CORVID19 OR "WN‐CoV" OR WNCoV OR "HCoV‐19" OR HCoV19 OR CoV OR "2019 novel*" OR Ncov OR "n‐cov" OR "SARS‐CoV‐2" OR "SARSCoV‐2" OR "SARSCoV2" OR "SARS‐CoV2" OR SARSCov19 OR "SARS‐Cov19" OR "SARSCov‐19" OR "SARS‐Cov‐19").

2.2. Inclusion and exclusion criteria

The criteria were in line with the PRISMA-P protocol, which encompasses articles that specifically state the methods used to identify studies (i.e., a search strategy), strategies for study selection (e.g., eligibility criteria and selection process), and explicitly detailed methods of synthesis to screen related references [5]. All types of SRs (qualitative SRs, quantitative SRs, etc.) were eligible for inclusion. When several SRs addressing one problem by the same team were identified, we considered the most recent publication. The following were excluded: (a) duplicate reports; (b) studies with insufficient information (e.g., conference proceedings, abstracts, letters, and comments); and (c) studies not published in peer-reviewed journals (e.g., studies only appearing on medRxiv or a similar preprint server).

2.3. Study selection and data extraction

Screening and data extraction were performed independently by two reviewers. When the opinions of the two reviewers differed, differences were resolved through consultation with a third reviewer. EndNote X9 software (Thomson Corporation; Stamford, CT) was used to identify and to reject duplicates. Following extraction, the two reviewers screened the titles and abstracts of the included studies. If both reviewers excluded a study, it was removed from further review. If at least one reviewer included a study or if there was insufficient information to make a decision based on the title and abstract, the full article was obtained for review. The full text of the article was then reviewed by the two reviewers to determine its suitability for inclusion. Using a predesigned table, we then extracted the following data: year of publication; the name and country of the first author; the publishing journal and its impact factor; and full date of submission and publication. We also extracted details concerning the clinical topic [15], the number of original studies included in the SR, and the study population.

2.4. Quality assessment

The Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) [6] was used to evaluate the methodological quality of the included SRs. AMSTAR 2 consists of 16 items. Each item was evaluated using one of three options: “Yes,” “Partial Yes,” or “No.” The assessment process was conducted online (https://amstar.ca/Amstar_Checklist.php) to automatically generate the overall quality assessment results (“Critically low,” “Low,” “Moderate,” or “High”). The 27-item PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) [5] protocol was used to evaluate the reporting quality of reviews, resulting in a maximum score of 27 points (1 point for each of the 27 items). Extensions of PRISMA were used for some special types of SRs (e.g., diagnostic SRs were assessed using “PRISMA-DTA”). Two reviewers independently performed the quality assessments, and any disagreements were resolved via consensus and discussion with a third reviewer.

2.5. Data synthesis and analysis

Currently, there is a lack of reporting guidelines or methodological guidance with regard to EM. Therefore, we conducted this study using the methodology of Global Evidence Mapping [17], Campbell evidence, and gap maps [18], and our previous findings [19] concerning EM and the evidence and gap map methodology, and made the necessary expansion on this basis [20,21]. All authors have fully discussed the extension of each methodology and the construction of the framework of this article. Excel 2019 (Microsoft Corporation, Redmond, WA, USA) was used to extract, manage, and analyze the data. Tables, line charts, bar charts, and geographical maps were created to display the results. In addition, a bubble plot was designed to display information in four dimensions as follows [22,23]: (a) each bubble represents one SR and different colors represent various research populations; (b) the bubble size represents the number of original studies included in the SRs; (c) the reporting quality is represented on the X-axis; and (d) the methodological quality is represented on the Y-axis. We also conducted a narrative synthesis to expand upon the mapping to provide more details about the included studies. These details include prevalence, clinical manifestations, etiology, prevention, diagnosis, treatment, prognosis, evidence gaps, author country, and study population.

3. Results

3.1. Study selection

A flow chart of the literature selection process is displayed in Fig. 1 . A total of 1,349 relevant records were initially included. Of these, 481 were excluded as duplicates. Titles and abstracts were screened for the remaining 868 studies, of which 460 were deemed to be unsuitable for inclusion. The full texts of the remaining 408 articles were screened for further information, and another 165 studies were excluded (Appendix Table S2). Ultimately, 243 SRs met our inclusion criteria and were included in the analysis.

Fig. 1.

Flow chart of the literature screening process and results.

3.2. Study characteristics

The general characteristics of the included studies are shown in Table 1 and Appendix Table S3. A total of 243 studies were included in this mapping, more than 90% of which were written in English. Of the journals that published the studies, the overall impact factor was relatively high: 43.2% of journals had an impact factor ≥3. The average publication cycle was 28 days, which is helpful for rapid dissemination of research. A total of 33 countries distributed on 5 continents contributed to these 243 studies, mainly countries located in Asia (127, 52.3%) and Europe (63, 25.9%). Among these 33 countries, 19 developed countries published 115 studies (47.3%), with the USA contributing the most studies (31, 12.8%), and 14 developing countries published 128 studies (52.7%), with China contributing the most studies (76, 31.3%). It is notable that China, USA, and Italy, the top three countries in terms of publication, published 131 articles, accounting for 53.9% of the included SRs.

Table 1.

Essential characteristics of the included studies

Category	Characteristic	Number	Percentage n = 243
Language of publication	English	223	91.8
	Others (including Chinese)	20	8.2
Journal impact factor	≤3.0	88	36.2
	3.1–6.0	76	31.3
	6.1–9.0	18	7.4
	>9.0	11	4.5
	None	50	20.6
First author's country	China	76	31.3
Developing countries	Iran	15	6.2
	India	14	5.8
	Indonesia	7	2.9
	Brazil	7	2.9
	Others	9	3.7
Developed countries	USA	31	12.8
	Italy	24	9.9
	UK	22	9.1
	Canada	7	2.9
	Netherlands	4	1.6
	Singapore	4	1.6
	Australia	4	1.6
	Greece	3	1.2
	Sweden	3	1.2
	Korea	3	1.2
	Switzerland	2	0.8
	Others	8	3.3
Clinical topic	Prevalence	24	9.9
	Clinical manifestation	69	28.4
	Etiology	30	12.3
	Prevention	25	10.3
	Diagnosis	43	17.7
	Treatment	65	26.7
	Prognosis	104	42.8
Population	General patients with COVID-19	161	66.3
Infected	Children patients with COVID-19	22	9.1
	Pregnant patients with COVID-19	18	7.4
	COVID-19 patients with cerebrovascular and cardiovascular diseases	10	4.1
	COVID-19 patients with lung diseases	2	0.8
	COVID-19 patients with diabetic	2	0.8
	Healthcare workers with COVID-19	1	0.4
	COVID-19 patients with cancer	1	0.4
	COVID-19 patients with immunosuppression	1	0.4
	COVID-19 patients with liver and kidney diseases	1	0.4
Uninfected	Healthy people	17	6.7
Not provided	NP	7	2.9

*NP, not provided.

Before a publication date of May 2020, most studies were published by China. Several developed countries, such as the USA and UK, rapidly increased their publication after May 2020. The studies explored seven clinical topics related to COVID-19, including prevalence, clinical manifestation, etiology, prevention, diagnosis, treatment, and prognosis. The most common topic among the studies was prognosis (104, 42.8%). These studies mainly focused on general patients with COVID-19 (161, 66.3%), followed by children (22, 9.1%) and pregnant patients (18, 7.4%). Based on publication date, research on pregnant and children has grown rapidly since May (Fig. 2 ).

Fig. 2.

Trends in publishing countries (A) and study populations (B) of COVID-19 SRs. SRs, systematic reviews.

4. Quality assessment

4.1. Methodological quality

The methodological quality of the 243 included studies was assessed using the AMSTAR 2 checklist (see Appendix Table S5 for details). As shown in Fig. 3 , among these studies, 0 was of high quality, 30 (12.3%) were of moderate quality, 63 (25.9%) were of low quality, and 150 (61.7%) were of critically low quality. Of the items on the checklist, only 4 items (PICO: populations, interventions, comparisons, and outcomes; duplicate study selection; duplicate data extraction; conflict of interest) were fully reported by more than half of the included studies, and over 90% of SRs only fully reported item 1 (PICO: populations, interventions, comparisons, and outcomes) and item 16 (conflict of interest). Few studies reported the checklist items in full. In particular, items 2, 4, 7, and 10 (protocol and registration, comprehensive literature search strategy, listing of excluded studies, and funding reported for individual studies) were fully reported by less than 5% of the included studies. Furthermore, the methodological quality of the studies was found to have changed very little over time, remaining quite low.

Fig. 3.

Methodological quality of COVID-19 SRs. A represents how the methodological quality of COVID-19 SRs changes with the date of publication. B represents adherence of individual items of COVID-19 SRs assessed by the AMSTAR-2 checklist. SRs, systematic reviews; AMSTAR-2, Assessment of Multiple Systematic Reviews-2.

4.2. Reporting quality

The reporting quality of the 243 studies, including five diagnostic SRs, was evaluated using the PRISMA checklist (see Appendix Tables S6–S7 for details). As shown in Fig. 4 , of the 243 studies, the median (range) PRISMA score was 14 (10–18). Fifty-six studies (23.0%) scored below 10, whereas only 31 studies (12.8%) scored above 20. Over half of the studies fully reported the 13 items on the checklist, and most SRs fully reported item 3 (the rationale for the review) and item 4 (PICOS: populations, interventions, comparisons, outcomes, and study design). Most items were not adequately reported—especially item 5 (protocol and registration), which was fully reported by only 29 studies (11.9%).

Fig. 4.

Reporting quality of COVID-19 SRs. A represents the PRISMA scores of COVID-19 SRs. B represents adherence of individual items of COVID-19 SRs assessed by the PRISMA checklist. SRs, systematic reviews; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

4.3. Mapping

As shown in Fig. 5 , bubble plots were used to visualize the SRs in terms of their research populations, number of included studies, methodological quality, and reporting quality. We also conducted a narrative synthesis to expand upon this mapping and provide more details about the included studies (Appendix Table S4).

Fig. 5.

Mapping of COVID-19 SRs. Each bubble represents one SR and different colors represent various research populations. The bubble size represents the number of original studies included in the SRs. The reporting quality is represented on the X-axis, whereas the methodological quality is represented on the Y-axis. SR, systematic review. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

4.4. Prevalence

As shown in Fig. 5, 24 SRs (9.9%) focused on the prevalence of COVID-19. Regarding methodological quality, 17 (70.8%) SRs were assessed as critically low quality, 6 (25.0%) SRs were assessed as low quality, and 1 (4.2%) SR was assessed as moderate quality. For reporting quality, the median (range) PRISMA score was 12 (10.8–14). Five studies (20.8%) scored below 10, whereas only 2 studies (8.3%) scored above 20. Of the SRs focused on prevalence, most (11, 45.8%) focused on children, 8 (33.3%) SRs focused on general patients with COVID-19, and 1 (4.2%) SR focused on pregnant patients.

4.5. Clinical manifestations

As shown in Fig. 5, 69 (28.4%) of the included SRs focused on clinical manifestations of COVID-19. Regarding methodological quality, 44 (63.8%) SRs were assessed as critically low, 14 (20.3%) SRs were assessed as low, and 11 (15.9%) SRs were assessed as moderate. For reporting quality, the median (range) PRISMA score was 13 (8–18). Twenty-two studies (31.9%) scored below 10, whereas only 12 studies (17.4%) scored above 20. Of the SRs focused on clinical manifestations, 47 (68.1%) focused on general patients with COVID-19, 13 SRs (18.8%) focused on children, 6 SRs (8.7%) focused on pregnant patients, and 1 SR (1.4%) focused on healthcare workers with COVID-19.

4.6. Etiology

As shown in Fig. 5, 30 (12.3%) of the included studies focused on the etiology of COVID-19. Regarding methodological quality, 21 (70.0%) SRs were assessed as critically low, 8 (26.7%) SRs were assessed as low, and 1 (3.3%) SR was assessed as moderate. For reporting quality, the median (range) PRISMA score was 12 (10.3–13.8). Seven studies (23.3%) scored below 10 and no study scored above 20. Of the SRs focused on etiology, 12 SRs (40.0%) focused on pregnant patients, 11 (36.7%) SRs focused on general patients with COVID-19, and 6 SRs (20.0%) focused on children.

4.7. Diagnosis

As shown in Fig. 5, 43 (17.7%) of the included studies focused on the diagnosis of COVID-19. Regarding methodological quality, 25 (58.1%) SRs were assessed as critically low, 10 (23.3%) SRs were assessed as low, and 8 (18.6%) SRs were assessed as moderate. For reporting quality, the median (range) PRISMA score was 15 (11–18). Eighteen studies (41.9%) scored below 10, whereas only 7 studies (16.3%) scored above 20. Among the SRs focused on diagnosis, 28 (65.1%) SRs focused on general patients with COVID-19, 10 SRs (23.3%) focused on children, and 2 SRs (4.7%) focused on pregnant patients.

4.8. Treatment

As shown in Fig. 5, 65 (26.7%) of the included studies focused on treatment of COVID-19. Regarding methodological quality, 40 (61.5%) SRs were assessed as critically low, 16 (24.6%) SRs were assessed as low, and 9 (13.8%) SRs were assessed as moderate. For reporting quality, the median (range) PRISMA score was 11 (9–18). Twenty-one studies (32.3%) scored below 10, whereas only 4 studies (6.2%) scored above 20. Among the SRs focused on treatment, 54 (83.1%) SRs focused on general patients with COVID-19, and 8 (12.3%) SRs focused on children.

4.9. Prognosis

As shown in Fig. 5, 104 (42.8%) of the included studies focused on the prognosis of COVID-19. Regarding methodological quality, 65 (62.5%) SRs were assessed as critically low, 29 (27.9%) SRs were assessed as low, and 10 (9.6%) SRs were assessed as moderate. For reporting quality, the median (range) PRISMA score was 15 (12–18.3). Eleven studies (10.6%) scored below 10, whereas 13 studies (12.5%) scored above 20. Among the SRs focused on prognosis, 60 (57.7%) SRs focused on general patients with COVID-19, 14 (13.5%) focused on pregnant patients, 11 (10.6%) SRs focused on children, 10 (9.6%) SRs focused on COVID-19 patients with underlying cardiovascular comorbidities, 2 (1.9%) SRs focused on COVID-19 patients with lung diseases, 2 (1.9%) SRs focused on COVID-19 patients with diabetes, 1 (1.0%) SR focused on COVID-19 patients with cancer, 1 (1.0%) SR focused on COVID-19 patients with immunosuppression, and 1 (1.0%) SR focused on COVID-19 patients with liver and kidney diseases.

4.10. Prevention

As shown in Fig. 5, 25 (10.3%) of the included studies focused on the prevention of COVID-19. Regarding methodological quality, 17 (68.0%) SRs were assessed as critically low, 6 (24.0%) SRs were assessed as low, and 2 (8.0%) SRs were assessed as moderate. For reporting quality, the median (range) PRISMA score was 11 (7–17). Eleven studies (44.0%) scored below 10, whereas only 1 study (4.0%) scored above 20. Among the SRs focused on prevention, 17 (68.0%) SRs focused on healthy people, 4 (16.0%) SRs focused on general patients with COVID-19, 1 (4.0%) SR focused on children, and 1 (4.0%) SR focused on pregnant patients.

5. Discussion

In this study concerning COVID-19 SRs, we systematically searched for relevant published studies before June 2020. A total of 243 studies were included in this mapping, most of which were written in English or Chinese. Most published studies were from China, followed by the United States and Italy. The impact factors of the journals that published these studies were generally high, which is a good method for the rapid spread of information. The included studies comprehensively discussed many clinical topics related to COVID-19, including prevalence, clinical manifestations, etiology, prevention, diagnosis, treatment, and prognosis. Most studies focused on clinical manifestations, treatment, and prognosis. For some special populations (excluding general patients with COVID-19 and healthy people)— especially patients with comorbidities, children, and pregnant women—prognosis was one of the main research concerns.

It is worth noting that developing countries, represented by China, Iran, and India, contributed many SRs. In particular, China contributed more than one-third of the evidence. In early stage of COVID-19 research (i.e., February and March 2020), more than 80% of SRs were published by developing countries. Developed countries like the USA, which have technical superiority and greater scientific research strength, contributed much less (especially early in the outbreak), which may be related to the fact that the outbreak was first detected in China and other countries. As time passed, more and more studies focused on the treatment of COVID-19. Although there are lack of effective specific antivirals or drug combinations available for the treatment of COVID-19 supported by high-quality evidence [24], [25], [26], [27], [28], according to the current research findings, there are some potential drugs can help COVID-19 patients [29], [30], [31], [32], [33], which requires a great deal of continuity research with high quality. At the same time, prognosis was one of the most concerned subject by researchers, especially for pregnant and children, which still has a lot of uncertainty for researchers to pay attention [34].

Overall, the methodological and reporting quality of the included studies were low, highlighting the particular need to improve the design and conduct of SRs in the future. Regarding methodological quality, more than 80% of the included studies were assessed as low or critically low quality, and none of the included studies were assessed as high quality. In particular, four quality items (protocol and registration, comprehensive literature search strategy, listing of excluded studies, and funding reported for individual studies) were fully reported by less than 5% of studies. Researchers should pay greater attention to these items in the future. Similarly, during previous pandemics (such as SARS and MERS), the methodological quality of related SRs seems to have been low. By contrast, SRs on SARS showed higher methodological quality than those on MERS and COVID‐19 [16]. Regarding the reporting quality, the median PRISMA score was 14, which means that the average level of item reporting among the included studies was 14 out of 27 items; for some items related to methods and results, the reporting quality was lower. Notably, for item 5 (protocol and registration), only 29 of the 243 studies fully reported this information, which should be of concern to researchers as reporting the protocol and registration of studies may reduce research resource waste and improve research methodology and reporting quality to some extent [3,35].

The following causes may help to explain the poor methodological and reporting quality of SRs. First, the publication and writing cycles of SRs may be short (Appendix Table S3). Researchers and journal editors must speed up article writing and review to quickly respond to the range of issues brought about by COVID-19, which may lead to lower methodological and reporting quality of SRs. Second, there are few relevant original studies and the heterogeneity between studies may also be high. At the beginning of the outbreak, the number of RCTs was relatively small, and many COVID-19 SRs included only observational studies. Third, the methodological and reporting quality of SRs seem to vary from topic to topic (Fig. 5). The methodological and reporting quality of SRs related to prevention and etiology seems to be lower, whereas the quality of SRs involving treatment and prognosis seems to be higher. For some special populations (such as children and pregnant women), both methodological and reporting quality of SRs appear to be low (Fig. 5), which requires further attention. In addition, due to the need to rapidly share information related to COVID-19, it seems difficult for researchers to achieve detailed methodological descriptions and high reporting quality. However, such low study quality greatly reduces confidence in the use of this information by healthcare decision makers. Thus, whether conventional quality assessment tools [5,6] for SRs are still applicable during a public health emergency, and if special quality assessment tools should be used instead, warrants discussion.

6. Strengths and limitations

Compared with other studies [15,16], our study has several advantages. First, we systematically evaluated the methodological and reporting quality of the evidence accumulated over the last 6 months, which is of practical value for policy makers, clinicians, and researchers. Second, we explored the contributions of different countries and the trends in clinical topics and methodological and reporting quality of evidence over time, which is important for summarizing this epidemic. Third, the study revealed some evidence gaps that could help future researchers avoid wasting scientific resources. However, our study also has some limitations. First, the number of relevant SRs is rapidly increasing, and our findings are only based on SRs before the search date (June 15, 2020). With the emergence of newly related studies, regular updates of the existing results will be performed in 2 years. Second, we only analyzed SRs, excluding other types of studies. This choice was made because of the importance of SRs for decision makers. Third, AMSTAR-2 was used to assess SRs of diagnosis, prognosis, prevalence; however, this tool was developed to assess the methodological quality of SRs of healthcare interventions only. It was not developed for critical appraisal of other types of SRs. An important limitation is that we did not use tools specifically designed for critical appraisal of SRs of diagnosis, prognosis, prevalence (such as the AMSTAR II-DTA extension or ROBIS (Risk of Bias in Systematic Reviews).

7. Conclusion

This study assessed the methodological and reporting quality of SRs of COVID-19 using a comprehensive and innovative approach. In addition, it presented a summary and trend analysis of the clinical topics, countries, and populations of these studies. As such, this study serves as a comprehensive reference for researchers. Overall, the methodological and reporting quality of COVID-19 SRs were low, highlighting the importance of improving the design and conduct of SRs in the future. In addition, it is necessary to discuss whether conventional quality assessment tools for SRs are suitable during a public health emergency. Furthermore, we concluded that the existing SRs regarding specific transmission methods, treatments, prevention measures, and prognosis of COVID-19 are insufficient, as are the existing SRs of specific populations, such as children and pregnant women.

CRediT authorship contribution statement

Yanfei Li: Conceptualization, Methodology, Visualization, Formal analysis, Resources, Writing - original draft, Writing - review & editing. Liujiao Cao: Software, Validation, Visualization, Formal analysis, Investigation, Resources, Writing - original draft. Ziyao Zhang: Software, Validation, Investigation, Resources, Writing - original draft. Liangying Hou: Validation, Investigation, Resources, Writing - original draft. Yu Qin: Methodology, Validation, Investigation, Resources, Writing - original draft. Xu Hui: Validation, Investigation, Resources, Writing - original draft. Jing Li: Validation, Investigation, Resources, Writing - original draft. Haitong Zhao: Validation, Investigation, Resources, Writing - original draft. Gecheng Cui: Software, Investigation, Resources, Writing - original draft. Xudong Cui: Software, Formal analysis, Investigation, Resources, Writing - original draft. Rui Li: Investigation, Resources, Data curation, Writing - original draft. Qingling Lin: Investigation, Resources, Data curation, Writing - original draft. Xiuxia Li: Methodology, Validation, Data curation, Writing - review & editing, Project administration. Kehu Yang: Conceptualization, Validation, Data curation, Writing - review & editing, Supervision, Funding acquisition.