Assessment of the quality of systematic reviews on COVID‐19: A comparative study of previous coronavirus outbreaks

Abstract

Several systematic reviews (SRs) have been conducted on the COVID‐19 outbreak, which together with the SRs on previous coronavirus outbreaks, form important sources of evidence for clinical decision and policy making. Here, we investigated the methodological quality of SRs on COVID‐19, severe acute respiratory syndrome (SARS), and Middle East respiratory syndrome (MERS). Online searches were performed to obtain SRs on COVID‐19, SARS, and MERS. The methodological quality of the included SRs was assessed using the AMSTAR‐2 tool. Descriptive statistics were used to present the data. In total, of 49 SRs that were finally included in our study, 17, 16, and 16 SRs were specifically on COVID‐19, MERS, and SARS, respectively. The growth rate of SRs on COVID‐19 was the highest (4.54/month) presently. Of the included SRs, 6, 12, and 31 SRs were of moderate, low, and critically low quality, respectively. SRs on SARS showed the optimum quality among the SRs on the three diseases. Subgroup analyses showed that the SR topic (P < .001), the involvement of a methodologist (P < .001), and funding support (P = .046) were significantly associated with the methodological quality of the SR. According to the adherence scores, adherence to AMSTAR‐2 items sequentially decreased in SRs on SARS, MERS, and COVID‐19. The methodological quality of most SRs on coronavirus outbreaks is unsatisfactory, and those on COVID‐19 have higher risks of poor quality, despite the rapid actions taken to conduct SRs. The quality of SRs should be improved in the future. Readers must exercise caution in accepting and using the results of these SRs.

Research Highlights

• Researchers have taken rapid actions to conduct systematic reviews (SRs) during the COVID‐19 pandemic.

• A total of 49 SRs on COVID‐19, MERS and SARS were identified.

• The methodological quality of SRs on coronavirus outbreaks should be improved in the future.

• It is suggested to perform rigorous assessments when using the results of SRs.

1. INTRODUCTION

A novel coronavirus infection (now known as COVID‐19) was first reported in late 2019 in Wuhan, China, which then swept the globe rapidly. COVID‐19 is caused by severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2), which belongs to the genus betacoronavirus. At present, COVID‐19 has developed into a global pandemic, with more than 850 000 confirmed cases reported worldwide. ¹ Thus, evidence for clinical decision and policy making is urgently required.

Systematic reviews (SRs), a type of literature review, seek to collate the data of all primary studies that fit prespecified eligibility criteria to address a specific research question. Practitioners mainly rely on SRs for evidence to provide evidence‐based recommendations. Before the COVID‐19 pandemic, outbreaks of severe acute respiratory syndrome (SARS) and the Middle East respiratory syndrome (MERS) were observed in 2003 and 2012, respectively; substantial numbers of SRs have been conducted on these outbreaks. The causative agents of these three outbreaks belong to the same genus of coronavirus (CoV), and SARS‐CoV‐2 shares a 79.5% sequence identity with SARS‐CoV. Thus, the available SRs on SARS and MERS are helpful in guiding COVID‐19 management. However, the quality of these SRs was unclear. In addition, besides these existing SRs, we also found that a large number of newly‐conducted SRs with or without meta‐analyses have been published rapidly, shortly after the COVID‐19 outbreak, and the number of such SRs keeps rising. As we know, at the preliminary stage of any public emergency, primary studies are usually lacking, and most studies are observational. Thus, it is not easy to conduct an SR. These recent SRs on COVID‐19 are more likely to have the potential methodological flaws. Therefore, we hypothesized that there may be a difference in methodological quality between these SRs on COVID‐19 and previous SRs on SARS and MERS.

Up to now, no studies appraising the methodological quality of the SRs on COVID‐19 and previous MERS and SARS outbreaks are available. In this comparative study, we investigated the present status of conducting SRs on COVID‐19, MERS, and SARS, appraised the methodological quality of these SRs using the a measurement tool to assess systematic reviews (AMSTAR 2), and performed a preliminary examination of the potential risk factors associated with the quality of SRs, with the aim of providing suggestions from the aspects of methodological quality for conducting and using SRs during the COVID‐19 pandemic.

2. MATERIALS AND METHODS

2.1. Data sources and search

We searched for eligible SRs (and/or meta‐analyses) in both English and Chinese databases (Medline, Embase, Cochrane library, Chinese National Knowledge Infrastructure [CNKI], Chinese Biomedical Literature Database [CBM], and Wanfang database) up to 23 March 2020. We used a combination of subject terms with free‐text terms during the search. Searches in these databases were individually performed twice to obtain SRs on COVID‐19 and those of SARS and MERS. We also tracked the references of the included studies for additional publications. Google Scholar was used to search for relevant studies in the gray literature. Details of the search strategy are presented in the Supplementary Information Materials.

2.2. Inclusion and exclusion criteria

We included all SRs which discussed one of the following diseases: COVID‐19, SARS, and MERS. SRs were identified based on the following criteria: (a) label of “systematic review” in the title, abstract, or full text; and (b) a literature search was performed. There were no restrictions on the SR topic. SRs with or without meta‐analysis were both eligible. Only SRs published in academic journals were included. Studies were excluded based on the following criteria: (a) letters, editorials, and expert opinions; case reports and literature reviews; and other narrative reviews; (b) abstract only or data unavailable; or (c) studies in non‐English or non‐Chinese languages. When a duplicate or updated publications were identified, only the most recent one was included.

2.3. Study selection and data extraction

Two reviewers independently screened the titles and abstracts of all articles identified in the initial search and then checked the full texts of potentially eligible studies. Any disagreement was resolved by discussion or consultation with a third reviewer. The same reviewers used a predesigned table to collect the following information on the included SRs: first author, publication year, the growth rate of SRs, region or country in which the study was performed, study topic (epidemiology, clinical characteristic, diagnosis, treatment, etc), source of publication, use of meta‐analysis, presence or absence of funding support, and involvement of a methodologist in conducting SRs. The growth rate was calculated by the following formula: rate = the number of SRs/the time interval from the beginning of the outbreak to the time of conducting this present study. The region of the study was defined as the location of the first author's institution; the source of publication was classified on the basis of SJR Best Quartile ³ and Chinese core journal criterion of PKU, ⁴ and funding was identified based on the declaration in the publication. A methodologist was defined as a contributing author specializing in evidence‐based medicine, epidemiology, or statistics.

2.4. Assessment of the methodological quality of the included SRs

The methodological quality of the included SRs was independently assessed by two well‐trained reviewers using the AMSTAR‐2 tool. ⁵ , ⁶ Any disagreement between the two reviewers was resolved by discussion, and in case of persistent disagreement, a resolution was arrived at by consulting with a third person.

The AMSTAR‐2 is a critical appraisal tool for SRs of randomized control trials and/or observational studies. ⁶ The AMSTAR‐2 contains 16 items, of which seven are critical domains (Table 1). A question‐specific point scale (“Yes, No;” “Yes, Partial yes, No;” or “Yes, No, No meta‐analysis”) was used to score each item. Based on the degree of weaknesses detected in critical and noncritical items, the AMSTAR‐2 classifies the overall confidence in the results of the SRs into four levels: high, moderate, low, and critically low. We graded each included SR on the AMSTAR‐2 official website and generated the overall confidence using their online calculator. In addition, to compare adherence to items among SRs, a total score was calculated for each SR according to the response to each item, that is, “Yes,” “Partial yes,” “No meta‐analysis,” and “No” were scored as “+1,” “+0.5,” “0,” and “−1,” respectively.

Table 1.

AMSTAR 2 critical domains

Item 2	Protocol was registered before the commencement of the review
Item 4	Adequacy of the literature search
Item 7	Justification for excluding individual studies
item 9	Risk of bias from individual studies being included in the review
Item 11	Appropriateness of the meta‐analytical methods
Item 13	Consideration of risk of bias when interpreting the results of the review
Item 15	Assessment of the presence and likely impact of publication bias

Abbreviation: AMSTAR, a measurement tool to assess systematic reviews.

2.5. Data analysis

Descriptive statistics were used to present the characteristics and methodological quality of the included SRs. Categorical variables were expressed as frequencies and percentages. The response to each AMSTAR‐2 item was recorded and tabulated for all included SRs. Overall adherence to the 16 items were analysed with a percent stacked bar chart. The adherence scores in each disease group were presented as mean ± standard deviation (SD). Subgroup analysis was performed to compare the differences in quality among subgroups with different study characteristics. Statistical significance was tested using the Kruskal‐Wallis rank test and two‐sample Mann–Whitney U test. P < .05 was considered to be statistically significant, and all P values were two‐sided. Excel 2019 and SPSS V25 were used for all data management and analyses.

3. RESULTS

3.1. Selection of SRs

A total of 363 records were identified during the initial search, including 283 records from English databases and 80 records from Chinese databases. After removing duplicate records, 280 records were sent for the first round of screening. We excluded 223 records in the first round of screening of the titles and abstracts. The full texts of the remaining studies were reviewed, and eight SRs were excluded due to the following reasons: not a related topic, not an SR, duplicate publication, and no full text. No additional SRs were identified on Google Scholar. Thus, 49 SRs ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ were finally included in this study for methodological quality assessment. The flow diagram of the study selection process is shown in Figure 1.

Figure 1.

Flow diagram of the search and selection process for SRs. COVID‐19, the coronavirus disease 2019; MERS, Middle East respiratory syndrome; SARS, severe acute respiratory syndrome; SR, systematic review

3.2. Characteristics of the included SRs

Of the 49 SRs, 17 SRs ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ were targeted at COVID‐19, 16 SRs ⁷ , ⁸ , ⁹ , ¹⁰ , ¹³ , ¹⁴ , ²⁰ , ²² , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³¹ , ³² , ³⁸ at MERS, and 16 SRs ¹¹ , ¹² , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²¹ , ²³ , ²⁴ , ³⁰ , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ at SARS. The publication years of these SRs ranged from 2004 to 2020, and 59% of SRs were published recently, within the past 4 years. All SRs on COVID‐19 were recent, published within 3 months of the search cut‐off date. The growth rate of SR number decreased sequentially from COVID‐19 to MERS to SARS. Most SRs were performed in Asia (65%, n = 32), followed by Europe (14%, n = 7) and North America (14%, n = 7). Thirty‐eight SRs were published in English journals and 11 SRs in Chinese journals; of these, 63% (n = 31) were published in Q1 or core journals. These SRs covered nearly all aspects of an infective disease, with most of them discussing therapeutic options, and clinical characteristics and outcomes. SRs on COVID‐19 focused on a limited number of topics compared to SRs on MERS and SARS. More than half (63%, n = 31) of the SRs did not have a methodologist involved during the course of the review, with the highest proportion of which in SRs on COVID‐19 (82%, n = 14) and the lowest proportion of which in SRs on SARS (31%, n = 5). In addition, 55% of the SRs (n = 27) involved meta‐analyses, and nearly half of the SRs (49%, n = 24) were supported by at least one funding source (Table 2).

Table 2.

Characteristics of SRs and factors related to methodological quality

Characteristics	Group			Total (%)	AMSTAR rating
	COVID‐19 (n = 17)	MERS (n = 16)	SARS (n = 16)		High (n = 0)	Moderate (n = 6)	Low (n = 12)	Critically low (n = 31)	P value
Growth rate (/month)	4.54	0.17	0.08	…	…	…	…	…	…
Publication year
2002‐2006	0	0	10	10 (20%)	0	4	2	4	.152
2007‐2011	0	0	2	2 (4%)	0	0	1	1
2012‐2016	0	6	2	8 (16%)	0	1	2	5
2017‐2020	17	10	2	29 (59%)	0	1	7	21
Location
Asia	11	9	12	32 (65%)	0	3	7	22	.642
Europe	4	1	2	7 (14%)	0	2	0	5
North America	1	4	2	7 (14%)	0	1	3	3
South America	1	0	0	1 (2%)	0	0	1	0
Africa	0	2	0	2 (4%)	0	0	1	1
Journal rank
English SRs (n = 38)
Q1	9	9	6	24 (49%)	0	3	5	16	.279
Q2	4	4	2	10 (20%)	0	1	2	7
Q3	0	3	0	3 (6%)	0	0	2	1
Q4	0	0	1	1 (2%)	0	1	0	0
Chinese SRs (n = 11)
Core	2	0	5	7 (14%)	0	1	0	6	.121
Noncore	2	0	2	4 (8%)	0	0	3	1
Topics
Clinical characteristics and outcomes	10	6	4	20 (41%)	0	0	6	14	<.001
Epidemiology and transmission	0	2	1	3 (6%)	0	0	0	3
Diagnostic approach	0	1	0	1 (2%)	0	0	1	0
Therapeutic options (Western medicine)	6	4	3	13 (27%)	0	1	2	10
Therapeutic options (integrated traditional Chinese and Western medicine)	0	0	7	7 (14%)	0	4	3	0
Integrative assessment	1	3	0	4 (8%)	0	0	0	4
Psychological wellbeing of healthcare workers	0	0	1	1 (2%)	0	1	0	0
Involvement of methodologist
Yes	3	4	11	18 (37%)	0	6	7	5	<.001
No	14	12	5	31 (63%)	0	0	5	26
Meta‐analysis
Yes	9	5	13	27 (55%)	0	4	9	14	.088
No	8	11	3	22 (45%)	0	2	3	17
Funding support
Yes	9	9	6	24 (49%)	0	3	6	15	.046
No	2	5	1	8 (16%)	0	0	0	8
Not reported	6	2	9	17 (35%)	0	3	6	8

Abbreviations: COVID‐19, the coronavirus disease 2019; MERS, Middle East respiratory syndrome; SARS, severe acute respiratory syndrome; SR, systematic review.

3.3. Methodological quality of SRs

3.3.1. Quality rating by the AMSTAR‐2

Of the 49 SRs, only 6 SRs (12%)  ¹¹ , ¹⁷ , ²³ , ²⁴ , ³³ , ³⁴ were of moderate quality, 12 (24%) ¹⁰ , ¹² , ²² , ²⁹ , ³⁵ , ³⁶ , ³⁸ , ⁴³ , ⁴⁸ , ⁴⁹ , ⁵⁴ , ⁵⁵ were of low quality, and the remaining 31 SRs (63%) ⁷ , ⁸ , ⁹ , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ³⁰ , ³¹ , ³² , ³⁷ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ were of critically low quality. Not a single SR was rated as having high quality. The quality of all SRs on COVID‐19 was rated as low (29%, n = 5) or critically low (71%, n = 12), similar to that of SRs on MERS (low, 25%, n = 4; critically low, 75%, n = 12) but inferior to that of SRs on SARS (moderate, 38%, n = 6; low, 19%, n = 3; critically low, 44%, n = 7).

3.3.2. Factors related to SR quality

The potential factors affecting the quality of SRs were investigated in all samples. Subgroup analyses of different variables showed that the topic of the SR, involvement of a methodologist, and funding support were significantly associated with the methodological quality of the SRs (Table 2; P < .001,<.001, and .046, respectively). Regarding other variables, no significant differences in quality were observed.

3.3.3. Adherence to each item of the AMSTAR‐2

Figure 2 illustrated the methodological quality of 49 SRs according to each item of the AMSTAR‐2. Excluding three items (items 11, 12, and 15) related to meta‐analysis, items 1 (PICO: populations, interventions, comparisons, and outcomes, 63%) and 16 (conflict of interest, 63%) were found to have the best adherence, followed by item 5 (study selection in duplicate, 51%). Items 11 (statistical combination of results), 12 (impact of risk of bias [RoB] in individual studies on meta‐analysis), and 15 (publication bias) received a “Yes” response for 63% (17/27), 33% (9/27), and 36% (10/28) of SRs with meta‐analysis, respectively. The scores calculated for adherence to each item showed that the overall adherence to the methodological items decreased in the order of SRs on SARS (mean score ± SD, −1.00 ± 8.89), MERS (−2.63 ± 4.35), and COVID‐19 (−3.94 ± 5.04). Details of the assessment of each item are described in the Table S1.

Figure 2.

Adherence to each item in AMSTAR‐2. AMSTAR, a measurement tool to assess systematic reviews; MA, meta‐analysis

4. DISCUSSION

SRs provide the highest level of evidence in evidence‐based medicine, and they are an important source of information for clinical practitioners and policy makers. However, due to the relatively low requirements and costs of conducting SRs, the number of SRs published in various fields is quite high. The increase in quantity has also brought concerns about the quality of SRs. ⁵⁶ In this study, we found that a large number of SRs were conducted de novo after the COVID‐19 outbreak, and together with previous SRs on MERS and SARS, they formed an evidence map covering various topics. It was necessary to conduct SRs specific to the population of COVID‐19 because the evidence from SRs on SARS and MERS was indirect for COVID‐19 patients. A total of 49 SRs that met our criteria were identified. It was encouraging to find that more than one‐third of the SRs were targeted at COVID‐19 and that all of them were published within 3 months of the COVID‐19 outbreak. The growth rate of SRs on COVID‐19 was significantly higher than that of SRs on MERS and SARS, showing the fast response of researchers to an emerging disease pandemic. However, the methodological quality of SRs has not improved significantly. Evaluation of methodological quality of the included SRs using the AMSTAR‐2 tool suggested that the overall confidence level for SRs was unsatisfactory, with most SRs having critically low quality; however, surprisingly, SRs on SARS showed superior overall confidence than those on MERS and COVID‐19. In addition, the adherence to methodological items in SRs decreased with the order of outbreaks of these diseases. These findings indicated that methodological dilemmas commonly exist in the SRs of interest and that the current SRs on COVID‐19 have higher risks of poor methodological quality.

Following may be the reasons for the poor quality of SRs. First, conducting rigorous SRs is actually a time‐consuming and laborious task. A previous survey showed that it takes an average of 67.3 weeks to complete and publish an SR, and funded SRs require longer than nonfunded SRs. ⁵⁷ However, the urgency associated with the outbreaks has forced researchers to shorten the time for conducting SRs. Second, experienced systematic reviewers are often lacking in teams conducting SRs. At present, no licence is required to conduct SRs, which results in most SR teams lacking members specializing in conducting SRs. The subgroup analysis in our study revealed that the methodological quality of SRs with methodologist involvement was significantly superior to that of those without methodologist involvement. Third, the absence of research funding also has an impact on the quality of SRs. Funded SRs always have a tailor‐made and peer‐reviewed protocol, which help in improving the quality of SRs. Fourth, the primary studies available for review are scarce, and heterogeneity is high across a limited number of studies. For unknown emerging diseases, the scarcity of large samples and high‐quality original reports are major challenges in scientific research. In particular, in the early stages of disease outbreaks, evidence is mostly acquired from observational studies, ⁵⁸ such as case reports and case series, which are possibly biased by multiple confounding factors; thus, homogeneity across published studies is extremely low. Heterogeneity in evidence increases the difficulty in conducting SRs and demands that reviewers be highly experienced and skilled to deal with the heterogeneity and to accurately interpret data. Fifth, the association between SR topics and quality is yet to be confirmed. Although SRs on treatments seem to have good quality, we believe that this finding is not a direct result of the differences between topics because SRs on treatments in this survey were often more likely to involve methodologists (an important variable related to SR quality).

Although SRs are often considered to be the strongest form of scientific evidence because they can increase the statistical power and help resolve conflicting results across studies, ⁵⁹ it is imperative that SRs have high quality. Several clinicians are at a disadvantage while distinguishing the quality of evidence, especially in situations involving infectious disease outbreaks, because data from low‐quality SRs may be mistaken for high‐quality evidence, resulting in misguided clinical practice. Variations in the methodological quality of SRs might lead to different answers to the same question, ⁶⁰ which could result in confusion among public officials and thus hamper decision making. In addition, it is troubling that SRs are conducted and published by entities that have a clear stake in the publishing of positive results, such as pharmaceutical companies. During outbreaks of infectious diseases, panic‐stricken individuals can be easily misled by incorrect interpretations of data. Thus, considering that the number of SRs on the COVID‐19 pandemic will continue to increase, we suggest that researchers conducting SRs follow rigorous and scientific standards to improve methodological quality. SRs must be conducted for the right reasons, and the selected questions should be useful to clinicians and public health officials for avoiding wastage of resources. As we found in this study, the COVID‐19‐related topics covered by SRs are still limited, with several SRs focusing on the same question. Development and registration of protocols before beginning SRs can help optimize the use of finite resources to avoid unnecessary duplications, reduce the selective outcome‐reporting bias, and improve collaboration. ⁶¹ First‐line clinical professionals can cooperate with full‐time systematic reviewers to improve the transformation efficiency of SR results. At the beginning of a disease outbreak, researchers should pay more attention to creating primary evidence than to performing SRs. The impact of the potential RoB should be considered while drafting the results and conclusions of the SR, and they must be clearly reported to the readers in corresponding texts. As for clinicians, policy makers, guideline panels, and other stakeholders, it is important to better critique available evidence, and caution must be exercised while using SRs as the basis for policy and decision making.

This is the first study to investigate the methodological quality of SRs on COVID‐19, SARS, and MERS. We performed a rigorous search to assess all relevant SRs. Furthermore, we selected an updated tool (AMSTAR‐2) as the quality assessment tool for this study. However, our study has some limitations. First, only studies published in English or Chinese were included in this investigation. Noninclusion of data from other languages may influence the results of this study. Second, the judgment of a methodologist was mainly based on the information that has been released. There may be some discrepancies with the reality. Third, although several factors were found to have an association with SR quality, the limited number of SRs included in this study may have influenced this result. Finally, we addressed the quality of SRs using only the AMSTAR‐2 tool. The assessment of quality according to AMSTAR‐2 only represents the assessment of SR conduction and is not really related to the reporting or evidence quality. For more comprehensive evaluation, other tools, such PRISMA and GRADE, showed be used for assessment.

5. CONCLUSIONS

SRs specific to COVID‐19, SARS, and MERS are being heavily relied upon during the current COVID‐19 pandemic. The methodological quality of most SRs is unsatisfactory, and those on COVID‐19 have higher risks of poor quality, despite the rapid actions taken to conduct SRs. Teams that may want to conduct a SR should focus on the study design and focus on improving the quality of the SR. SR findings should be used more cautiously, and it is not advisable that users accept the results of a single SR without critical appraisal.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

AUTHOR CONTRIBUTIONS

YY and HC conceived the idea for this study. YY designed the study. YY and QLS performed the literature search. YY, PZ, and HC selected the eligible studies. LG, HYL, PXT, BHG, and DFW extracted the data. YY and QLS assessed the quality of the included studies. YY, QLS, and HC performed the data analysis and interpreted the results. YY drafted the first version of the article. All authors read and provided important revisions to this article and approved the final version.

Supporting information

Supplementary information

Supplementary information

Supplementary information

Yu Y, Shi Q, Zheng P, et al. Assessment of the quality of systematic reviews on COVID‐19: A comparative study of previous coronavirus outbreaks. J Med Virol. 2020;92:883–890. 10.1002/jmv.25901