The mass production of systematic reviews about COVID‐19: An analysis of PROSPERO records

Abstract

Objective

This study aimed to assess the characteristics of different designs of systematic reviews (SRs) registered in the International Prospective Register of Systematic Reviews (PROSPERO) about COVID‐19.

Methods

The search was performed in the PROSPERO database using the strategy proposed by the database and considered only human studies. The last date of the search was April 27, 2020. Full text of all records was accessed, and data were extracted by a single researcher, which was further double‐checked by another researcher. A descriptive analysis was performed considering record characteristics using tables.

Results

We included 564 records from which the vast majority were registered as SRs (n = 513, 91%). In general, we found poor reporting and missing or confusing information, since 84% of the records (n = 474) did not report the full search that would be adopted, 16% (n = 90) did not report clearly the databases that would be used, and 49.1% (n = 277) did not report the number of primary outcomes. The main focus of most of the records involved clinical, epidemiological, complication, and laboratory characteristics (n = 173, 30.7%) or the treatment of COVID‐19 (n = 138, 24.5%).

Conclusion

A large number of SRs about COVID‐19 have been conducted, and many of the assessed records were poorly reported and would be difficult to replicate. Besides, collected data points to an epidemic of redundant reviews on COVID‐19.

1. INTRODUCTION

The novel coronavirus pandemic is heavily impacting almost all countries in the world, both from health and economic perspectives. At the moment, about 350 000 (updated data can be accessed in: https://www.worldometers.info/coronavirus/) people have died due to coronavirus disease 2019 (COVID‐19) complications. To reduce these numbers, researchers are responding at breathtaking speed, conducting everything from basic research to controlled trials to test possible treatment options ¹ ; however, at present, no treatment, prophylaxis, or vaccine is effective in treating COVID‐19, ² though countries that were highly affected by the pandemic tried to redirect different drugs to the treatment of COVID‐19. ³

Although there is an urgency in finding an effective answer to manage COVID‐19, no definitive response will be found without proper research design, conduct, and report, and there are already some retractions in the literature about studies for the treatment of COVID‐19. ⁴ Another example of the importance of well‐designed and conducted studies during the pandemic is an earlier study conducted on a small sample of patients that reported a positive effect of the adjunctive use of hydroxychloroquine and azithromycin in critical patients, reducing the convalescence time and fatality rates. ⁵ However, it turned out that a larger multinational real‐world analysis not only did not confirm these previous findings but also observed an increased hazard for a clinically significant occurrence of ventricular arrhythmias and in‐hospital death with COVID‐19. ³ The adjunctive use of other drugs was also considered and debated (as dexamethasone, favipiravir, ivermectin, remdesivir), but none prove to be efficient, to date. ⁶ , ⁷ , ⁸ , ⁹

It is well known that systematic reviews with meta‐analysis can provide the best levels of evidence, ¹⁰ being an important tool to define health care strategies and resource allocation planning. Previous papers ¹ , ² have warned about the risk of low quality and duplication of ongoing research that could result in “massive waste in research.” ¹ Still, in this sense, the Cochrane Handbook for Systematic Reviews of Interventions ¹¹ emphasizes that a systematic review protocol should be developed and registered in specific databases before its start, to avoid reporting bias or the change of the previously planned study design. However, a study by Tricco et al ¹² observed that a third of systematic reviews changed or did not specify their primary outcome on the register and that only 18% of published systematic reviews have previously been registered.

Considering these aspects, this study aimed to assess the characteristics of different designs of systematic reviews (SR) records registered in the International Prospective Register of Systematic Reviews (PROSPERO) about COVID‐19.

2. METHODS

The study protocol was registered in the open science framework (OSF) and is available at the following link: https://osf.io/9umqj/.

2.1. Inclusion criteria and search

We selected SRs or any other knowledge synthesis designs (ie, rapid reviews, umbrella reviews, and living reviews) about COVID‐19 registered in the PROSPERO database. Studies were included regardless of review questions and characteristics. The search was performed in the PROSPERO database using the strategy proposed by the database to identify study records about COVID‐19, considering only human studies (Supporting Information). The last search was conducted on April 27, 2020.

2.2. Records screening

Records identified in PROSPERO were downloaded in Endnote format as control of included studies, and the records’ full texts were accessed directly on the PROSPERO platform. Only the first version of each record was considered at this time to standardize the records because it is possible to change information during the study development.

We created a standardized form using the Excel software, which was tested by all the researchers to reach a consensus of data collections.

2.3. Data extraction

Registers were divided among three researchers to perform data extraction, and after, one researcher double‐checked all data. In order to register an SR, the PROSPERO database requires different information varying from the review title, details about the organization, and authors responsible by the study to details related to the methodology. We decide to collect data related to the characteristics of the study and methodological data related to study replication. The following data was collected: registration date, country of conduct, funding (based on the Cochrane Effective Practice and Organisation of Care Review Group recommendations), review question/s, databases, description of full search strategy, type of study to be included, intervention studied, number of primary outcomes in the record (classified as “Unclear” when it was impossible to judge the number of primary outcomes), number of secondary outcomes in the record (classified as “Unclear” when it was impossible to judge the number of secondary outcomes), synthesis strategy, additional analysis, risk of bias tool, type, and method of review. Also, we categorized the different tested treatments in records of COVID‐19 treatments.

2.4. Data analysis

A descriptive analysis was performed considering the characteristics of the records using proportions, median, and IQR through tables. To map different tested treatments, we also created a table displaying all records about COVID‐19 treatments and a map depicting the origin and distribution of SR records using the Adobe Photoshop CS6 software.

3. RESULTS

The search yielded 574 records, and 10 were excluded because the full texts were not identified in the PROSPERO platform during the data extraction phase. The majority of these studies were registered as SRs (n = 513, 91%; Table 1). The conduct of the majority of the registered records (n = 210, 37.2%; Table 1) was not started before April 27, 2020, and only four studies had already performed data extraction/analysis (data extraction n = 1, 0.2%; data analysis n = 3, 0.5%; Table 1). The great majority of the records did not report the existence or the absence of conflict of interest (n = 501, 88.3%; Table 1), and 34.4% of the records received some type of funding (Table 1).

TABLE 1.

Characteristics of systematic review registration records/PROSPERO records included (n = 564)

Characteristics		n	%
Type of systematic review
	Systematic review	513	91.0%
	Rapid review	26	4.6%
	Living systematic review	17	3.0%
	Network meta‐analysis	3	0.5%
	Rapid and living systematic review	2	0.3%
	Qualitative systematic review	1	0.2%
	Umbrella review	1	0.2%
	Unclear	1	0.2%
Last stage started
	Not started	210	37.2%
	Preliminary searches	130	23.0%
	Formal screening of search results against eligibility criteria	123	21.1%
	Piloting of the study selection process	90	16.0%
	Risk of bias (quality) assessment	7	1.2%
	Data extraction	1	0.2%
	Data analysis	3	0.5%
Conflict of interest
	Not reported	501	88.3%
	None	55	9.7%
	Yes	8	1.4%
Funding
	No funding	370	65.6%
	Governmental organization	107	19.0%
	Unclear	31	5.5%
	Mixed	20	3.5%
	Other	15	2.7%
	Research funding body	9	1.6%
	Charitable trust	5	0.9%
	Health care provider organization	5	0.9%
	Commercial organization	2	0.3%
Main focus
	Clinical, epidemiological, complication, and laboratory characteristics	173	30.7%
	Treatment	138	24.5%
	Unclear	68	12.1%
	2 or more	51	9.0%
	Other	41	7.3%
	Psychological questions	40	7.1%
	Diagnosis	22	3.9%
	Association of factors	17	3.0%
	Prevention	14	2.5%
Full search not reported		474	84.0%
Number of databases considered “Unclear”		90	16.0%
Number of databases (median/IQR)		4	3‐6
Databases searched
	MEDLINE/PubMed	557	98.9%
	SCOPUS	135	24.0%
	EMBASE	434	77.1%
	Web of Science or Web of Knowledge	239	42.4%
	GoogleScholar	113	20.1%
	Others	457	81.2%
Number of primary outcomes considered “Unclear”		277	49.1%
Number of primary outcome (median/IQR)		2	1‐3
Number of secondary outcomes considered “Unclear”		176	31.2%
Number of secondary outcomes (median/IQR)		0	0‐2
Synthesis strategy
	Meta‐analysis	333	59.0%
	Descriptive/narrative and meta‐analysis	126	22.3%
	Descriptive analysis	48	8.5%
	Narrative analysis	25	4.4%
	Unclear	15	2.7%
	Network meta‐analysis	11	1.9%
	Other	6	1.1%
Additional analysis
	Subgroup	386	68.4%
	None	101	17.9%
	Subgroup analysis and one or more additional analysis	56	9.9%
	Unclear	17	3.0%
	Publication	3	0.5%
	Sensitivity analysis	1	0.2%
Risk of bias tool reported
	Cochrane risk of bias tools	147	26.1%
	Two or more different tools	128	22.7%
	Newcastle Ottawa	81	14.4%
	Other	79	14.0%
	Unclear	63	11.2%
	Joanna Briggs Institute Critical Appraisal Checklist	16	2.8%
	NHI tools	16	2.8%
	None	16	2.8%
	GRADE	15	2.7%
	Not reported	3	0.5%
Study designs included
	One or more observational studies	131	23.2%
	Randomized and/or nonrandomized controlled trials and observational studies	122	21.6%
	All study designs	100	17.7%
	Randomized controlled trials	99	17.5%
	Others	39	6.9%
	Unclear	35	6.2%
	Randomized and nonrandomized controlled trials	23	4.1%
	Clinical studies	9	1.6%
	Studies performed on human beings	6	1.1%

This article is being made freely available through PubMed Central as part of the COVID-19 public health emergency response. It can be used for unrestricted research re-use and analysis in any form or by any means with acknowledgement of the original source, for the duration of the public health emergency.

The main focus of most of the records involved clinical, epidemiological, complication, and laboratory characteristics (n = 173, 30.7%; Table 1) or the treatment of COVID‐19 (n = 138, 24.5%; Table 1). Despite that, 84% of the records (n = 474, Table 1) did not report the full search that would be adopted. Also, in 16% of the assessed records, even the databases that would be used were unclear (n = 90, Table 1). The number of primary outcomes was unclear in 49.1% of the records (n = 277, Table 1). The most proposed synthesis strategy method was meta‐analysis (n = 333, 59%; Table 1), and, as additional analysis, subgroup analysis (n = 386, 68.4%; Table 1). The risk of bias tool claimed to be used varied between Cochrane risk of bias tools (n = 147, 26.1%; Table 1), Newcastle‐Ottawa tool (n = 81, 14.4%; Table 1), or the use of two or more different tools (n = 128, 22.7%; Table 1).

Table 2 describes the options of treatments that are proposed to be explored, and notably, there was a large variety of treatment options. Despite that, it should be emphasized that there were 36 records exploring the effects of Chinese medicine, and 16 focused on chloroquine/hydroxychloroquine records, which could be indicative of a risk for duplication of data and overlapping of such studies.

TABLE 2.

Treatment proposed to be evaluated by the included records (n = 157)

Treatments	n	%
Unclear	19	11.9
Traditional Chinese medicine	19	11.9
Chloroquine and/or hydroxychloroquine	12	7.5
Traditional Chinese and Western medicine	6	3.8
Lianhua Qingwen capsules	5	3.1
Blood components	5	3.1
Acupuncture	4	2.5
Corticosteroids	4	2.5
Different types of treatment	4	2.5
Favipiravir	3	1.9
Lopinavir/ritonavir	3	1.9
Pharmacological treatment	3	1.9
Physiotherapy techniques	2	1.3
Angiotensin‐converting enzyme inhibitors and angiotensin II receptor blockers	2	1.3
Atazanavir	2	1.3
Chloroquine plus azithromycin	2	1.3
Extra corporeal membrane oxygenation	2	1.3
External treatment of traditional Chinese medicine	2	1.3
Immunosuppressive drugs	2	1.3
Interferon	2	1.3
Jinhua Qinggan granule	2	1.3
Noninvasive ventilation	2	1.3
Qing Fei Bai Du decoction	2	1.3
Remdesivir	2	1.3
Ribavirin	2	1.3
Chinese Herbal medicine	2	1.3
Traditional medicine as adjunctive therapy	2	1.3
Xuebijing injection	2	1.3
Favipiravir and remdesivir	1	0.6
Interleukin 1 inhibitor agents (specifically Anakinra) VS anti‐IL6 monoclonal antibodies (tocilizumab, sarilumab, siltuximab)	1	0.6
Traditional Chinese medicine injections	1	0.6
Volume‐controlled ventilation	1	0.6
Adjuvant pharmacologic therapy, standard care, definitive pharmacologic therapy, combined definitive pharmacologic	1	0.6
Advanced respiratory support; basic respiratory support	1	0.6
All suggested pharmacological therapies	1	0.6
Arbidol	1	0.6
Antiparasitic drugs	1	0.6
Antiviral drugs	1	0.6
Bacillus Calmette–Guérin vaccine (BCG) vaccination	1	0.6
Chinese exercise	1	0.6
Combination of traditional Chinese and Western medicine	1	0.6
Convalescent plasma transfusion vs standard care	1	0.6
Convalescent plasma or hyperimmune immunoglobulin	1	0.6
Enoxaparin or fondaparinux	1	0.6
Favipiravir, remdesivir, galidesivir, ivermectin, oseltamivir, ganciclovir, lopinavir/ ritonavir, darunavir, chloroquine, hydroxychloroquine, arbidol, azithromycin, amoxicillin, moxifloxacin, ceftriaxone, antifungals, androgen receptor blockers, tea, and traditional Chinese medicine	1	0.6
Hydroxychloroquine vs azithromycin	1	0.6
Hydroxychloroquine vs standard treatment	1	0.6
Interferons alone or in combination with other drugs	1	0.6
Lopinavir combined with ritonavir	1	0.6
Integrative traditional Chinese and Western medicine	1	0.6
Interferon, lopinavir, ritonavir, ribavirin, remdesivir	1	0.6
Rheumatic disease therapy	1	0.6
Monoclonal antibodies	1	0.6
Macrolides	1	0.6
Mesenchymal stem cell‐based therapy	1	0.6
Vitamin C	1	0.6
Tocilizumab alone, tocilizumab combination therapy with antivirals and/or other drugs	1	0.6
Siddha medicine	1	0.6
Pharmacological interventions, fluid therapy, invasive or noninvasive ventilation, or similar interventions	1	0.6
Qingfei Paidu decoction	1	0.6
Traditional Chinese exercise	1	0.6
Moxibustion	1	0.6
Tai Chi	1	0.6
Traditional Chinese medicine nonpharmacological interventions	1	0.6
Pharmacologic and nonpharmacologic	1	0.6
Massage therapy	1	0.6

This article is being made freely available through PubMed Central as part of the COVID-19 public health emergency response. It can be used for unrestricted research re-use and analysis in any form or by any means with acknowledgement of the original source, for the duration of the public health emergency.

Table 3 shows the ten countries that contributed the most records. China had the most contribution with 201 records (35.6%), followed by the United Kingdom (n = 67, 11.9%) and Brazil (n = 44, 7.8%).

TABLE 3.

Illustration of the 10 countries that contributed the most records

Position	Country	n	%
1	China	201	35.6
2	England	67	11.9
3	Brazil	44	7.8
4	Canada	26	4.6
5	India	20	3.6
6	USA	18	3.2
7	Iran	16	2.8
8	Italy	14	2.5
9	Ethiopia	11	2.0
10	Australia	8	1.4
	Spain	8	1.4

This article is being made freely available through PubMed Central as part of the COVID-19 public health emergency response. It can be used for unrestricted research re-use and analysis in any form or by any means with acknowledgement of the original source, for the duration of the public health emergency.

4. DISCUSSION

We assessed the characteristics of the SR records on COVID‐19 in PROSPERO, the amount of which is exponentially increasing (as of May 26, 2020, the number of human records registered reached 1017). Without a doubt, there are signs of poor reporting of the records (absence of proper description of outcomes considered, databases explored, a full search strategy that would be adopted, among other important method characteristics), resulting in imprecise studies/results, which would be difficult to replicate. Also, our results demonstrated a great duplication of records assessing similar/equal topics, unleashing a “massive waste in research,” as previously suggested by Glasziou et al. ¹ Page et al ¹³ state that duplicates systematic reviews waste time and resources, creating extra work for health care providers and other users who need to determine what unique information, if any, each review provides. Duplication can also create confusion when SRs addressing the same research question reach conflicting findings. ¹⁴ It is important to bear in mind that some duplicated reviews can be explained by the fact that there is a gap between the submission of a record and its publication. This gap could take around 30 days, as it is currently stated on the PROSPERO website. Also, due to high traffic and increased number of registers, some of the records can be published without being checked by the PROSPERO staff if the gap from submission to publishing exceeds 30 days.

In our analysis, China was the country that had registered the most significant number of records. It is clear that many researchers from different regions around the world desire to contribute toward addressing the COVID‐19 pandemic and that such attempt could lead to duplication of SRs being conducted. However, that focus would be better expended on creating strategies of collaboration among different research groups to provide more reliable data. ¹³

It is also understandable that during a pandemic period, the time‐consuming peer‐review process of records submitted for registering platforms may become compromised, but this only emphasizes the need for researchers to carefully consider and evaluate if there is not already a previous record on any given theme. If a register already exists, perhaps looking for a collaboration to develop a better record would be wiser than creating a duplicate study. Another factor to consider is that the urgency of well‐designed studies and high‐quality data to provide high scientific evidence is not rapidly provided ² , ¹⁵ , ¹⁶ ; thus, it is understandable and necessary that SRs are replicated through time. ¹³

A replicated SR is defined as using the same or very similar methods as a previous SR to determine whether comparable results are obtained or intentionally broadening or narrowing the question addressed in a previous review to check how operationalization of concepts in the previous review influenced the results. ¹³ , ¹⁷ Thus, replication of reviews could be important for evidence syntheses, especially during special times such as the COVID‐19 pandemic. It is important to emphasize, though, that the uncontrolled duplication of SRs at the same time is not categorized as replication, and the present study identified what seems to be more likely an epidemic of redundant reviews on COVID‐19. ¹³ Based on our results, there is evidence that most of the SR records in PROSPERO about COVID‐19 are impossible to be replicated because some information, such as full search strategy and details about outcomes, were poorly reported.

A few limitations of our analyses should be mentioned. First, although GRADE is not a risk of bias tool, many authors cited the use of such tool in their PROSPERO records and, therefore, we have opted to include it in the table about the risk of bias tool used; however, it must be emphasized that those records that reported the use of GRADE misused such tool. The results should be interpreted with caution since we performed a cross‐sectional analysis, and during the year 2020, more and more records are being created on PROSPERO about COVID‐19; hence, the mass production of systematic reviews reported in our study could be even bigger. Our assessment was based on the report of the PROSPERO records, and it is possible that certain studies were performed more rigorously than specified in the record. Finally, we assessed only the first version of each PROSPERO record, and it is possible that such records were or will be modified. The assessment of only the first version of the record aimed to standardize the records and to evaluate the first body of the planned methods of these reviews. In the future, it will be possible to monitor changes that were made during the execution of the systematic review. In this sense, although changes in PROSPERO records are acceptable when justified, they may reflect mistakes on planning or lack of expertise on the analysis. It has been greatly defended that the protocol elaboration is a crucial step in the systematic review process. Thus it should be carefully and adequately planned, registered, and made freely accessible to the scientific community as a reflection of good research practice. ¹⁸ , ¹⁹ Also, it is important to highlight that guidelines such as PRISMA‐P are available and are strongly recommended to enable quality reports of systematic review protocols creation. ²⁰

In conclusion, a massive number of SRs about COVID‐19 have been conducted, whose PROSPERO records present varied characteristics. Furthermore, many of the assessed records were poorly reported and would be difficult, if not impossible, to replicate, but these results should be interpreted with caution. Even considering that pandemic periods demand urgency in evidence compilation, it is mandatory to conduct research with good practices, focusing on quality reporting, adopting the necessary methods to access the knowledge on the theme, and attempting to establish collaborations among research groups rather than duplicate SR studies.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Supporting information

Supporting Information

Dotto L, Kinalski MdA, Machado PS, Pereira GKR, Sarkis‐Onofre R, dos Santos MBF. The mass production of systematic reviews about COVID‐19: An analysis of PROSPERO records. J Evid Based Med. 2021;14:56–64. 10.1111/jebm.12426