Assessing the reporting quality of randomized controlled trials on COVID-19 vaccines: a systematic review

ABSTRACT

This systematic review evaluated the reporting quality of COVID-19 vaccine randomized controlled trials (RCTs). Relevant RCTs published between July 20, 2020 and June 11, 2021 were identified in the PubMed database by two independent reviewers. Study quality was evaluated with the 2010 AND 2001 Consolidated Standards of Reporting Trials (CONSORT) adherence scores. A total of 22 RCTs were included. The median CONSORT adherence score according to the 2010 criteria was 21 (range, 12–25), thus indicating that 75% of the items in more than half of the RCTs had clear reports. Univariate analysis showed that CONSORT adherence scores were not predicted by category; analysis of variance also showed no significant difference between groups. Our results indicated that the overall quality of COVID-19 vaccine RCTs was very good. Current evidence indicates that a variety of COVID-19 vaccines are effective. No RCTs have reported serious adverse effects such as mortality.

Introduction

Evidence-based medicine is fundamentally dependent on the quality of available clinical evidence. The results of randomized controlled trials (RCTs) provide the highest level of primary evidence, and the use of large sample sizes improves the power of statistical tests and reduces the risk of bias. Due to the lack of targeted drugs for COVID-19, many countries began to concurrently develop COVID-19 vaccines in the early stages of the pandemic.¹

Vaccination is considered to be the most effective measure for preventing the further spread of COVID-19.² Vaccines stimulate the body’s immune system to produce antibodies against a specific virus, thus reducing the probability of future infection. Vaccinations prevent 2–3 million deaths from infections annually.³ The effectiveness and breadth of COVID-19 vaccination will be the main determinant of how long the pandemic will last.⁴ The first approved COVID-19 vaccine was produced by Pfizer-BioNTech and has been widely administered in the UK. The need for the rapid development of vaccines to combat the COVID-19 pandemic has necessitated the introduction of temporary regulations to expedite the authorization of their use in humans.⁵ As a result, the risk of side effects (e.g., serious disease, mortality) have only been based on experimental research results from the first three stages of vaccine development; epidemiological research results typically available in the fourth stage are currently lacking.⁶

By April 2020, approximately 100 different COVID-19 vaccines had been developed by research and development departments in different countries all over the world, with some having proceeded to the human trial stage.⁷ If sufficient protection can be obtained after the first vaccine dose, the second dose can be delayed; this would ensure that a greater number of people in regions with limited access to vaccines can receive the first dose.⁸ Recently, an increasing number of RCTs have investigated the effectiveness of COVID-19 vaccines by comparing infection rates between vaccinated (experimental group) and unvaccinated (control group) individuals.⁹ The purpose of this systematic review was to evaluate the quality of these RCTs and to summarize the effectiveness and adverse effects of currently available COVID-19 vaccines. The overarching aim was to provide a frame of reference to facilitate vaccination selection.

Materials and methods

Search strategy

This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines that are in the supplemental materials. Relevant articles were identified by using the search terms “vaccines” and “COVID 19” in the PubMed database. The authors of the present review were not involved in the conduct of any previous RCTs pertaining to this topic.

Scope of the literature search

A comprehensive search was conducted in PubMed using the following search terms: (“COVID 19 vaccines”[MeSH Terms] OR (“COVID 19”[All Fields] AND ”vaccines”[All Fields]) OR “COVID 19 vaccines”[All Fields] OR (“COVID19”[All Fields] AND “vaccine”[All Fields]) OR “COVID19 vaccine”[All Fields]) AND (randomized controlled trial [Filter]). Only studies published in English were included. All identified RCTs pertaining to COVID-19 vaccines were published between July 20, 2020 and June 11, 2021 (Figure 1).

Figure 1.

A flowchart of the screening process for randomized controlled trials (RCT) articles of the COVID-19 vaccine.

Reporting quality assessment

All extracted data were independently compiled by two reviewers. The reporting quality of each study was evaluated using the 19-item 2001 Consolidated Standards of Reporting Trials (CONSORT) statement (Supplemental Table S1) and the 28-item 2010 CONSORT standardized evaluation checklist (Table 1). The overall report quality score was referred to as the CONSORT adherence score.

Table 1.

Overall quality of reporting: rating using items based on the 2010 CONSORT statement (n = 22)

Item	Criteria	Description	No. of positive trials	%	Cohen’s k coefficient
1	Title	Identification as a randomized trial in the title	13	59	0.91
2	Abstract structure	Structured summary of trial design, methods, results and conclusions	21	95	1
3	Background	Adequate description of the scientific background and explanation of rational	22	100	NA*
4	Objectives	Description of the specific objectives or the scientific hypotheses in the introduction	21	95	0.89
5	Trial design	Description of trial design, including allocation ratio	20	91	0.81
6	Participants	Description of the eligibility criteria for participants	20	91	0.83
7	Settings and location	Description of the settings and locations where the data were collected	10	45	1
8	Interventions	Details of the interventions intended for each group	20	91	1
9	Outcomes	Definition of primary and secondary outcome measures, including how and when they were assessed	18	82	0.94
10	Sample size	Description of sample size calculation	15	68	0.92
11	Randomization, sequence generation	Definition of the method used to generate the random allocation sequence	15	68	0.95
12	Randomization, restriction	Description of the type of randomization details of any restriction	12	55	0.79
13	Allocation concealment	Description of the mechanism used to implement the random allocation sequence to assure concealment until interventions were assigned	18	82	1
14	Implementation	Description of who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions	9	41	0.85
15	Blinding	Whether or not participants, those administering the interventions, or those assessing the outcomes were blinded to group assignment; if relevant, description of the similarity of interventions	18	82	0.88
16	Statistical methods	Description of the statistical methods used to compare groups for primary and secondary outcomes	18	82	0.98
17	Ancillary analysis, method	Description of the methods for additional analyses, such as subgroup analyses and adjusted analyses	5	23	0.83
18	Diagram	A CONSORT diagram was presented to show the flow of participants	19	86	0.86
19	Participant flow	Details on the flow of participants through each stage of the trials (number of patients randomly assigned, receiving intended treatment, and were analyzed for the primary outcome)	13	59	0.96
20	Recruitment	Dates defining the periods of recruitment and follow-up	15	68	1
21	Baseline data	A table showing baseline demographic and clinical characteristics for each group	22	100	NA*
22	Intent-to-treat analysis	Number of patients in each group included in each analysis and whether patients were analyzed according to the group to which they were randomly assigned	4	18	0.77
23	Outcomes measures	For each primary and secondary outcome, a summary of results for each group, the estimated effect size and its precision (eg, 95% CI) are provided	12	55	0.87
24	Ancillary analyses	Results of subgroup analyses and adjusted analyses, distinguishing prespecified from exploratory	7	32	1
25	Adverse event classification	Description of all important adverse events in each group, with classification	20	91	0.74
26	Registration	Presentation of the registration number and name of trial registry	22	100	NA*
27	Protocol	Where the full trial protocol can be accessed	20	91	0.73
28	Funding	Sources of funding and other support	19	86	0.86

Abbreviations: CI: confidence interval; CONSORT: Consolidated Standards of Reporting Trials; NA: not available.

* Cohen’s k indices could not be calculated because the positive rates awarded by the 2 investigators were both 100% for these items.

Research selection and data extraction

The inclusion criteria comprised the following: (a) evaluation of a COVID-19 vaccine using a randomized controlled design; (b) use of a COVID-19 vaccine in the experimental group; and (c) articles published in English. Studies were excluded if they did not report safety or effectiveness data, or were duplicate publications or secondary reports of previously published RCTs. If the results of a single RCT were reported in multiple publications, the one with the most complete data was selected.

The difference between the level 1 screening (titles and abstracts), two reviewers were resolved through discussion.

Data collection

Two independent reviewers extracted the following data: first author name; year of publication; whether or not the term “RCT” was used in the study title; use of a structured or non-structured abstract; experimental design and allocation ratio to the intervention and control groups; specific content recorded in the article or protocol; study setting; place of the data collection; drug information; primary and secondary outcomes; measurement information; methods used for sample size calculation and randomization; allocation concealment; blinding method; whether or not an intent-to-treat (ITT) analysis and subgroup analysis were performed; flowchart; recruitment and follow-up time; results for vaccine efficacy; experimental registration number; and source of funding. Any discrepancies were resolved by consensus between the two reviewers.

Statistical analysis

The main purpose of this study was to assess the quality of RCTs that have evaluated COVID-19 effectiveness and safety. Using CONSORT criteria, we assigned 1 point for each criterion and calculated the total score of each item. SPSS Statistics 25 was used to analyze the collected data, and descriptive statistics were used to calculate the median and mean. The linear regression coefficient generated by the CONSORT adherence scores was used as the dependent variable to obtain the regression coefficient and P value. The difference between the groups and whether the classification could predict the dependent variable were evaluated.

Results

As shown in the flow chart in Figure 1, a total of 11053 articles were retrieved from PubMed. Title and abstract screening excluded 10999 non-RCTs. Full texts of the remaining 54 studies were evaluated according to our predefined inclusion and exclusion criteria. Thirty-two studies were excluded, as they were either secondary reports of previous RCTs (29 articles) or duplicate studies (3 articles). Thus, a total of 22 studies were included in our analysis (Table 2)^10–31 and their characteristics are summarized in Table 3; data are expressed as absolute counts and proportions.

Table 2.

The basic characteristics of clinical experiments

NO.	Author	Year	Score	Region in which trials were conducted	Journal	Blinding	Type of vaccine	Sample size	Phase	Endpoints
1	Peter Richmond	2021	21	Australia	Lancet	triple blind	SCB-2019	148	phase 1	Safety (adverse reactions) and immunogenicity
2	Laurence Chu	2021	24	US	Vaccine	triple blind	mRNA-1273	600	phase 2	Safety (adverse reactions) and immunogenicity
3	Kathryn E. Stephenson	2021	20	USA	JAMA	triple blind	Ad26.COV2.S	25	phase 1	immunogenicity
4	Hong-Xing Pan	2021	23	China	Chinese Medical Journal	double blind or double-masked	KCONVAC	560	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
5	L.R. Baden	2021	23	USA	The new england journal of medicine	triple blind	mRNA-1273	30420	phase 3	effectiveness of the vaccine
6	V.Shinde	2021	21	South Africa	The new england journal of medicine	triple blind	NVX-CoV2373	4406	phase 2	Safety (adverse reactions) and effectiveness of the vaccine
7	Katherine R W Emary	2021	12	UK	Lancet	unblinded or open label	ChAdOx1nCoV-19(AZD1222)	520	Phase 2 and phase 3	effectiveness of the vaccine
8	J.Sadoff	2021	19	Belgium and the United States	The new england journal of medicine	double blind or double-masked	Ad26.COV2.S	810	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
9	Pedro M Folegatti	2020	24	UK	Lancet	double blind or double-masked	ChAdOx1nCoV-19	1077	Phase 1 and phase 2	Safety (adverse reactions), immunogenicity and effectiveness of the vaccine
10	Denis Y Logunov	2021	24	Russia	Lancet	double blind or double-masked	Gam-COVID-Vac	21 977	phase 3	Safety (adverse reactions), immunogenicity and effectiveness of the vaccine
11	Yanjun Zhang	2021	22	China.	Lancet Infect Dis	double blind or double-masked	CoronaVac	744	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
12	Mark J. Mulligan	2020	15	USA	Nature	single blind or single-masked	BNT162b1	45	Phase 1 and phase 2	Not included
13	Maheshi N Ramasamy	2020	21	UK	The Lancet	single blind or single-masked	ChAdOx1 nCoV-19	560	phase 2	Safety (adverse reactions), immunogenicity and effectiveness of the vaccine
14	Edward E. Walsh	2020	16	the United States	The new england journal of medicine	unblinded or open label	BNT162b1	195	phase 1	Safety (adverse reactions) and immunogenicity
15	Shengli Xia	2020	21	China	JAMA	double blind or double-masked	inactivated COVID-19 vaccine	320	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
16	Fernando P. Polack	2020	20	United States, Argentina, Brazil, South Africa, Germany and Turkey,	The new england journal of medicine	unblinded or open label	BNT162b2	43548	Phase 2 and phase 3	Safety (adverse reactions) and effectiveness of the vaccine
17	Shengli Xia	2020	22	China	Lancet Infect Dis	triple blind	BBIBP-COV	640	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
18	C. Keech	2020	17	Australia	The new england journal of medicine	unblinded or open label	NVX-CoV2373	131	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
19	S.A. Madhi	2021	18	South Africa	The new england journal of medicine	double blind or double-masked	ChAdOx1 nCoV-19	2026	Phase 1 and phase 2	effectiveness of the vaccine
20	Jing Pu	2021	16	China	Vaccine	double blind or double-masked	inactivated SARS-CoV-2	192	phase 1	Safety (adverse reactions) and immunogenicity
21	Feng-Cai Zhu	2020	25	China	The Lancet	triple blind	Ad5-vectored COVID-19	508	phase 2	Safety (adverse reactions) and immunogenicity
22	Raches Ella	2021	24	India	Lancet Infect Dis	triple blind	BBV152	375	phase 1	Safety (adverse reactions) and immunogenicity

Table 3.

Trial characteristics

Characteristic	No. of studies (n = 22)	%
Year of publication
2020	9	41
2021	13	59
Region in which trials were conducted
Asia	7	32
Europe and North America	10	45
Others	5	23
Journal
The new england journal of medicine	7	32
The Lancet	6	27
Journal of the American Medical Association	2	9
Lancet Infectious Diseases	3	14
Vaccine	2	9
Nature	1	5
Chinese Medical Journal	1	5
Journal impact factor
<30	6	27
30–80	9	41
>80	7	32
Sample size
Median(range)	560(25–43548)
<500	8	36
500–1000	8	36
>1000	6	27
Sources of trial funding
Government/foundation	11	50
Completely funded by industry	4	18
Partially funded by industry	7	32
Type of vaccine
inactivated vaccine	6	27
Nucleic acid vaccine (DNA, mRNA)	7	32
Adenovirus vector vaccine	8	36
Protein subunit vaccine	1	5
Phase
1	14	43
2	15	45
3	4	12

The majority of the RCTs were conducted in countries in Europe and North America. The impact factor of most of the journals in which the studies were published (73%, n = 16) exceeded 30. Over half (63%, n = 14) of the studies included more than 500 participants. Most of the studies were either phase I or II vaccine trials; only 12% (n = 4) were phase III trials.

Inter-rater agreement for the 2010 CONSORT standardized evaluation checklist were classified via Cohen’s κ statistic as substantial, good, or perfect (Table 1). CONSORT adherence scores ranged from 0 to 28. As the reported reference median CONSORT adherence score was 21 (range, 12–25), this indicated that 75% of the items in more than half of the RCTs in the present review had clear reports. The results of the descriptive analysis showed that less than 10% of the studies had a score of <14, thus indicating that study quality was “very good” (Table 1). The results showed the positive number of CONSORT in the frequency distribution diagram (Figure 2). All studies provided detailed scientific background information and reported baseline participant characteristics in both the experimental and control groups. Clinical outcomes after vaccination were summarized and presented in the form of tables. Since univariate analysis did not show significant differences between categories, CONSORT adherence scores could not be predicted according to category. Analysis of variance showed that there was no significant difference between the groups (Table 4).

Figure 2.

Percentage of literature that meets the 28-item 2010 Consolidated Standards of Reporting Trials (CONSORT) standardized evaluation checklist.

Table 4.

Publication characteristics associated with 2010 overall reporting quality

Publication characteristic	Mean CONSORT adherence scores (95% CI)	Estimate (95% CI)*	p
Year of publication
2020	20.3 (17.44, 22.79)	Reference	0.783
2021	20.7 (18.38, 22.70)	−0.17 (−0.99, 0.65)
Region in which trials were conducted
Asia	22.4 (19.16, 24.55)	Reference	0.351
Europe and North America	20 (16.95, 23.05)	0.61 (−0.11, 1.32)
Others	19 (17.04, 20.96)	−0.33 (−2.48, 1.81)
Journal impact factor
<30	22.5 (18.69, 24.98)	Reference	0.395
30–80	20.3 (17.03, 23.64)	−0.28 (−1.29, 0.73)
>80	19.1 (16.91, 21.37)	0.07 (−1.84, 1.99)
Sample size
<500	18.8 (16.08, 21.42)	Reference	0.248
500–1000	21.5 (17.60, 24.40)	0.54 (−0.18, 1.27)
>1000.	21.7 (19.12, 24.21)	0.14 (−1.16, 1.44)
Sources of trial funding
Government/foundation	21.5 (18.75, 23.43)	Reference	0.58
Completely funded by industry	20.3 (14.99, 25.51)	0.14 (−0.30, 0.58)
Partially funded by industry	19.3 (15.84, 22.73)	0.08 (−0.27, 0.44)

Abbreviation: CONSORT adherence scores rated on a scale of 0 to 28. *The estimates indicate the benefit observed compared with the reference. Any positive value indicates incremental benefit compared with the reference, whereas any negative value indicates detriment compared with the reference.

The median CONSORT adherence score according to the 2001 CONSORT statement criteria was 16 (range, 7–19); this indicated that 84% of the RCTs had clear reports. The results of the descriptive analysis showed that less than 10% of the RCTs had a score of <9. This reflected a “very good” study quality and was consistent with the results obtained using the 2010 CONSORT standardized evaluation checklist (Supplementary Table S1).

We evaluated adverse event report scores based on the rating of the hazardous recommendations. Our analysis indicated that inactivated vaccines, nucleic acid vaccines, adenovirus vector vaccines, protein subunit vaccines, and other types of COVID-19 vaccines had good efficacy. These vaccines were also safe, as no serious adverse reactions such as death were reported (Supplementary Table S2).

The use of allocation concealment and blinding across the included studies is shown in Supplementary Table S3. Over half (59%, n = 13) of the studies used a centralized randomization method, and 82% (n = 18) used the blind method. Only 27% (n = 6) of the studies performed an ITT analysis. Studies that did not report the use of ITT were assumed to have not used this analysis method. Supplementary Table S4 summarizes the endpoints used in the 22 trials. Most trials ended with adverse reactions or immunogenicity; safety and adverse reactions were the most commonly reported outcomes (77%, n = 17), and the vaccine effectiveness was only 36% (n = 8). The results of Supplementary Table S5 are similar to those of the univariate analysis of CONSORT 2010, which showed that there was no significant difference between categories. Therefore, CONSORT adherence scores were not predicted by category.

Discussion

The results of our review indicate that the quality of RCTs on COVID-19 vaccines was not affected by the specific stage of vaccine development that was under investigation. Indeed, the CONSORT adherence scores indicated that the reporting quality of these RCTs was very good. This finding is pertinent for governments worldwide, as they are responsible for the majority of funding for vaccine research and development.

Vaccines are one of the most effective and safest means for preventing the further spread of COVID-19.³² A number of different factors may affect the quality of research reports. For example, study quality is often significantly associated with the type of funding source. Journals with more published papers have higher impact factors, which are often associated with increased study quality. Studies in such journals are more likely to have a large sample size and include a wide range of age groups, from adolescents to the elderly.

At present, the incidence of new COVID-19 cases has not plateaued in many countries. In addition, some countries have even reported mutated variants with increased transmissibility. The emergence of COVID-19 variants indicates that a second vaccine dose is necessary, as previous studies have found that vaccine effectiveness after the first dose decreases after a period of time.³³ A third dose can further maintain effectiveness over the long-term and should be considered in countries where the proportion of the population with both first and second doses has reached a certain threshold.³⁴ Different vaccine types can be selected by countries according to their actual situation.³⁵ The results of the present review indicate that the majority of the investigated vaccine types are very effective. From the conclusion that there is no significant difference in univariate analysis, it can be seen that the literature quality of different categories with different characteristics is similar. It can be concluded that the quality of these RCTs is very good; the reporting was very specific and detailed, regardless of journal impact factor, funding source, region in which trials were conducted, and sample size.

Some RCTs did not provide details regarding the random allocation of study participants, as well as whether allocation concealment was performed. Some RCTs did not report whether researchers or patients were blinded to treatment allocation. The current stage of COVID-19 vaccine research has mainly focused on outcomes pertaining to adverse reactions, immunogenicity, and vaccine effectiveness; the latter outcome has been limited by the inability to mass produce experimental vaccines for evaluation in clinical trials. Nevertheless, a plethora of studies on COVID-19 vaccines are planned or in progress, and their results will provide important data on actual vaccine effectiveness. To date, the completed RCTs on COVID-19 vaccines have been of very good quality; this may be attributed to the individual efforts of research personnel, as well as the large amount of invested human, material, and financial resources. High-quality RCTs not only provide a greater reference value for future studies, but also contribute more to the global efforts to combat the COVID-19 pandemic. This is pertinent, as greater challenges for vaccine development are expected with the continuous emergence of COVID-19 variants. Long-term studies are required to determine whether existing vaccines can effectively and safely prevent infection by different variant strains.^36,37

Nevertheless, we found that some studies omitted certain CONSORT checklist items, such as declaring that the study was a RCT in the title; this criterion was only satisfied in 59% of the included studies. The implementation of randomization was only described in 41% of the studies. Thus, the majority of studies did not adhere to the principles of randomization; alternatively, they may have followed these principles but failed to report it. This resulted in a reduced study quality to some degree.

Some limitations are acknowledged in the present review. For example, we did not perform a detailed analysis of study follow-up duration and specific types of adverse reactions. Furthermore, the included studies did not provide detailed data on participant race, sex, age, or other differences. In addition, as the univariate analyses did not yield statistically significant associations with CONSORT adherence scores, we were unable to conduct a multivariate analysis to adjust for confounding factors.

In conclusion, based on the use of the CONSORT criteria, we determined that the RCTs on COVID-19 vaccines that have been published to date are of very high quality. This may be attributed to not only the adherence of study authors to established research reporting guidelines, but also the strict evaluation of manuscripts by referees during the peer review stage.

Funding Statement

This work was supported by the National Natural Science Foundation of China under Grant [81672861] and the Science and Technology Development Plan of Shandong Province under Grant [2017GSF218029].

Disclosure statement

No potential conflict of interest was reported by the author(s). 

Data availability statement

Data are available upon reasonable request.

Supplemental data

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/21645515.2022.2031453