Methodological Analysis: Randomized Controlled Trials for Pfizer and Moderna COVID-19 Vaccines

Abstract

Critical appraisal of randomized controlled trials (RCTs) determines rigor, quality, and whether the findings are applicable to the populations served in clinical practices. The authors conducted a rigorous analysis using the RCT Critical Appraisal Skills Programme (CASP) Checklist for the two RCTs Pfizer (New York, NY) and Moderna (Cambridge, MA) conducted and the reporting of these RCTs using the Consolidated Standards of Reporting Trials checklist. The goals for this analysis were twofold: (1) enable health care providers to understand the methods and outcomes of these RCTs, and (2) enable health care providers and community leaders to become champions for the vaccines to reduce vaccine hesitancy among all populations. The analysis is presented using each of the 11 questions on the CASP tool while comparing the methodology and results for each vaccine. Most CASP tool items were positive or yes for both the Pfizer and Moderna RCTs. Items that were not scored as yes are discussed. The analysis outcomes revealed that both RCTs were rigorously conducted and provide an assurance to all health care providers and the public of the safety and efficacy of both vaccines to impact the astounding morbidity and mortality of COVID-19 disease. The authors believed that the analysis was an essential component of the distribution process to develop plans and communication strategies to reduce potential vaccine hesitancy and resistance.

INTRODUCTION

The “warp speed” concept for the development, testing, clinical trials, and the Food and Drug Administration (FDA) emergency use authorization of two COVID-19 vaccines and the nationwide distribution of the vaccines—“shots in arms”—to combat the severe acute respiratory coronavirus 2 (SARS-CoV-2) pandemic has brought to the forefront issues related to vaccine hesitancy. One way to address vaccine hesitancy is to critically analyze the methodology and results for the randomized controlled trials (RCTs) conducted for both FDA emergency use authorization COVID-19 vaccines: Pfizer (New York, NY; Polack et al., 2020) and Moderna (Cambridge, MA; Baden et al., 2021) and for providers to inform their patients of the outcomes of these RCTs.

Critical appraisal of RCTs determines rigor, quality, and whether the findings are applicable to the populations served in clinical practices. Specifically, the Critical Appraisal Skills Programme (CASP) Checklists (Critical Appraisal Skills Programme, 2018) includes an appraisal tool designed to evaluate RCTs. In addition, research journals require RCTs to be reported using the Consolidated Standards of Reporting Trials (Consolidated Standards of Reporting Trials, 2010) guidelines. Thus, we conducted a rigorous analysis using the RCT CASP tools for the two RCTs, Pfizer (Polack et al., 2020) and Moderna (Baden et al., 2021), and the reporting of these RCTs using the Consolidated Standards of Reporting Trials (CONSORT) checklist. The authors present the findings from this analysis which revealed that both the Pfizer and Moderna vaccine demonstrated safety and efficacy. The goals for this analysis are twofold: (1) enable health care providers to understand the methods and outcomes of these RCTs, and (2) enable health care providers and community leaders to become champions for the vaccines to reduce vaccine hesitancy among all populations.

General Information: COVID-19 Vaccines

Both the Pfizer and Moderna vaccines are messenger ribonucleic acid (mRNA) vaccines. The mRNA-based vaccines have advantages in that they do not generate infectious components or have the potential to cause infection, do not affect the genes or genome of the person or host cell, do generate a strong immune response with only one or two low-doses of vaccine, and can be quickly produced in a large-scale to treat mass populations (Maruggi, Zhang, Li, Ulmer, & Yu, 2019; Wang, Kream, & Stefano, 2020). Once the gene sequence and antigen of the virus pathogen have been determined, the mRNA vaccine can be rapidly produced. In addition, the success of mRNA vaccines for cancer and viral pathogens in creating antigen-specific immune responses has been studied and documented since the 1990s (Maruggi et al., 2019).

The BNT162b2 mRNA vaccine from Pfizer is a lipid nanoparticle-formulated vaccine that encodes a prefusion stabilized full-length spike protein of the virus that causes COVID-19, SARS-CoV-2 (Polack et al., 2020). The mRNA-1273 vaccine from Moderna also encodes the full-length spike protein and is similar in composition to the Pfizer vaccine (Baden et al., 2021). Once the vaccine is injected into the person or host, the mRNA protein is processed by immune cells that produce the targeted protein directly (Wang et al., 2020). This replication of proteins subsequently activates B cells and T cells to recognize the newly produced viral protein and make antibodies against SARS-CoV-2 (Wang et al., 2020).

Methodology for RCT

In appraising the phase three RCTs of the vaccines, the CASP checklist for RCTs was used systematically for both the Pfizer and Moderna vaccine articles. The checklist for this critical appraisal tool comprises 11 questions designed with prompts to explore details of the study, focusing on the validity of results, the details of the results, and whether the results are helpful (Critical Appraisal Skills Programme, 2018). We present our analysis using each question on the CASP tool while comparing the methodology and results for each vaccine.

Definitions of Efficacy Versus Effectiveness

Two terms, efficacy and effectiveness, are used to assess health care interventions (Haynes, 1999). Efficacy is defined as “the extent to which an intervention does more good than harm under ideal circumstances” (Haynes, 1999, p. 1). Effectiveness is defined as a means of “assessing whether an intervention does more good than harm when provided under usual circumstances of health care practice” (Haynes, 1999, p. 1). Both RCTs report on the efficacy of the vaccines in preventing severe SARS-CoV-2 disease in individuals who were in RCTs to assess the efficacy of the vaccines. The effectiveness of the vaccines will be determined when data become available when a large number of individuals receive the vaccines in populations in the United States and throughout the world. Pfizer and Moderna used different statistical methodologies to determine vaccine efficacy.

WHAT IS VACCINE EFFICACY, AND WHAT DO THE REPORTED PROBABILITIES MEAN?

Pfizer

Vaccine efficacy (VE) describes the impact of the vaccine on the primary endpoint of infection. VE is often misinterpreted as the percentage of vaccinated individuals who will not become infected when exposed to the virus. What VE does reflect is the probability of infection in the treatment group as compared with the placebo group. Specifically, it is calculated as 1 − (p _treatment/p _control), where p _treatment = the probability or rate of infection in the treatment group, and p _control = the probability of infection in the control group. VE can range from 0% (vaccine offers no protection from infection) to 100% (complete protection). In the study sample, it was found that the rate of infection in the vaccination group was 4.9% that of the placebo group, and so VE is determined to be 95%.

Because these results are based on sample data (and not the entire population), we need to determine the likelihood of this result of the VE in the larger population. This is done using the null hypothesis that VE is ≤ 30% (prespecified criterion). The Bayesian beta-binomial model was used to calculate the likelihood that the true VE in the population exceeds this threshold (> 30%) on the basis of the effect observed in the sample (Yang, Gilbert, Longini, & Halloran, 2008). This is the 99.9% probability that is reported. That is, given the findings in the sample (VE = 95%), there is a 99.9% likelihood that the true effect of the vaccine (VE) in the population exceeds 30%. The Bayesian methods for calculating the probability of VE are standard methods and provide assurance of VE.

Moderna

Moderna used different statistical methods to assess VE, using a stratified Cox proportional hazards model to determine the percentage of hazard reduction associated with the vaccine relative to the placebo. However, the statistical methods achieved comparable outcomes for the analysis of VE at 94.1% as the Pfizer RCT. VE was assessed for all participants who received at least one dose of the Moderna vaccine on the basis of a modified intention-to-treat based on the population who received the vaccine without evidence of COVID-19 disease on day 1 before the first vaccine was administered and participants who received two doses of the vaccine without a deviation from the protocol (Baden et al., 2021). This statistical methodology also provides assurance of VE.

QUESTION ONE: DID THE TRIAL ADDRESS A CLEARLY FOCUSED ISSUE?

Pfizer

Yes, the Pfizer RCT focused on the “safety and efficacy findings from the phase 2/3 part of a global phase 1/2/3 trial evaluating the safety, immunogenicity, and efficacy of 30 ug of BNT162b2 in preventing COVID-19 in persons 16-years of age and older” (Polack et al., 2020, p. 2). Individuals admitted to the clinical trial were healthy or had stable chronic medical conditions. Individuals excluded from the study were those who had a medical history of having COVID-19 disease, treatment with immunosuppressive therapy, or diagnosis with an immunocompromising condition. The primary endpoints for Pfizer RCT were data solicited from an electronic diary from study participants concerning local or systemic adverse events and the use of antipyretics within 7 days of receiving the vaccine. The major secondary endpoint for the Pfizer vaccine was the efficacy of the vaccine to protect against severe COVID-19 disease. The intervention of the Pfizer vaccine BNT162b2, an mRNA vaccine, was administered in a two-dose regimen of 30 µg per dose, with 0.3 mL per dose, administered in the deltoid, 21 days apart. The placebo control group received 0.3 mL of saline per dose, also administered in the deltoid, 21 days apart. All study participants were observed for 30 min after receiving each dose.

Moderna

Yes, the Moderna RCT addressed a critical and timely issue of SARS-CoV-2 infection, a worldwide pandemic with devastating mortality and morbidity and an extremely high level of contagiousness. This RCT studied a population of adults stratified on the basis of age and COVID-19 complications risk criteria with groups of people aged ≥ 65 years, aged < 65 years who were at risk for severe COVID-19, and < 65 years without risk. Participants were categorized as being at risk for severe COVID-19 if they had at least one of the following on the basis of the Centers for Disease Control and Prevention criteria at the time of trial design: chronic lung disease, cardiac disease, severe obesity, diabetes, liver disease, or HIV infection. The intervention was the mRNA-1273 vaccine or the saline placebo control. The vaccine was provided as a sterile liquid at a concentration of 0.2 mg/mL in a volume of 0.5 mL containing 100 μg of mRNA-1273, and the placebo control was 0.5 mL of saline. Either was administered by intramuscular injection into the deltoid muscle in a two-dose regimen given 28 days apart in the same arm. Safety outcomes included adverse events and cases of COVID-19 and severe COVID-19. Moderna specified efficacy outcomes as primary and secondary endpoints, with the primary endpoint being efficacy of the vaccine in preventing symptomatic infection at least 14 days after the second injection of vaccine. The secondary endpoints were efficacy in preventing infection with severe symptoms, the efficacy of a single dose, having any symptoms of infection, and having a positive test. Thus, efficacy included not only preventing infection but also preventing symptomatic or severe infections.

QUESTION TWO: WAS THE ASSIGNMENT OF PATIENTS TO TREATMENTS/INTERVENTIONS RANDOMIZED?

Pfizer

Yes, all study participants, who were screened to meet study eligibility criteria, completed information in an interactive Web-based system used to randomly assign study participants, aged ≥ 16 years, in a 1:1 ratio, to the placebo-controlled or vaccine (BNT162b) intervention groups.

Moderna

Yes, the participants were randomized in a one-to-one ratio to receive vaccine or placebo control, using a centralized interactive response technology system that was blinded to study staff and those receiving the vaccine.

QUESTION THREE: WERE ALL THE PATIENTS WHO ENTERED THE TRIAL PROPERLY ACCOUNTED FOR AT ITS CONCLUSION?

Pfizer

No, all study participants in the vaccine intervention group or the placebo control group were accounted for in a detailed flow diagram and in the narrative that detailed the characteristics of study participants. However, unique to the Pfizer RCT was that the study was multinational, with 152 participating sites worldwide. The demographic characteristics of the participant table (Table 1 in the article) reported data for participants from Argentina, Brazil, South Africa, and the United States. However, the VE table (Table 3 in the article) only reported data from Argentina, Brazil, and the United States. Therefore, it is not clear if participants from South Africa were accounted for in the study. In addition, 196 patients with a diagnosis of HIV positive participated in the study, but their data were to be analyzed separately and was not included in this article.

Moderna

Yes, all participants who were either intervention with vaccine or control with placebo control are detailed in a flow diagram as directed by the CONSORT guidelines and detailed in the narrative of the article. The participants were analyzed in the groups on which they were analyzed and in subgroups of age, age and health risk, sex, race and ethnicity, and risk for severe COVID-19 illness. The RCT was not stopped early.

QUESTION FOUR: WERE PATIENTS, HEALTH WORKERS, AND STUDY PERSONNEL BLIND TO TREATMENT/INTERVENTION?

Pfizer

Yes, patients, health workers, and study personnel were blinded to study participant placebo control and vaccine intervention group assignments. In addition, site staff who were responsible for safety evaluations by observing participants for 30 min postvaccination did not have any knowledge about group assignments.

Moderna

No, study staff and those receiving the vaccine were blind to treatment/intervention and placebo control; however, vaccine doses were prepared and administered by pharmacists and vaccine administrators who were aware of treatment/intervention and placebo control assignments. These pharmacists and vaccine administrators had no other role in the RCT.

QUESTION FIVE: WERE THE GROUPS SIMILAR AT THE START OF THE TRIAL?

Pfizer

Yes, the demographic characteristics of sex, race or ethnic groups, country, age entering the study, age at vaccination, and body mass index, and coexisting conditions were similar for the vaccine intervention and placebo-controlled groups at the start of the trial.

Moderna

Yes, the groups of analysis in the intervention and control of the RCT were similar. In addition, the selection of study sites and enrollment were adjusted to increase the number of persons from racial and ethnic groups in the RCT, resulting in a population described as generally representative of demographics in the United States.

QUESTION SIX: ASIDE FROM THE EXPERIMENTAL INTERVENTION, WERE THE GROUPS TREATED EQUALLY?

Pfizer

There are two answers to this question. Yes, on the days of their injection, all participants were monitored for 30 min after receiving the injection. Participants in both the vaccine intervention and the placebo-controlled groups were prompted by an electronic diary to record specific local or systemic adverse events and use of antipyretics or pain medication within 7 days after receipt of each dose per their assigned groups. All participants could also record in the electronic diary without prompting any adverse event through 1-month after the second dose and unsolicited serious adverse events through 6 months after the second dose. In addition, we cannot tell, as no information was provided regarding instructions to participants; for example, were all participants instructed to wear masks, keep social distance, and wash hands frequently? What was the protocol for testing all participants for COVID-19 infection before and after receiving the vaccine? For those who were positive after receiving the vaccine, was contact tracing completed?

Moderna

Yes, participants were monitored for solicited local and systemic adverse events (7 days after each injection), unsolicited adverse reactions (28 days after each injection), adverse events leading to discontinuation from a dose and/or RCT, serious adverse events (1–759 days and cases of COVID-19 and severe COVID-19 continuously from randomization onward), regardless of intervention or placebo control. As previously noted, injections of 0.5 mL were given 28 days apart in the same arm.

QUESTION SEVEN: HOW LARGE WAS THE TREATMENT/INTERVENTION EFFECT?

Pfizer

Based on the evidence from the RCT, the Pfizer vaccine was 52% effective in the interval between the first and second dose and 91% effective in the first 7 days after dose 2. Full efficacy of 95% protection against COVID-19 in persons aged ≥ 16 years after dose 2 for those in the vaccine intervention group (credible interval, 90.3–97.6).

Moderna

Based on evidence from the RCT, the Moderna vaccine was 94.1% effective at preventing laboratory-confirmed symptomatic COVID-19 illness. For the primary endpoint, 196 cases of COVID-19 were diagnosed with 11 cases in the vaccine intervention group (3.3 per 1,000 person-years; 95% confidence interval [CI], 1.7–6.0) and 185 cases in the placebo control group (56.5 per 1,000 person-years; 95% CI, 48.7–65.3). For the secondary endpoint, 30 participants in the RCT had severe COVID-19, and all 30 were in the placebo control group, indicating a VE of 100% (95% CI, could not be estimated to 1.0).

QUESTION EIGHT: HOW PRECISE WAS THE ESTIMATE OF THE TREATMENT/INTERVENTION EFFECT?

Pfizer

The researchers reported eight cases of COVID-19 with onset at least 7 days after the second dose among the vaccine intervention group (BNT162b2b) and 162 cases among those randomly assigned to the placebo-controlled group. Ten cases of severe COVID-19 were reported after the first dose was administered to both the vaccine intervention and the placebo-controlled group: one case occurred in the intervention group and nine cases in the placebo control group.

Moderna

In addition to the effects of the intervention noted above, participants assessed 14 days after the first dose of vaccine for the presence of SARS-CoV-2 infection, noting 225 cases with placebo control and 11 cases with the vaccine, indicating a VE of 95.2% (95% CI, 91.2–97.4). Participants who were SARS-CoV-2 seropositive at baseline were also included per-protocol in the analysis (187 cases with placebo control, 12 cases with cases; one participant assigned to receive vaccine was inadvertently given placebo control, indicating a VE of 93.6% [95% CI, 88.6–96.5]). These data conclude a precise treatment effect among varying analyses of the RCT.

QUESTION NINE: CAN THE RESULTS BE APPLIED TO THE LOCAL POPULATION OR IN YOUR CONTEXT?

Pfizer

Yes, study participants were similar to those who would receive the vaccine in the United States and to those study sites in Argentina and Brazil as efficacy data are reported for those countries.

Moderna

Yes, the participants in the trial were similar enough to those who would be administered the vaccine, and the results can be applied to a variety of populations, on the basis of age, healthy or with a health risk or chronic condition, race and ethnicity, and sex.

QUESTION 10: WERE ALL CLINICALLY IMPORTANT OUTCOMES CONSIDERED?

Pfizer

No, it was not possible to consider all clinically important outcomes given the “warp speed” conduction of the RCT, which was essential as the world is living in a pandemic. The researchers are planning to continue follow-up of study participants for the next 2 years. At this time, there is a severe shortage of available vaccines, and there is consideration of administering the single dose, which showed a 52% efficacy rate 7 days after the initial dose.

Moderna

No, all clinically important outcomes were not considered as this RCT was of short duration because of the current COVID-19 pandemic. Other outcomes to consider are efficacy of single dose of vaccine, reduced or half doses of vaccine, the extended interval between two-dose regimen (e.g., > 28 days between doses), and long-term-efficacy of the vaccine and whether booster doses will be needed in the future. However, these outcomes do not affect the decision to proceed with the use of the vaccine to prevent morbidity and mortality in the population at this time, focusing on risk for severe COVID-19 such as age and health risk.

QUESTION 11: ARE THE BENEFITS WORTH THE HARMS AND COSTS?

Pfizer

Yes, the benefits of the vaccine are worth the cost of production and administration. The 95% efficacy rate achieved after dose 2 has the potential to significantly impact the morbidity and mortality resulting from acquiring SARS-CoV-2 infection.

Moderna

Yes, on the basis of the results of the phase 3 RCT, the benefits of the Moderna vaccine far outweigh the costs. The vaccine demonstrated 94% efficacy for the prevention of symptomatic SARS-CoV-2 infection compared with placebo control, measured starting 14 days after the second dose, and 100% efficacy in prevention of severe COVID-19. Safety was demonstrated in the relatively mild, short duration, and frequency of local and systemic adverse events, which were similar to placebo control. The most common local event was pain after injection, which occurred much more commonly in the vaccine intervention group, and the most common systemic events were fatigue and headache, which were similar to the placebo control.

REPORTING OF THE RCTs

As part of the appraisal process, the RCTs for both Pfizer and Moderna articles were evaluated using the CONSORT 2010 checklist of information to include when reporting an RCT. Both RCTs failed to include identification as an RCT in the title. Both articles were evaluated as including all other key information for reporting an RCT; however, we felt it was important to note that both included supplementary materials on the journal Web site.

SUMMARY OF CRITICAL APPRAISAL

We conducted a rigorous analysis of the two RCTs using the CASP checklist (Critical Appraisal Skills Programme, 2018), and we found that both Pfizer and Moderna vaccines demonstrated safety and efficacy. Most checklist items were positive or yes for both Pfizer and Moderna RCTs. For the Pfizer vaccine, all the patients who entered the trial were not properly accounted for at its conclusion (question three). For the Moderna vaccine, study personnel of pharmacists and vaccine administrators were not blind to treatment/intervention (question four). All clinically important outcomes were not considered for both vaccines (question 10); however, this was attributed to the need for a vaccine in a pandemic with significant morbidity and mortality. In addition, we conducted an analysis of the reporting of these RCTs using the CONSORT checklist (Consolidated Standards of Reporting Trials, 2010) and found that both Pfizer and Moderna vaccine RCT articles included the necessary details for reporting an RCT with the exception of not including RCT in the title.

Using the CASP checklist, we noted that the checklist does not account for vaccine-specific RCT concerns such as side effects of the vaccine. We found the side effects for both vaccines to be mostly minor and local or specific to vaccine administration sites. In a critical appraisal of a vaccine or other types of RCT, it may be helpful to focus on questions specific to the RCT.

Of note is the dosing for Moderna at 100 µg per dose of mRNA-123 compared with Pfizer's dosing of 30 µg. Thus, the Moderna vaccine is providing three times the amount per dose as Pfizer with no difference in outcomes. As more is learned about these vaccines as the initial clinical trials are further studied, long-term safety and efficacy both in the laboratory and from continued analysis of outcomes for individuals who participated in the clinical trials, dosage recommendations will be further analyzed. Data continues to be collected and analyzed for the phase 2/3 clinical trial phase to determine vaccine immunogenicity and durability of the immune response. In addition, all individuals who have taken the vaccine in the United States should complete the Centers for Disease Control and Prevention (http://www.cdc.gov) postvaccination monitoring system, Vaccine Adverse Event Reporting System, to assure a continued knowledge base for COVID-19 and vaccine safety, efficacy, and immunogenicity responses.

CONCLUSIONS AND IMPLICATIONS FOR PRACTICE

Nurse practitioner educational programs introduce CASP tools in the research and evidence-based practice courses and have students continue to use the tools throughout clinical courses to enable students to conduct a critical analysis of published research studies with the goal of determining whether the study outcomes can be applied to their patient populations. All practicing clinicians should continue this analysis as new studies emerge, and new treatment modalities become available for implementation into clinical practice. Thus, the authors conducted the analysis of the RCTs published by both Pfizer and Moderna after receiving emergency use authorization for distribution and administration of vaccines by the FDA. The authors believed that the analysis was an essential component of the distribution process to develop plans and communication strategies to reduce potential vaccine hesitancy and resistance. The analysis outcomes revealed that both RCTs were rigorously conducted and provide an assurance to all health care providers and the public of the safety and efficacy of both vaccines to impact the astounding morbidity and mortality of COVID-19 disease. When asked by patients, “Do you recommend the Pfizer and Moderna vaccines?” Nurse practitioners (NPs) should review their patients’ health status, and if the patient meets the study population, they can confidently say, yes, take the vaccine when available to you.

Pfizer and Moderna are conducting RCT for these vaccines in children, and we can assure parents that when the clinical trials are completed and the evidence is published, that NPs will review the outcomes and make recommendations for practice at that time. In addition, NPs and all health care providers are awaiting the results of the longer-term immunogenicity studies for both vaccines to make additional recommendations for their patient populations as to whether a booster dose will be needed in the coming years.

Uncited References:

Anderson et al., 2020, Devos et al., 2018

Biographies

Donna Hallas, Clinical Professor and Director, Pediatric Nurse Practitioner Program, New York University Rory Meyers College of Nursing, New York City, NY.

Regena Spratling, Associate Professor and Associate Dean and Chief Academic Officer for Nursing, School of Nursing, Byrdine F. Lewis College of Nursing and Health Professions, Georgia State University, Atlanta, GA.

Jason Fletcher, Senior Biostatistician, New York University Rory Meyers College of Nursing, New York, NY.