Data discrepancies and substandard reporting of interim data of Sputnik V phase 3 trial

Restricted access to data hampers trust in research. Access to data underpinning study findings is imperative to check and confirm the findings claimed. It is even more serious if there are apparent errors and numerical inconsistencies in the statistics and results presented. Regrettably, this seems to be what is happening in the case of the Sputnik V phase 3 trial.¹

Several experts3, 4 found problematic data in the published phase 1/2 results.² We have made multiple independent requests for access to the raw dataset, but these were never answered. Despite publicly denying some problems, formal corrections were made to the Article,² thus addressing some concerns.⁵ Notwithstanding the previous issues and lack of transparency, the interim results from the phase 3 trial of the Sputnik V vaccine¹ again raise serious concerns.

We have a serious concern regarding the availability of the data from which the investigators draw their conclusions. The investigators state that data will not be shared before the trial is completed, and then only by approval of stakeholders, including a so-called security department. Data sharing is one of the cornerstones of research integrity; it should not be conditional and should follow the FAIR principles.

The second concern pertains to the trial protocol, as already described in an open letter by the Russian Society for Evidence-Based Medicine.³ The Sputnik V investigators mention that three interim analyses were added to the study on Nov 5, 2020,¹ but this change was not recorded on ClinicalTrials.gov (NCT04530396). Unfortunately, the full study protocol has not been made publicly available, so the rationale behind this change or the type I error rate adjustment, if any, is not known. According to the ClinicalTrials.gov record NCT04530396, the primary outcome was changed on Sept 17, 2020. Initially, the primary outcome was to be assessed after the first dose, but the evaluation was postponed to after the second dose. The presented primary result (efficacy of 91·6%) is dependent on this change, but the reasons for the change have not been made public. Moreover, the latest ClinicalTrials.gov record (Jan 22, 2021) defines the primary outcome inconsistently: “Primary Outcome Measures: percentage of trial subjects…after the first dose…based on the percentage…after the second dose”.

Besides these protocol amendments, the definition of the primary outcome is unclear in the Article,¹ where it says that when COVID-19 was suspected, participants were assessed with “COVID-19 diagnostic protocols, including PCR testing”. Here, we lack some crucial information, such as the clinical parameters determining suspected COVID-19, what diagnostic protocols were used, when the PCR testing was done, what specific method was used, or how many amplification cycles were used. The way cases of suspected COVID-19 were defined could have led to bias in PCR testing used to assess the number of confirmed COVID-19 cases, which is crucial for the efficacy determination.

A final point of concern about the study protocol relates to the enrolment and randomisation of patients. According to the trial profile in figure 1 of the Article,¹ 35 963 individuals were screened and 21 977 individuals were randomised. The ClinicalTrials.gov record for NCT04530396 (Jan 20, 2021) mentions that 33 758 patients were enrolled. We would expect that this last figure should be equal to either the number of participants screened or randomised. Moreover, there is no information about what caused the exclusion of 13 986 participants, as per the trial profile.

The third concern relates to the data reported and numerical results. We found the following data inconsistencies: (1) in figure 2 of the Article,¹ data for the vaccinated group on day 20 refer to more individuals than at day 10, as if there was either information missing for 100 participants at day 10, or participants were enrolled after day 10 (figure 2 was formally corrected on Feb 20, 2021, but the correction statement did not state the reasons leading to such correction); and (2) in table S1 of the appendix,¹ the number of participants reported for the different vaccinated age cohorts do not add up to the reported total (n=338 vs n=342). With such inconsistencies, we question the accuracy of the reported data.

A very peculiar result of the major subgroup analysis of the primary outcome caught our attention. The vaccine efficacy was said to be high for all age groups. The reported percentages were 91·9% in the 18–30-year age group, 90·0% in the31–40-year age group, 91·3% in the 41–50-year age group, 92·7% in the 51–60-year age group, and 91·8% in participants older than 60 years. We checked the homogeneity of vaccine efficacy across age groups (interaction tests): the p value of the Tarone-adjusted Breslow-Day test was 0·9963, and the p value of a non-asymptotic test was 0·9956,⁶ indicating a very low probability of observing a homogeneity this good if the actual homogeneity is perfect. By applying 18 other homogeneity tests (six in table 1, seven in table S6, six in table 2 of the Article¹), we could not find other major abnormality in the overall distribution of p values (appendix).

We also found some highly coincidental results reported in table S3 of the appendix. In particular, two upper confidence limit values for two different distributions (placebo group at baseline for unstimulated and antigen-stimulated measures) both equal 0·708. Of course, this is possible, but we call once more for access to the data from which the statistics originate for close scrutiny.

In line with our earlier concerns with the phase 1/2 results⁴ and the substandard reporting of the phase 3 interim results,¹ we invite the investigators once more to make publicly available the data on which their analyses rely. Access to the protocol, its amendments, and the individual patient records is paramount, as much for clarification as for open discussion of all the issues.

We also invite the Editors of The Lancet to clarify the consequences of further denying access to the data needed for assessing the results presented, should the authors still deny it.

EMB is the owner of Resis Srl. All other authors declare no competing interests. Code to test the homogeneity of vaccine efficacy across age groups is available on the Open Science Framework, https://osf.io/sudxe/