There is not enough evidence to support the claim that Covid‐19 vaccination BNT162b2 temporarily impairs semen concentration and total motile count

Dear Editor,

Gat et al. ¹ report results from 37 sperm donors (SD) who provided samples before (T0) receiving two doses of the BNT162b2 vaccine and at three time‐points after; termed short‐ (T1), intermediate‐ (T2), and long‐term (T3) evaluations. Repeated measures analysis, via generalized estimating equations (GEE), as well as simple comparisons of the first sample from each time point and the average of all samples at each time‐point yielded significant differences between T0 and T2 for semen concentration (SC) and total motile count (TMC), with lower values at T2 relative to T0. Semen volume and sperm motility did not significantly differ between T0 and the other time‐points on any of the three analyses. The authors conclude that “[…] in this longitudinal multicenter study, we found a selective temporary decline of sperm concentration and total motile count 3 months postvaccination followed by recovery among SD.” The title and discussion section of the article specify that these findings are specific to sperm donors, and the discussion notes the limitation of the retrospective design, but ultimately this is assumed to have a small impact on the findings.

This article has received attention in both the scientific ² and popular press, ³ and, perhaps more importantly, on social media, where as of August 30 2022, the article link has been tweeted over 43,000 times. ⁴ A different study from the same issue, ⁵ which did not find any significant differences in semen parameters around 70 days post‐BNT162b2 vaccination, has received far less popular attention (tweeted less than 30 times ⁶ ). However, as we describe below, there are several methodological flaws that could make us reconsider the conclusions drawn in Gat et al. We emailed the corresponding author a request to share the data underlying this report but have not heard back at the time of submission. Requests were sent 7/18/2022 and 7/25/2022. Additional requests were sent by the editorial team of Andrology.

Several aspects of the statistical analysis (i.e., those that would be required to recreate the results) are not described in sufficient detail. For instance, it is not clear what working correlation structure was assumed in the GEE analysis, and what standard errors were calculated. Multiple contrasts between T0 and the other time‐points are reported for the four semen parameter endpoints. However, no mention of corrections for multiplicity is noted, and we assume that none were made. There are no strict rules for exactly when and how to adjust for multiple comparisons, but it is generally accepted that studies testing a predefined hypothesis, termed “confirmatory,” appropriate control of error rates is necessary. ⁷ For exploratory research, it has been suggested that corrections for multiple tests are not strictly required, but that these findings should be clearly labeled exploratory and subject to confirmation. ⁷ Whether or not the original intent of this research was exploratory, confirmatory, or somewhere in between, the title, abstract, and discussion all read as confirming a particular finding (i.e., that the vaccine temporarily impairs certain parameters). These results should have been discussed with a lot more caution and explicitly labeled as exploratory pending confirmatory testing. Further, given the understandable public interest in vaccine outcomes, we suggest that more care to control familywise error rates should have been taken. ⁸

The authors note their three separate analysis approaches as strengthening confidence in their findings as “(…) these results were not solely observed by (repeated measures) analysis but also by using a single sample as well as samples’ mean per donor for each time frame.” Both the GEE analysis and the t‐tests/Wilcoxon tests are looking at mean (or location) differences as a function of time point, so it is not surprising that they agree (especially when variables have been transformed to better approximate normal distribution). Further, from Table 1, we can see that the number of samples per donor ranges from 2.1 at T2 to 1.7 at T3 (1.9 at T1). Therefore, it is also not surprising that the results of only analyzing the first sample would align with analyzing the per donor average. There is an issue with Table 3, where some of the median changes between T0 and other time points fall outside the corresponding interquartile ranges (i.e., the 25 and 75 percentiles; e.g., Semen Volume T0 versus T1: Median [50%ile] = 0, 25%ile = −0.94, 75%ile = −0.45). This is not possible.

The rationale for splitting the number of postvaccine days into three time‐points (T1 = 15–45 days after vaccine; T2 = 75–125 days; T3 = 145+) is not given, and we wonder to what extent the findings are dependent on this particular grouping. This may have been prespecified, but given that this was a retrospective study, we doubt this is the case. Categorizing a continuous variable discards valuable information (it assumes that 75 days postvaccine is the same as 125 days) and can cause other unforeseen problems (in this case, as we discuss next, different donors gave samples at different time‐points). A better approach would have been to model the relationship between days postvaccine and semen parameters via smoothing splines. ⁹

Of the 37 sperm donors, only 28 (76%), 29 (78%), and 22 (59%) provided samples at T1, T2, and T3, respectively. The different composition of the samples available at the different time points is an issue and raises the distinct possibility that time‐point is confounded with donor characteristics that relate to semen quality (e.g., age, BMI). However, information on potential confounders is not reported at this level of granularity or adjusted for in the analysis. Finally, a nonsignificant difference between T0 and T3 is not “evidence of recovery” given that p > 0.05 cannot be used to support the null hypothesis of no difference.

In summary, the claim that “Covid‐19 vaccination BNT162b2 temporarily impairs SC and total motile count” requires a far higher degree of evidence than that reported in Gat et al. Further we believe that the causal statement made, that the vaccine temporarily impairs SC and TMC, is unfounded and not supported by the authors’ methodology and could be misleading to the thousands of people that have encountered this article via the popular press and social media.

CONFLICT OF INTEREST

The authors have no conflict of interest to disclose.