In a recent timely systematic review, Chu et al. [1] assessed the effectiveness of face masks, eye protection, and physical distancing for preventing COVID-19. Because the sample sizes are not large, especially in some studies of COVID-19, this review contains a considerable number of studies with zero counts of infection events, creating challenges in estimating effect sizes. If zero counts appear in both groups, this double-zero-event study (DZS) is omitted from the analyses, as implied in the forest plots in Chu et al. [1] Specifically, at least 9 out of 44 studies in this review are DZS with 1784 subjects. An omission of information about the rare outcome in DZS or artificial correction of the zero counts could impact the conclusions. [2], [3], [4], [5].
We re-analyzed the meta-analyses containing DZS in Chu et al. [1] using a bivariate generalized linear mixed-effects model in SAS version 9.4. [6] It includes zero counts by modeling with the binomial likelihood. [7,8] Appendix A presents the detailed methods.
Key findings
-
•
Some conclusions of the meta-analyses of prevention measures for COVID-19 changed noticeably after including double-zero-event studies.
-
•
The inclusion of double-zero-event studies can be implemented using a generalized linear mixed-effects model.
What this adds to what is known?
-
•
Double-zero-event studies in meta-analyses should be properly taken into consideration.
What is the implication and what should change now?
-
•
Double-zero-event studies are frequently removed from meta-analyses, but they contribute important information to evidence synthesis.
-
•
Sensitivity analyses with and without double-zero-event studies are recommended to assess the robustness of synthesized evidence.
Table 1 compares the results of our re-analyses including DZS with those excluding DZS presented in Figures 2, 4, and 6 and Appendix 6 in Chu et al. [1] These results are also visualized in Appendix B. Inclusion of DZS changed the statistical significance and RR point estimates of several interventions. For example, in the subgroup of MERS, the relative risk (RR) of physical distancing was nonsignificant when excluding DZS, while it became significant after including DZS. In contrast, the RR of eye protection changed from significant to non-significant. Such changes were also observed when including DZS in two meta-analyses presented in Appendix 6 in Chu et al., [1] which focused on combinations of different prevention approaches. By including DZS, with physical distance of 2 m, the effect of eye protection on reducing infection risk was no longer statistically significant, and the reduced risk of physical distancing might not be statistically significant when using N95 respirators. The main changes of the RR point estimate after the DZS inclusion are described in Appendix B.
Table 1.
Comparison between the synthesized intervention effects with and without double-zero-event studies in meta-analyses on physical distancing, face masks, and eye protection.
| Meta-analysis | No. of subjects in DZS (percentage of total sample size) | No. of DZS (percentage of all studies) | RR (95% CI) with DZS excluded | RR (95% CI) with DZS included |
|---|---|---|---|---|
| Risk with physical distancing | ||||
| MERS (n=1158, N=8) | 201 (17.4%) | 4 (50.0%) | 0.23 (0.04, 1.20) | 0.18 (0.03, 0.99) |
| SARS (n=9101, N=17) | 41 (0.5%) | 1 (5.9%) | 0.35 (0.23, 0.52) | 0.40 (0.27, 0.58) |
| COVID-19 (n=477, N=7) | 204 (42.8%) | 2 (28.6%) | 0.15 (0.03, 0.73) | 0.07 (0.01, 0.98) |
| Overall (n=10736, N=32) | 446 (4.2%) | 7 (21.9%) | 0.30 (0.20, 0.44) | 0.32 (0.21, 0.49) |
| Risk with face masks | ||||
| Healthcare setting (n=9445, N=26) | 358 (3.8%) | 6 (23.1%) | 0.30 (0.22, 0.41) | 0.34 (0.23, 0.51) |
| Overall (n=10170, N=29) | 358 (3.5%) | 6 (20.7%) | 0.34 (0.26, 0.45) | 0.37 (0.25, 0.54) |
| Risk with eye protection | ||||
| MERS (n=1056, N=4) | 34 (3.2%) | 1 (25.0%) | 0.24 (0.06, 0.99) | 0.22 (0.01, 4.41) |
| SARS (n=2581, N=8) | 134 (5.2%) | 2 (25.0%) | 0.34 (0.21, 0.56) | 0.36 (0.24, 0.55) |
| Overall (n=3713, N=13) | 244 (6.6%) | 4 (30.8%) | 0.34 (0.22, 0.52) | 0.35 (0.21, 0.59) |
| Risk with eye protection comparing different physical distances | ||||
| 0 m (n=881, N=4) | 102 (11.6%) | 1 (25.0%) | 0.21 (0.13, 0.34) | 0.23 (0.15, 0.32) |
| 1 m (n=1786, N=6) | 118 (6.6%) | 3 (50.0%) | 0.53 (0.39, 0.72) | 0.55 (0.35, 0.85) |
| 2 m (n=1084, N=5) | 62 (5.7%) | 2 (40.0%) | 0.24 (0.06, 0.99) | 0.21 (0.01, 7.09) |
| Overall (n=3756, N=15) | 288 (7.7%) | 6 (40.0%) | 0.34 (0.22, 0.52) | 0.35 (0.21, 0.50) |
| Risk with physical distancing for different face masks | ||||
| N95 respirators (n=1213, N=12) | 370 (30.5%) | 6 (50.0%) | 0.38 (0.16, 0.91) | 0.51 (0.22, 1.15) |
| All non-N95 masks (n=9531, N=20) | 78 (0.8%) | 1 (5.0%) | 0.27 (0.17, 0.43) | 0.27 (0.16, 0.46) |
| Overall (n=10744, N=32) | 448 (4.2%) | 7 (21.9%) | 0.30 (0.20, 0.44) | 0.33 (0.22, 0.50) |
n, number of subjects; N, number of studies; SARS, severe acute respiratory syndrome; MERS, Middle East respiratory syndrome; RR, relative risk; DZS, double-zero studies; CI, confidence interval.
These changes were related to the proportions and sample sizes of DZS in the corresponding meta-analyses (Table 1). In the subgroup of MERS, four out of eight (50%) studies were DZS, containing 201 out of 1,158 (17.4%) subjects. In the subgroup of COVID-19, two out of seven (29%) studies were DZS, containing 204 out of 477 (42.8%) subjects. The DZS contained considerable information in these subgroups; thus, the RRs produced by excluding and including DZS had noticeable differences.
Given that high-quality evidence about COVID-19 is in great demand and some intervention effects can be changed by including DZS, we suggest future systematic reviews and meta-analyses on COVID-19 properly consider the impact of including DZS. Sensitivity analyses that include DZS are recommended. If including DZS does not substantially change the conclusions, the meta-analyses can be used as reliable and robust evidence by policymakers. Researchers who wish to include DZS in future meta-analyses can find our computer code in Appendix C.
Financial support
This research was supported in part by the U.S. National Institutes of Health/National Library of Medicine grant R01 LM012982. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Declaration of Competing Interest
None.
Footnotes
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jclinepi.2021.01.021.
Appendix. Supplementary materials
References
- 1.Chu DK, Akl EA, Duda S, Solo K, Yaacoub S, Schünemann HJ. Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis. The Lancet. 2020;395(10242):1973–1987. doi: 10.1016/S0140-6736(20)31142-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Friedrich JO, Adhikari NKJ, Beyene J. Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Med Res Methodol. 2007;7:5. doi: 10.1186/1471-2288-7-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Cheng J, Pullenayegum E, Marshall JK, Iorio A, Thabane L, Joseph S, Hamilton H. Impact of including or excluding both-armed zero-event studies on using standard meta-analysis methods for rare event outcome: a simulation study. BMJ Open. 2016;6 doi: 10.1136/bmjopen-2015-010983. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ren Y, Lin L, Lian Q, Zou H, Chu H. Real-world performance of meta-analysis methods for double-zero-event studies with dichotomous outcomes using the Cochrane Database of Systematic Reviews. J Gen Intern Med. 2019;34(6):960–968. doi: 10.1007/s11606-019-04925-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Xu C, Li L, Lin L, Chu H, Thabane L, Zou K, Sun X. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol. 2020;123:91–99. doi: 10.1016/j.jclinepi.2020.03.020. [DOI] [PubMed] [Google Scholar]
- 6.Chu H, Nie L, Chen Y, Huang Y, Sun W. Bivariate random effects models for meta-analysis of comparative studies with binary outcomes: methods for the absolute risk difference and relative risk. Stat Methods Med Res. 2012;21(6):621–633. doi: 10.1177/0962280210393712. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.White IR, Turner RM, Karahalios A, Salanti G. A comparison of arm-based and contrast-based models for network meta-analysis. Stat Med. 2019;38(27):5197–5213. doi: 10.1002/sim.8360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Jackson D, Law M, Stijnen T, Viechtbauer W, White IR. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat Med. 2018;37(7):1059–1085. doi: 10.1002/sim.7588. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.