Revisiting the evidence for physical distancing, face masks, and eye protection – Authors' reply

We appreciate the comments we received on our urgent evidence synthesis addressing use of masks, eye protection, and distancing early on in the COVID-19 pandemic.¹ Although we appreciate Willem Lijfering's concerns, he appears to have misunderstood the intent of our analysis to be a comparison of rates between countries, which would be an ecological analysis. As clearly reflected in our stated objective and eligibility criteria, we included only comparative studies and focused on relative effects for all intervention effects. Furthermore, we do not claim that our study has no bias but describe how we minimised bias in our evidence synthesis, assessed the risk of bias in included studies, did sensitivity analyses to test the robustness of our findings, and rated the certainty in the effects based on a structured approach to assessing the evidence. Indeed, we generally rated the certainty as low and adopted a conservative approach by not rating up the certainty of evidence for large effects found for face masks and eye protection. We also reflected our low certainty ratings by using terms such as might and probably in our interpretation of the findings.

We were cautious to not compare apples with oranges, as Didier Pittet and colleagues appear to suggest, and that is why we included betacoronaviruses rather than all respiratory viruses. We made that decision a priori and at a time when little direct evidence was available (March, 2020) to inform public health decision making. We acknowledged this indirectness in our review.

Luca Scorrano and colleagues suggest that we recommended universal face mask use. We intentionally made no recommendations and described in the Article and elsewhere that baseline risk is critical in any decision making about mask use and that many factors (particularly the baseline risk of infection) would have to be considered before making recommendations. It is not the role of a systematic review to make practice recommendations.² What we did recommend was that robust randomised trials be undertaken “to better inform the evidence for these interventions”. We further agree that it is challenging to evaluate the independent effect of eye protection. That is the reason why we attempted to identify studies that correctly adjusted for the use of other personal protective equipment.

Scorrano and colleagues focus on statistical significance of single studies which is not a relevant issue in systematic reviews. We also already and expressively acknowledged the possibility of recall bias. We share with Qi Zhou and colleagues concern about preprints, particularly when their peer reviewed versions report different results. In fact, we organised a highly successful Guidelines International Network conference in 2020 around this topic with leading scientific journal editors who endorsed that concern. In our revised analysis we use the data as reported in the final article version.

John Conly and colleagues raise concerns about recommendations of widespread use of N95 respirators in the accompanying Comment by C Raina MacIntyre and Quanyi Wang.³ We did not issue recommendations, we instead discussed the limitations of the data and expressed low certainty that N95 respirators might reduce transmission more than medical masks. Furthermore, we highlighted the critical need for high quality studies and the anticipated challenges for policy making due to the uncertainty in the evidence and the need to urgently respond to the evolving pandemic. We do not believe that we misclassified several studies with regards to mask use. The actual problem is that studies were very poorly reported, opening the door to alternative ways of interpreting the data. Seeing that problem, we were very careful with classifying studies. Mohamed Abbas and colleagues and Peter Jüni and colleagues raise issues about inclusion of studies and possible duplication of included studies. In general, we believe that most discrepancies noticed by other authors are based on different interpretations or differences in the analysis. For example, Abbas and colleagues questioned why the studies by Pei and colleagues and Loeb and colleagues were excluded; indeed they were not excluded and this criticism seems unwarranted. In agreement with Abbas and colleagues suggestion, we stratified studies by health-care versus non-health-care setting and our formal assessment of effect modification suggested it to be plausible but potentially spurious. Abbas and colleagues comment on the inclusion of studies by Liu and colleagues and Ma and colleagues. Indeed, these are duplicate publications of the same study. In our correction, we use the data from both articles as one study to supplement any missing information needed to estimate effects with a preference for the study by Liu and colleagues. We did include the wrong reference to another study by Liu and colleagues which caused confusion. For the two studies by Nishiura and colleagues and Nishiyama and colleagues, there was partial overlap in the two separate publications. To avoid any duplicate use of data we focused on the larger data set by Nishiura and colleagues.4, 5 We have, based on the feedback we received here and elsewhere, audited all included studies and our raw data in appendix 1 and corrected identified errors. Based on our audit, and the comments received here and elsewhere, the effect estimates of our corrected analysis did not substantially change (updated figures are in appendix 2).

For the reported findings, in which we focused on the adjusted estimates (the adjusted odds ratio [aOR]), we provide the following corrections.

Distancing: the originally presented aOR 0·18 (0·09–0·38) and relative risk (RR) 0·30 (0·20–0·44) are now aOR 0·17 (0·08–0·70) and RR 0·30 (0·20–0·46).

Face masks: the originally reported aOR 0·15 (0·07–0·34) and RR 0·34 (0·26–0·45), after excluding the reports by Ma and colleagues and Nishiyama and colleagues and correcting errors, are now aOR 0·23 (0·11–0·52) and RR 0·38 (0·28–0·50).

For the comparison N95 and similar face masks compared to no face masks, the previous estimate aOR 0·15 (0·07–0·34) changes to aOR 0·23 (0·11–0·52), and for the subgroup analysis of N95 and similar face masks compared with no face masks the previous estimate of aOR 0·04 (0·004–0·30) changes to aOR 0·05 (0·004–0·66) and surgical face mask or similar compared with no face mask aOR 0·33 (0·17–0·61) changes to aOR 0·42 (0·23–0·76).

Eye protection: we reported aOR 0·22 (0·12–0·39) and RR 0·34 (95% CI 0·22–0·52) where only the RR changes to 0·33 (0·20–0·56).

© 2021 Christian Ender/Getty Images

We declare no competing interests.