Review of publication bias in studies on publication bias

Editor—The empirical evidence about the existence of publication bias is comparatively certain, although the direction and extent of specific types of publication related biases and the consequences of publication bias are much less convincing.¹

Questioning whether studies on publication bias themselves suffer from the bias they studied is reasonable. Dubben and Beck-Bornholdt used a funnel plot to tackle this issue and found no evidence of publication bias in studies of publication bias.² Although their short article is readable and interesting, they acknowledge that the analysis is handicapped by insufficient power (with only 26 included studies) and also by the diverse definitions of publication bias in the primary studies. However, the study has other, more important, limitations.

Firstly, the design may not be appropriate. The ideal and most robust design would be to directly compare the findings of published and unpublished studies on publication bias, although it may be difficult, if not impossible, to identify relevant unpublished studies.

Secondly, the funnel plot, although widely used, is an unreliable tool in detecting the existence of publication bias. An asymmetrical funnel plot may be due to many possible factors other than publication bias, and the existence of publication bias cannot be safely ruled out even if the funnel plot is symmetrical. Therefore, it is likely, despite Dubben and Beck-Bornholdt's findings, that studies on publication bias are just as susceptible to biased selection for publication as other types of research.