Rejoinder to “Quantifying Publication Bias in Meta-Analysis”

We thank co-editor Michael Daniels for organizing the discussion and Drs. Nancy Geller, Dan Jackson, and Christopher Schmid (henceforth NG, DJ, and CS, respectively) for their outstanding discussion of our work. All three discussants pointed out the importance of assessing publication bias in meta-analysis as well as the difficulties of doing so. Indeed, publication bias is full of uncertainties: how publication criteria lead to bias varies greatly case by case. Studies may be suppressed from publication because their p-values are not statistically significant (Copas et al., 2013; Citkowicz and Vevea, 2017), or their effect sizes are too negative (Duval and Tweedie, 2000a), or their sample sizes are too small (Tang and Liu, 2000), as discussed by NG and CS. Because of this difference, no method can perform well in all cases. Meanwhile, these difficulties warrant future research on exploring the proposed methods’ performance using more simulations and case studies, as suggested by all three discussants. As discussed by CS, the missing data mechanism for publication bias is probably missing not at random, so conducting extensive simulations under various settings requires a considerable amount of work and is beyond the scope of this rejoinder, yet we believe such work could make a good contribution to the literature, similar to the simulations by Duval and Tweedie (2000b), Macaskill et al. (2001), Peters et al. (2006), and Bürkner and Doebler (2014).

In addition to these uncertainties, even the name “publication bias” is a bit controversial. Compared with selection models, funnel-plot-based methods tend to be favored possibly because checking a funnel plot’s asymmetry is intuitive. However, such asymmetry may arise from causes other than publication bias (e.g., poor study quality) and it is often confused with other sources of bias, such as reporting bias, mentioned by CS (Schmid, 2017). Hence, some researchers prefer to describe the problem as “small-study effects” (e.g., Harbord et al., 2006), instead of publication bias. However, small-study effects may only reflect one aspect of a funnel plot’s asymmetry. Like other publication bias detection methods based on asymmetry in funnel plots, our methods cannot distinguish whether a funnel plot’s asymmetry is due to publication bias or other sources of bias. Researchers need to employ other evidence and carefully examine whether a funnel plot’s asymmetry is truly due to publication bias. Moreover, compared with diagnostic or corrective analysis, prevention of publication bias is more desirable (Lau et al., 2006). As suggested by CS, publication bias may be diminished by extensively searching for a wide range of resources; and prospective registration of trials may also help prevent publication bias (Rothstein et al., 2005).

As DJ noted, we intended to emphasize measuring publication bias instead of merely testing for it, because the former has seldom been considered. However, we did not mean to discourage researchers from testing for publication bias. “Measuring” and “testing” are different but strongly related concepts, and both can provide valuable information about publication bias. Measures must possess certain features; for example, they should be invariant to the scale of treatment effects and the number of studies in a meta-analysis. So far, most papers on publication bias have focused on hypothesis testing. However, this is insufficient for researchers, who may want to quantify how serious the publication bias is. The measures of publication bias considered in our article, the regression intercept T_I and the skewness T_S, can help us evaluate the severity of publication bias in a meta-analysis. Currently, the regression intercept T_I has no intuitive cutoff points to distinguish the magnitude of publication bias, and the traditional cutoff points of the skewness, ±0.5 and ±1, may be fairly rough. In future work, we will explore empirical choice of cutoff points of these measures using real-world meta-analyses.

On the other hand, the publication bias measures themselves can be good test statistics. If the hypothesis test based on a publication bias measure is very powerful, the measure may be able to quantify publication bias accurately. The two publication bias measures discussed in our article, T_I and T_S, can directly serve as test statistics, and their performance is determined by the power of the regression test and the skewness-based test. However, not all test statistics can serve as publication bias measures; for example, the R₀ statistic used in the trim-and-fill method (Duval and Tweedie, 2000a) may not be a good measure, because it depends on the number of studies.

We are grateful for DJ’s many inspiring methodological suggestions. For example, the modified regression test of model (1) in our article requires an estimate of the between-study variance τ², and we used the classical method-of-moments estimator (DerSimonian and Laird, 1986). However, as DJ pointed out, publication bias has some impact on this estimator (Jackson, 2006, 2007). The performance of model (1) could be improved by employing the adjusted between-study variance obtained from the trim-and-fill method (Duval and Tweedie, 2000a). Nevertheless, using the simple method-of-moments estimator, our simulations have shown that the T_I statistic based on model (1) still controlled type I error rate well and had higher power than Egger’s original regression test in the presence of noticeable heterogeneity. The potential bias in the estimation of τ², caused by publication bias, could be remedied to some extent by the dispersion parameter σ in model (1), which is another concern of DJ. As DJ noted, under the true random-effects model, the errors in model (1) should follow the standard normal distribution, i.e., with variance one. However, in the presence of unknown publication bias, the errors do not strictly follow the standard normal and the σ parameter allows dispersion of the errors and gives more degrees of freedom for fitting the regression. The idea is similar to Egger’s original regression test. Specifically, researchers may have two different interpretations for Egger’s regression. First, it can be viewed as a result of the fixed-effect model with a dispersion parameter that adjusts the effect of potential publication bias on the regression fit. Second, one may also interpret Egger’s regression as a model with multiplicative heterogeneity. However, the rationale for multiplicative heterogeneity is usually considered weak and generally not recommended, while additive heterogeneity is more intuitively appealing and has been widely used among meta-analysts (Thompson and Sharp, 1999). Therefore, we prefer the former interpretation; this coincides with our model (1), which was designed under the additive random-effects setting, while Egger’s regression does not account for additive heterogeneity from this perspective. Last but not least, although model (1) allows heterogeneity by modeling each study’s underlying effect as a draw from a normal random effect, in practice the collected studies may be heterogeneous with an underlying distribution that is not approximately normal. We recognize this challenge to distinguish true publication bias from heterogeneity or subgroup effects (Sterne et al., 2011), which will call for further research and continuous attention.