The harmonic mean p-value: Strong versus weak control, and the assumption of independence

Wilson (1) proposes a multiple testing procedure based on the harmonic mean $p$-value (HMP). While this is a potentially useful method, he makes several claims that are not supported by the theory. Herein we identify 4 errors, for clarity described in terms of the version with equal weights $1/L$, so that $w_{\mathcal{R}}=\left|\mathcal{R}\right|/L$.

• First, Wilson claims strong familywise error rate (FWER) control for HMP using a closed testing argument. Indeed, if HMP rejects an intersection hypothesis $U_{\mathcal{R}}$ for some $\mathcal{R}$, then it also rejects $U_{\mathcal{R}*}$ for every $\mathcal{R}*\supseteq \mathcal{R}$, implying that HMP is a closed testing procedure (2). However, from this argument Wilson may only claim that, for each set $\mathcal{R}$ where HMP is significant, at least one of the genetic variants in $\mathcal{R}$ has signal. Strong FWER control, that is, the claim that all such genetic variants have signal, may be claimed only for the elementary hypotheses rejected by a closed testing procedure. Rejecting nonsingleton sets, as Wilson does (ref. 1, p. 1198), gives no more than weak FWER control on these sets.

• Second, Wilson claims, without proof, that HMP is valid without any dependence assumptions. However, Vovk and Wang (3) showed that the critical value for HMP under general dependence is smaller than $\alpha /\log \left(\left|\mathcal{R}\right|\right)$, much smaller than ${\alpha }_{\left|\mathcal{R}\right|}$ from table 1 in ref. 1, proving that Wilson’s HMP loses error guarantees under general dependence. A simulation shows lack of control even under moderate positive dependence. With $\left|\mathcal{R}\right|=L=10^5$, simulating standard normals with common correlation $\rho =0.2$, and testing with 1-sided Z-tests, we obtain a type I error of 0.164 for $\alpha =0.05$, and still an excessive rate of 0.091 with ${\alpha }_{\left|\mathcal{R}\right|}=0.029$ in table 1 in ref. 1.

• Third, Wilson claims (ref. 1, p. 1197) that HMP is more powerful than Bonferroni: When Bonferroni rejects we have $p_i\le \alpha /L$ for some $i$, so also HMP has ${\stackrel{\circ }{p}}_{\mathcal{R}}\le \alpha \hspace{0.17em}{w}_{\mathcal{R}}$ when $i\in \mathcal{R}$. However, this argument is not fair. Using table 1 in ref. 1, rejection with HMP requires ${\stackrel{\circ }{p}}_{\mathcal{R}}\le {\alpha }_{\left|\mathcal{R}\right|}\hspace{0.17em}{w}_{\mathcal{R}}<\alpha$, so HMP cannot be claimed to be more powerful than Bonferroni. Wilson’s argument even reverses for proper strong FWER control: HMP must reject some elementary $U_i$, which requires $p_i\le {\alpha }_L/L$, with ${\alpha }_L$ from table 1 in ref. 1. Because ${\alpha }_L<\alpha$, after leveling the playing field Bonferroni is more powerful than HMP, and, unlike HMP, robust to dependence.

• Fourth, Wilson claims (ref. 1, p. 1197) that HMP “produces significant results whenever the Simes-based BH [Benjamini–Hochberg] procedure does, although BH only controls the less stringent FDR [false discovery rate].” Indeed, HMP is smaller than the Simes/BH $p$-value. However, the proper critical value for Simes/BH is $\alpha$, while being ${\alpha }_{\left|\mathcal{R}\right|}\hspace{0.17em}{w}_{\mathcal{R}}<\alpha$ for HMP, so the comparison is not fair. If all $p$-values are in $\left({\alpha }_L,\alpha \right]$, BH rejects all hypotheses, and HMP none. Moreover, BH in fact controls a more stringent criterion than HMP: Both methods control FWER weakly, but BH additionally controls FDR. Finally, unlike HMP, Simes/BH is robust to positively dependent $p$-values (4).

Despite these concerns, we acknowledge that HMP is of interest. If HMP finds multiple significant regions, HMP’s weak control is simultaneous over these regions. HMP could be extended to control false-discovery proportions post hoc (5, 6). It is thus worth exploring further, but only with a realistic assessment of its properties.