Table 4. Summary of leading methods for online error rate control, giving dependence assumptions and pros & cons.
| FDR or mFDR | LORD++ [An online analogue of the BH procedure] | Independence of null p-values for FDR control, conditional super-uniformity of null p-values for mFDR control |
+ Extensions for prior weights, penalty weights, decaying memory, as well as local dependence (asynchronous and batch testing) + Empirically robust to positive dependence of p-values − Not robust to arbitrary dependence of p-values − Typically lower power than SAFFRON or ADDIS |
| SAFFRON [Adaptive algorithm based on an estimate of the proportion of true null hypotheses] | As above |
+ Higher power than LORD++ if there is a significant fraction of non-nulls and the signals are strong + Extensions for local dependence (asynchronous and batch testing) − Not robust to dependence of p-values |
|
| ADDIS [Combines adaptivity with discarding of conservative nulls] | As above |
+ Higher power than SAFFRON when there are conservative nulls + Extensions for local dependence (asynchronous testing) − Not robust to dependence of p-values |
|
| LOND | Controls FDR under positive dependence of p-values |
+ Provable FDR control for positive dependence (the ‘PRDS’ assumption) − Substantially lower power than the algorithms above |
|
| FDX | supLORD | Null p-values are conditionally super-uniform |
+ Also controls the mFDR and FDR at both fixed times and stopping times + User may choose the number of rejections after which we begin controlling FDX in exchange for more power − Unclear how robust to departures from conditional superuniformity |
| FWER | Alpha-spending | — |
+ Robust to arbitrary dependence of p-values − Very low power, rejects only a few hypotheses before becoming unable to reject any more hypotheses |