Table 4.
Empirical power for tests of equality of two means
| Test | Error distribution | n per group | α = 0.05 | α = 0.01 | α = 0.001 |
|---|---|---|---|---|---|
| t-test (Y) | Normal | 5 | 0.10939 | 0.02733 | 0.00322 |
| t-test (Blom) | Normal | 5 | 0.11653¥ | 0.04071¥ | 0 |
| Permutation test (Y) | Normal | 5 | 0.10446 | 0.02162 | 0 |
| Permutation test (Ranks) | Normal | 5 | 0.03204 | 0.02162 | 0 |
| t-test (Y) | LaPlace | 5 | 0.12052 | 0.02851 | 0.00314 |
| t-test (Blom) | LaPlace | 5 | 0.14028¥ | 0.05539¥ | 0 |
| Permutation test (Y) | LaPlace | 5 | 0.12864 | 0.03139 | 0 |
| Permutation test (Ranks) | LaPlace | 5 | 0.09391 | 0.03139 | 0 |
| t-test (Y) | 5 | 0.18244 | 0.05897 | 0.01084 | |
| t-test (Blom) | 5 | 0.26666¥ | 0.13369¥ | 0 | |
| Permutation test (Y) | 5 | 0.23624 | 0.09799 | 0 | |
| Permutation test (Ranks) | 5 | 0.19199 | 0.09799 | 0 | |
| t-test (Y) | Weibull (1, 0.5) | 5 | 0.34973 | 0.18931 | 0.07927 |
| t-test (Blom) | Weibull (1, 0.5) | 5 | 0.43828¥ | 0.31499¥ | 0 |
| Permutation test (Y) | Weibull (1, 0.5) | 5 | 0.44622 | 0.29085 | 0 |
| Permutation test (Ranks) | Weibull (1, 0.5) | 5 | 0.36267 | 0.29085 | 0 |
| t-test (Y) | LaPlace | 25 | 0.42706 | 0.20808 | 0.05924 |
| t-test (Blom) | LaPlace | 25 | 0.47360 | 0.24784 | 0.07972 |
| Kruskal–Wallis (Ranks) | LaPlace | 25 | 0.50998 | 0.26034 | 0.07448 |
| t-test (Y) | 25 | 0.46456 | 0.25234 | 0.08722 | |
| t-test (Blom) | 25 | 0.87178 | 0.68580 | 0.38978 | |
| Kruskal–Wallis (Ranks) | 25 | 0.86978 | 0.68242 | 0.37654 | |
| t-test (Y) | Weibull (1, 0.5) | 25 | 0.58656 | 0.39310 | 0.20444 |
| t-test (Blom) | Weibull (1, 0.5) | 25 | 0.98630 | 0.93680 | 0.77968 |
| Kruskal–Wallis (Ranks) | Weibull (1, 0.5) | 25 | 0.99022 | 0.95642 | 0.82032 |
| t-test (Y)* | Weibull (1, 0.5) | *25 | 0.08972 | 0.02034 | 0.00206 |
| t-test (Blom)* | Weibull (1, 0.5) | *25 | 0.62910 | 0.37776 | 0.14214 |
| Kruskal–Wallis (Ranks)* | Weibull (1, 0.5) | *25 | 0.60964 | 0.35880 | 0.12734 |
Note: 100,000 simulations
Population mean difference = 0.5 within group standard deviations, except * where Population mean difference = 0.125 within group standard deviations (last three rows). Kruskal–Wallis is the large-sample approximate test of the Wilcoxon–Mann–Whitney
Demonstrated significantly inflated Type 1 error rate (see Table 1)