Table 4.
A-C. A comparison of the mean number of significant results among four different procedures for evaluating significance of multiple comparisons: Type I errors, and Type II errors for 100 iterations of 15,000 simulated differential gene expression test using (1) α = 0.05 for all tests, (2) a Bonferroni correction to adjust the family-wise error rate (FWER) to 0.05, (3) the Benjamini-Hochberg procedure to adjust the false-discovery rate (FDR) to 0.05, and (4) optimal α
| Critical effect size (CES) | Average of 100 iterations of 15,000 tests | α = 0.05 | Bonferroni FWER = 0.05 | Benjamini-Hochberg FDR = 0.05 | Optimal α |
|---|---|---|---|---|---|
| A. | |||||
| CES = 1SD | # of significant results | 2046 | 0 | 1 | 6776 |
| # of Type I errors | 376 | 0 | 0 | 2143 | |
| # of Type II errors ≥ CES | 5829 | 7500 | 7499 | 2867 | |
| # of Type I and II errors | 6205 | 7500 | 7499 | 5010 | |
| % error reduction by using optimal α | 19.3% | 33% | 33% | - | |
| CES = 2SD | # of significant results | 5298 | 3 | 1709 | 7659 |
| # of Type I errors | 379 | 0 | 43 | 1130 | |
| # of Type II errors ≥ CES | 2581 | 7497 | 5834 | 970 | |
| # of Type I and II errors | 2960 | 7497 | 5876 | 2100 | |
| % error reduction by using optimal α | 29% | 72% | 64% | - | |
| CES = 4SD | # of significant results | 7848 | 61 | 7560 | 7608 |
| # of Type I errors | 378 | 0 | 190 | 212 | |
| # of Type II errors ≥ CES | 30 | 7439 | 130 | 105 | |
| # of Type I and II errors | 408 | 7439 | 320 | 317 | |
| % error reduction by using optimal α | 22% | 96% | 1% | - | |
| B. | |||||
| CES = 1SD | # of significant results | 1400 | 0 | 0 | 1456 |
| # of Type I errors | 562 | 0 | 0 | 590 | |
| # of Type II errors ≥ CES | 2912 | 3750 | 3750 | 2883 | |
| # of Type I and II errors | 3474 | 3750 | 3750 | 3473 | |
| % error reduction by using optimal α | 0.02% | 7% | 7% | - | |
| CES = 2SD | # of significant results | 3032 | 1 | 119 | 3537 |
| # of Type I errors | 562 | 0 | 5 | 791 | |
| # of Type II errors ≥ CES | 1280 | 3749 | 3636 | 1004 | |
| # of Type I and II errors | 1842 | 3749 | 3641 | 1795 | |
| % error reduction by using optimal α | 3% | 52% | 51% | - | |
| CES = 4SD | # of significant results | 4295 | 31 | 3665 | 3826 |
| # of Type I errors | 560 | 0 | 136 | 200 | |
| # of Type II errors ≥ CES | 15 | 3719 | 221 | 124 | |
| # of Type I and II errors | 575 | 3719 | 358 | 324 | |
| % error reduction by using optimal α | 44% | 91% | 9% | - | |
| C. | |||||
| CES = 1SD | # of significant results | 1012 | 0 | 0 | 10 |
| # of Type I errors | 680 | 0 | 0 | 5 | |
| # of Type II errors ≥ CES | 1167 | 1500 | 1500 | 1495 | |
| # of Type I and II errors | 1847 | 1500 | 1500 | 1500 | |
| % error reduction by using optimal α | 19% | 0% | 0% | - | |
| CES = 2SD | # of significant results | 1662 | 1 | 3 | 1083 |
| # of Type I errors | 677 | 0 | 0 | 334 | |
| # of Type II errors ≥ CES | 515 | 1499 | 1497 | 752 | |
| # of Type I and II errors | 1192 | 1499 | 1498 | 1086 | |
| % error reduction by using optimal α | 9% | 28% | 27% | - | |
| CES = 4SD | # of significant results | 2169 | 12 | 1261 | 1539 |
| # of Type I errors | 675 | 0 | 56 | 143 | |
| # of Type II errors ≥ CES | 6 | 1488 | 295 | 105 | |
| # of Type I and II errors | 681 | 1488 | 350 | 248 | |
| % error reduction by using optimal α | 64% | 83% | 29% | - | |
Type II error rates and optimal α levels were evaluated using three different critical effect sizes (CES), representing effects as large as 1, 2, and 4 standard deviations (SD) of the data. The 15,000 simulated tests had 4 replicates in the experimental and control groups, and were constructed such that (A) H A prior probability = 0.50, H o prior probability = 0.50; (B) H A prior probability = 0.25, H o prior probability = 0.75; (C) H A prior probability = 0.10, H o prior probability = 0.90