Table 3. Validity of new method compared to alternative methods.
| Independent Samples | Dependent Samples | |||||
| Method | Sensitivity | Specificity | Sample Sizes ( ) |
Sensitivity | Specificity | Sample Sizes ( =  ) |
| a) very informative variables | ||||||
| New Method: | 1.00 | 0.86 | 12; 10 | 1.00 | 0.82 | 11 |
T-Test without  -adjustment: |
1.00 | 0.95 | 1.00 | 0.95 | ||
T-Test with  -adjustment: |
0.90 | 1.00 | 0.90 | 1.00 | ||
| Random Forest: | 0.30 | 1.00 | 0.50 | 1.00 | ||
| New Method: | 1.00 | 0.78 | 14; 16 | 1.00 | 0.69 | 15 |
T-Test without  -adjustment: |
1.00 | 0.95 | 1.00 | 0.95 | ||
T-Test with  -adjustment: |
1.00 | 1.00 | 1.00 | 1.00 | ||
| Random Forest: | 0.20 | 1.00 | 0.20 | 1.00 | ||
| New Method: | 1.00 | 0.56 | 30; 30 | 1.00 | 0.41 | 30 |
T-Test without  -adjustment: |
1.00 | 0.93 | 1.00 | 0.96 | ||
T-Test with  -adjustment: |
1.00 | 1.00 | 1.00 | 1.00 | ||
| Random Forest: | 0.50 | 1.00 | 0.30 | 1.00 | ||
| b) semi-informative variables | ||||||
| New Method: | 1.00 | 0.87 | 12; 10 | 1.00 | 0.81 | 11 |
T-Test without  -adjustment: |
1.00 | 0.96 | 1.00 | 0.95 | ||
T-Test with  -adjustment: |
0.00 | 1.00 | 0.50 | 1.00 | ||
| Random Forest: | 0.70 | 0.99 | 0.30 | 1.00 | ||
| New Method: | 1.00 | 0.74 | 14; 16 | 1.00 | 0.67 | 15 |
T-Test without  -adjustment: |
1.00 | 0.96 | 1.00 | 0.95 | ||
T-Test with  -adjustment: |
0.70 | 1.00 | 0.60 | 1.00 | ||
| Random Forest: | 1.00 | 0.90 | 0.30 | 1.00 | ||
| New Method: | 1.00 | 0.59 | 30; 30 | 1.00 | 0.39 | 30 |
T-Test without  -adjustment: |
1.00 | 0.95 | 1.00 | 0.96 | ||
T-Test with  -adjustment: |
0.90 | 1.00 | 1.00 | 1.00 | ||
| Random Forest: | 1.00 | 0.98 | 1.00 | 0.92 | ||
| c) non-informative variables | ||||||
| New Method: | 0.10 | 0.87 | 12; 10 | 0.10 | 0.81 | 11 |
T-Test without  -adjustment: |
0.00 | 0.96 | 0.00 | 0.95 | ||
T-Test with  -adjustment: |
0.00 | 1.00 | 0.00 | 1.00 | ||
| Random Forest: | 0.00 | 0.98 | 0.00 | 1.00 | ||
| New Method: | 0.40 | 0.75 | 14; 16 | 0.30 | 0.69 | 15 |
T-Test without  -adjustment: |
0.00 | 0.95 | 0.00 | 0.95 | ||
T-Test with  -adjustment: |
0.00 | 1.00 | 0.00 | 1.00 | ||
| Random Forest: | 0.10 | 0.97 | 0.00 | 1.00 | ||
| New Method: | 0.40 | 0.57 | 30; 30 | 0.70 | 0.40 | 30 |
T-Test without  -adjustment: |
0.10 | 0.95 | 0.00 | 0.96 | ||
T-Test with  -adjustment: |
0.00 | 1.00 | 0.00 | 1.00 | ||
| Random Forest: | 0.00 | 1.00 | 0.00 | 1.00 | ||
Validity of the measure of relevance for the two sample-problem evaluated by sensitivity and specificity. For comparisons we used the a multiple-testing-adjusted approach based on the t-test and the tree-based Random Forest approach. For informative or semi-informative metric data sensitivity showed good concordance with multiple testing without 
-adjustment. This was irrespective of sample design and sample size. Specificity did not show as good results as sensitivity (for more details see text). However, the specificity results were better than those obtained from t-test without multiple-testing-adjustment.