Table 5.
Phase II selected model prediction accuracy, by search and prediction method.
| No. | Classification rule threshold | (Specificity + sensitivity)/2 (mean ± SD) |
||
|---|---|---|---|---|
| Naive single Cf. | Marginal single Cf. | Conditional single Cf. | ||
| SIMULATION SET 1 | ||||
| 1 | Threshold = 1 | 0.528 ± 0.03 | 0.533 ± 0.03 | 0.534 ± 0.03 |
| Cross-val thresh | 0.529 ± 0.03 | 0.532 ± 0.03 | 0.532 ± 0.03 | |
| 2 | Threshold = 1 | 0.589 ± 0.02 | 0.592 ± 0.02 | 0.593 ± 0.03 |
| Cross-val thresh | 0.587 ± 0.02 | 0.591 ± 0.02 | 0.591 ± 0.03 | |
| 3 | Threshold = 1 | 0.680 ± 0.03 | 0.690 ± 0.03 | 0.699 ± 0.04 |
| Cross-val thresh | 0.678 ± 0.03 | 0.688 ± 0.03 | 0.697 ± 0.03 | |
| 4 | Threshold = 1 | 0.751 ± 0.02 | 0.755 ± 0.02 | 0.767 ± 0.02 |
| Cross-val thresh | 0.750 ± 0.02 | 0.753 ± 0.02 | 0.766 ± 0.02 | |
| 5 | Threshold = 1 | 0.824 ± 0.03 | 0.842 ± 0.03 | 0.866 ± 0.03 |
| Cross-val thresh | 0.823 ± 0.03 | 0.841 ± 0.03 | 0.865 ± 0.03 | |
| 6 | Threshold = 1 | 0.858 ± 0.02 | 0.884 ± 0.02 | 0.903 ± 0.02 |
| Cross-val thresh | 0.857 ± 0.02 | 0.883 ± 0.02 | 0.903 ± 0.02 | |
| SIMULATION SET 2 | ||||
| 1 | Threshold = 1 | 0.500 ± 0.01 | 0.504 ± 0.01 | 0.505 ± 0.02 |
| Cross-val thresh | 0.501 ± 0.01 | 0.504 ± 0.02 | 0.504 ± 0.02 | |
| 2 | Threshold = 1 | 0.501 ± 0.01 | 0.505 ± 0.01 | 0.507 ± 0.02 |
| Cross-val thresh | 0.499 ± 0.01 | 0.505 ± 0.01 | 0.507 ± 0.01 | |
| 3 | Threshold = 1 | 0.528 ± 0.03 | 0.579 ± 0.06 | 0.583 ± 0.06 |
| Cross-val thresh | 0.539 ± 0.04 | 0.586 ± 0.06 | 0.593 ± 0.06 | |
| 4 | Threshold = 1 | 0.524 ± 0.03 | 0.595 ± 0.04 | 0.608 ± 0.05 |
| Cross-val thresh | 0.529 ± 0.04 | 0.593 ± 0.04 | 0.608 ± 0.05 | |
| 5 | Threshold = 1 | 0.694 ± 0.08 | 0.746 ± 0.06 | 0.754 ± 0.07 |
| Cross-val thresh | 0.707 ± 0.07 | 0.748 ± 0.06 | 0.758 ± 0.06 | |
| 6 | Threshold = 1 | 0.557 ± 0.05 | 0.652 ± 0.05 | 0.684 ± 0.05 |
| Cross-val thresh | 0.570 ± 0.05 | 0.652 ± 0.05 | 0.686 ± 0.05 | |
| SIMULATION SET 3 | ||||
| 1 | Threshold = 1 | 0.616 ± 0.05 | 0.618 ± 0.05 | 0.618 ± 0.04 |
| Cross-val thresh | 0.615 ± 0.05 | 0.616 ± 0.05 | 0.616 ± 0.05 | |
| 2 | Threshold = 1 | 0.783 ± 0.04 | 0.781 ± 0.04 | 0.780 ± 0.04 |
| Cross-val thresh | 0.781 ± 0.04 | 0.780 ± 0.04 | 0.779 ± 0.04 | |
| SIMULATION SET 4 | ||||
| 1 | Threshold = 1 | 0.817 ± 0.03 | 0.826 ± 0.03 | 0.841 ± 0.03 |
| Cross-val thresh | 0.816 ± 0.03 | 0.825 ± 0.03 | 0.840 ± 0.03 | |
| 2 | Threshold = 1 | 0.793 ± 0.03 | 0.801 ± 0.03 | 0.816 ± 0.03 |
| Cross-val thresh | 0.792 ± 0.03 | 0.800 ± 0.03 | 0.814 ± 0.03 | |
Reports, for the three simulation sets, and for each of the single-classifier classification rules, the mean and standard deviation of the average of the specificity and sensitivity for the classification of the external replication sets using the final models selected by the cross-validation algorithm. Both the Bayesian 0–1 loss function threshold (i.e., threshold = 1) and the cross-validation selected threshold (in which the threshold is selected so as to maximize Youden’s J in the cross-validation) are listed. The conditional and marginal classifications are based on the pleiotropic search method; the conditional classification uses both the subject genotype and the secondary phenotype status, whereas the marginal classification is based only on the subject genotype. The naive classification uses the naive search method to find the model, and as such does not use the secondary phenotype in either model selection or classification.