Table 2.
Comparison of classifier prediction accuracy on validation data P valid, which is used to optimize model regularization, versus independent test data P test
| FXvDM | ATvPM | MvDM | |||||||
|---|---|---|---|---|---|---|---|---|---|
| P valid | P test | Signif. | P valid | P test | Signif. | P valid | P test | Signif. | |
| LD‐L1 | 0.77 | 0.68 | 0.01 | 0.56 | 0.51 | 0.07 | 0.50 | 0.51 | 0.50 |
| LD‐L2 | 0.77 | 0.72 | 0.01 | 0.58 | 0.53 | 0.01 | 0.51 | 0.52 | 0.70 |
| LD‐PC | 0.77 | 0.68 | 0.00 | 0.55 | 0.52 | 0.25 | 0.47 | 0.50 | 0.88 |
| LD‐IC | 0.76 | 0.71 | 0.01 | 0.59 | 0.52 | 0.01 | 0.51 | 0.50 | 0.25 |
| QD‐L1 | 0.72 | 0.67 | 0.00 | 0.57 | 0.51 | 0.01 | 0.51 | 0.50 | 0.20 |
| QD‐L2 | 0.74 | 0.68 | 0.00 | 0.57 | 0.54 | 0.23 | 0.50 | 0.50 | 0.42 |
| QD‐PC | 0.67 | 0.60 | 0.00 | 0.52 | 0.51 | 0.35 | 0.51 | 0.50 | 0.47 |
| QD‐IC | 0.63 | 0.59 | 0.00 | 0.53 | 0.51 | 0.09 | 0.51 | 0.50 | 0.08 |
| LR‐L1 | 0.77 | 0.70 | 0.00 | 0.57 | 0.51 | 0.05 | 0.49 | 0.51 | 0.72 |
| LR‐L2 | 0.77 | 0.72 | 0.00 | 0.59 | 0.53 | 0.00 | 0.52 | 0.53 | 0.46 |
| LR‐PC | 0.79 | 0.70 | 0.00 | 0.53 | 0.51 | 0.15 | 0.47 | 0.49 | 0.68 |
| LR‐IC | 0.77 | 0.72 | 0.00 | 0.59 | 0.53 | 0.02 | 0.50 | 0.51 | 0.45 |
| SVM‐L1 | 0.77 | 0.71 | 0.01 | 0.60 | 0.52 | 0.05 | 0.51 | 0.51 | 0.47 |
| SVM‐L2 | 0.78 | 0.71 | 0.02 | 0.60 | 0.52 | 0.07 | 0.53 | 0.52 | 0.40 |
| SVM‐PC | 0.72 | 0.65 | 0.00 | 0.53 | 0.52 | 0.34 | 0.47 | 0.49 | 1.00 |
| SVM‐IC | 0.72 | 0.65 | 0.00 | 0.57 | 0.52 | 0.00 | 0.52 | 0.51 | 0.06 |
We plot median prediction across subject, for each classifier/regularizer model combination. The models are significantly biased if P valid is consistently higher than P test (significance assessed using non‐parametric paired Wilcoxon test; significant tests shaded in grey). Results are shown for the minimum sample size (N block = 1), for the three task contrasts.