Table 4.
Predictive Validity of Scales in Training/Test and Hold-Out Samples
| Performance Measure | SPCA | Regularization | Boosting | Naïve MLE |
|---|---|---|---|---|
| Scale Length (number of items) | 16 | 21 | 27 | 57 |
| Training/Test Sample N = 2472 | ||||
| Peak Accuracy (% Correct Classification) | 61% | 64% | 64% | 65% |
| Point Biserial Correlation | 0.26 | 0.32 | 0.35 | .37 |
| Spearman Correlation | 0.26 | 0.33 | 0.35 | .37 |
| Pseudo R2 | 0.08 | 0.12 | 0.15 | 0.16 |
| HR for +1 SD Scale Score | 1.51 | 1.6 | 1.78 | 1.87 |
| AUC | .65 | .65 | .69 | .71 |
| Hold-Out Sample N = 1237 | ||||
| Peak Accuracy (% Correct Classification) | 61% | 65% | 65% | 63% |
| Biserial Correlation | 0.25 | 0.31 | 0.33 | .33 |
| Spearman Correlation | 0.24 | 0.31 | 0.33 | .33 |
| Pseudo R2 | 0.07 | 0.11 | 0.13 | 0.14 |
| HR for +1 SD Scale Score | 1.47 | 1.57 | 1.73 | 1.76 |
| AUC | 0.64 | 0.68 | 0.69 | .69 |
Notes: AUC = Area Under the Receiver Operating Curve; HR = Hazard Rate; MLE = Maximum Likelihood Estimate; SPCA = Supervised Principal Components. The final column, “Naïve MLE”, pertains to a Cox model with no shrinkage or variable selection (i.e., standard Cox model MLEs for all 57 items).