Table 3.
Predictive performance statistics of the PsyMetRiC full- and partial models before and after logistic calibration in PsyMetab and PAFIP.
| Measure of Predictive Performance | Primary Analysis, Estimate (95% C.I.) |
After Logistic Calibration, Estimate (95% C.I.) |
||
|---|---|---|---|---|
| Full-Model | Partial-Model | Full-Model | Partial-Model | |
| PsyMetab (Switzerland) | ||||
| C-Statistic | 0.73 (0.68, 0.79) | 0.68 (0.62, 0.74) | 0.73 (0.68, 0.79) | 0.68 (0.62, 0.74) |
| r2 | 0.10 (0.06, 0.14) | 0.08 (0.03, 0.13) | 0.12 (0.09, 0,15) | 0.08 (0.04, 0.12) |
| Calibration Intercept | 0.11 (−0.05, 0.26) | 0.12 (0.05, 0.19) | −0.01 (−0.01, −0.01) | −0.01 (−0.01, −0.01) |
| Calibration Slope | 0.77 (0.72, 0.82) | 0.93 (0.86, 1.00) | 1.02 (1.01, 1.04) | 1.03 (1.01, 1.05) |
| Brier Score | 0.13 (0.09, 0.17) | 0.14 (0.08, 0.20) | 0.13 (0.08, 0.16) | 0.14 (0.10, 0.18) |
| PAFIP (Spain) | ||||
| C-Statistic | 0.72 (0.66, 0.78) | 0.66 (0.60, 0.71) | 0.72 (0.66, 0.78) | 0.66 (0.60, 0.71) |
| r2 | 0.10 (0.05, 0.15) | 0.05 (0.02, 0.08) | 0.10 (0.05, 0.15) | 0.05 (0.02, 0.08) |
| Calibration Intercept | 0.24 (0.09, 0.38) | −0.30 (−0.38, −0.22) | 0.01 (0.00, 0.01) | 0.01 (0.00, 0.01) |
| Calibration Slope | 1.09 (0.99, 1.20) | 0.91 (0.80, 1.02) | 1.03 (1.01, 1.05) | 1.04 (1.01, 1.06) |
| Brier Score | 0.13 (0.08, 0.16) | 0.12 (0.09, 0.16) | 0.11 (0.07, 0.16) | 0.12 (0.09, 0.16) |
The C-statistic is a measure of discrimination and estimates the probability that a randomly selected ‘case’ will have a higher predicted probability than a randomly selected non-case. Scores of 1.0 indicate perfect discrimination; scores of >0.70 are generally considered acceptable. The calibration intercept (ideally close to 0) and calibration slope (ideally close to 1) are estimates of model calibration (i.e., the agreement between the observed proportion and predicted risk). The Brier score (ideally close to 0, with scores >0.25 indicating poor performance) is an overall measure of algorithm performance. For comparison, results from the original PsyMetRiC external validation in the UK were: full-model: C=0.75 (95% C.I., 0.69–0.80; r2=0.21 (95% CI., 0.18–0.25); Brier score=0.07 (95% C.I., 0.04–0.10); intercept=-0.05 (95% C.I., −0.08, −0.02); partial-model: C=0.74 (95% C.I., 0.67–0.79); r2=0.17 (95% C.I., 0.14–0.20); Brier score=0.08 (95% C.I., 0.05–0.11); intercept=-0.07 (95% C.I., −0.11, −0.03). See the original PsyMetRiC manuscript for further details.25