Table 2.
Measure of predictive performance | Primary analysis, estimate (95% C.I.) |
After logistic recalibration, estimate (95% C.I.) |
||
---|---|---|---|---|
Full-Model | Partial-Model | Full-Model | Partial-Model | |
PsyMetRiC-HK | ||||
C-Statistic | 0.76 (0.69, 0.81) | 0.73 (0.65, 0.8) | 0.76 (0.69, 0.81) | 0.73 (0.65, 0.8) |
r2 | 0.14 (0.10, 0.18) | 0.12 (0.09, 0.15) | 0.17 (0.14, 0.21) | 0.14 (0.11, 0.17) |
Calibration intercept | 0.78 (0.20, 1.37) | 0.79 (0.42, 1.16) | 0.04 (0.01, 0.06) | 0.08 (−0.01, 0.17) |
Calibration slope | 1.30 (0.89, 1.72) | 1.15 (0.73, 1.58) | 1.05 (0.97, 1.13) | 1.07 (0.94, 1.13) |
Brier score | −0.04 (−0.12, 0.04) | 0.04 (−0.04, 0.11) | 0.03 (0.01, 0.05) | 0.04 (0.01, 0.08) |
PsyMetRiC–external validation in the UK | ||||
C-Statistic | 0.75 (0.69, 0.80) | 0.74 (0.67, 0.79) | ||
r2 | 0.21 (0.18, 0.25) | 0.17 (0.14, 0.20) | ||
Calibration intercept | −0.05 (−0.08, −0.02) | −0.07 (−0.11, −0.03) | ||
Brier score | 0.07 (0.04, 0.10) | 0.08 (0.05, 0.11) |
The C-statistic is a measure of discrimination and estimates the probability that a randomly selected ‘case’ will have a higher predicted probability than a randomly selected non-case. Scores of 1.0 indicate perfect discrimination; scores of >0.70 are generally considered acceptable. The calibration intercept (ideally close to 0) and calibration slope (ideally close to 1) are estimates of model calibration (i.e., the agreement between the observed proportion and predicted risk). The Brier score (ideally close to 0, with scores >0.25 indicating poor performance) is an overall measure of algorithm performance. For comparison, results from the original PsyMetRiC external validation in the UK are shown in the table, see the original PsyMetRiC manuscript for further details.14