Table 3. Performance metrics for best model per outcome in each study.
| Study ID | Outcome | Discrimination C- Statistic | Calibration | Other Global Performance Metrics | Classification Metrics |
|---|---|---|---|---|---|
| Studies Reporting Apparent Validity | |||||
| BHATTACHARYYA 2021 | Psychiatric rehospitalisation | 0.749 | Calibration plot only; No α or β | Brier score - 0.192 | N.R. |
| CHUA 2019 | EET Status at 2 years | 0.759 (95%CI: 0.728, 0.790) | N.R. | N.R. | Classification Accuracy - 0.759; PPV - 0.64; NPV - 0.78; PSI - 0.42 |
| DERKS 2010 | Andreasen symptom remission (6 months duration) with 1 year f/u | N.R. | N.R. | N.R. | Classification Accuracy - 0.63; Balanced Accuracy - 0.665; Sensitivity - 0.73; Specificity - 0.60; PPV - 0.73; NPV - 0.61; PSI - 0.34 |
| FLYCKT 2006 | Global functioning (Independent living, EET status, GAF ≥60) at mean 5.4 years | N.R. | N.R. | N.R. | Classification Accuracy - 0.81; Balanced Accuracy - 0.805; Sensitivity - 0.84; Specificity - 0.77 |
| GONZALEZ-BLANCH 2010 | Global functioning (EET status, DAS ≤1) at 1 year | N.R. | Hosmer–Lemeshow test - p = >0.05 | N.R. | Classification Accuracy - 0.750; Balanced Accuracy - 0.587; Sensitivity - 0.261; Specificity - 0.913; PPV - 0.500; NPV - 0.788; PSI - 0.288 |
| Studies Reporting Internal Validity | |||||
| AJNAKINA 2020 | Early treatment resistance from illness onset with 5 years f/u | 0.77 | α - 0.028; β - 1.264; No calibration plot | N.R. | Balanced Accuracy - 0.5; Sensitivity - 0; Specificity - 1.00; PPV - 0.48, NPV - 0.84; PSI - 0.32 |
| Later treatment resistance with 5 years f/u | 0.77 | α - 0.504; β - 1.838; No calibration plot | N.R. | Balanced Accuracy - 0.81; Sensitivity - 0.62; Specificity - 1.00; PPV - 0.42; NPV - 1.00; PSI - 0.42 | |
| BHATTACHARYYA 2021 | Psychiatric rehospitalisation | 0.66 | Calibration plot only; No α or β | Brier score - 0.232 | N.R. |
| DEMJAHA 2017 | Early Treatment Resistance from Illness Onset with 10 years f/u | N.R. | N.R. | Brier score - 0.146; McFadden pseudo R2 - 0.1 | N.R. |
| DENIJS 2019 | Andreasen Symptom Remission (6 months duration) at 3 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.644; Sensitivity - 0.76; Specificity - 0.50; PPV - 0.722; NPV - 0.548; PSI - 0.27 |
| GAF ≥65 at 3 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.676; Sensitivity - 0.749; Specificity - 0.584; PPV - 0.701; NPV - 0.642; PSI - 0.343 | |
| Andreasen symptom remission (6 months duration) at 6 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.647; Sensitivity - 0.787; Specificity - 0.465; PPV - 0.690; NPV - 0.590; PSI - 0.28 | |
| GAF ≥65 at 6 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.676; Sensitivity - 0.818; Specificity - 0.477; PPV - 0.718; NPV - 0.616; PSI - 0.334 | |
| KOUTSOULERIS 2016 | GAF ≥65 at 1 year | N.R. | N.R. | N.R. | Balanced Accuracy - 0.738; Sensitivity - 0.667; Specificity - 0.809; PPV - 0.515; NPV - 0.888; PSI - 0.403 |
| LEIGHTON 2021 | Andreasen symptom remission (6 months duration) at 1 year | 0.74 (0.73, 0.75) | β - 0.84 (95%CI: 0.81, 0.86); No calibration plot | N.R. | N.R. |
| PUNTIS 2021 | Psychiatric hospitalisation after discharge from early intervention | 0.76 (0.75, 0.77) | α - 0.01 (95%CI: -0.25, 0.24); β - 0.89 (95%CI: 0.88, 0.89); Calibration plot | Brier score - 0.078 | N.R. |
| Studies Reporting Internal-External Validity | |||||
| DENIJS 2019 | Andreasen symptom remission (6 months duration) at 3 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.638; Sensitivity - 0.629; Specificity - 0.647; PPV - 0.758; NPV - 0.485; PSI - 0.243 |
| GAF ≥65 at 3 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.648; Sensitivity - 0.658; Specificity - 0.638; PPV - 0.727; NPV - 0.565; PSI - 0.292 | |
| Andreasen symptom remission (6 months duration) at 6 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.625; Sensitivity - 0.685; Specificity - 0.565; PPV - 0.743; NPV - 0.493; PSI - 0.236 | |
| GAF ≥65 at 6 years | N.R. | N.R. | N.R. | Balanced Accuracy - 0.640; Sensitivity - 0.718; Specificity - 0.561; PPV - 0.732; NPV - 0.553; PSI - 0.285 | |
| KOUTSOULERIS 2016 | GAF ≥65 at 1 year | N.R. | N.R. | N.R. | Balanced Accuracy - 0.711; Sensitivity - 0.641; Specificity - 0.781; PPV - 0.472; NPV - 0.877; PSI - 0.349 |
| LEIGHTON 2019 (2) | EET Status at 1 year | 0.736 (95%CI: 0.702 - 0.771) | N.R. | N.R. | Classification Accuracy - 0.693 (95%CI: 0.660, 0.725); Balanced Accuracy - 0.694 (95%CI: 0.562, 0.812); Sensitivity - 0.722 (95%CI: 0.573, 0.821); Specificity - 0.666 (95%CI: 0.550, 0.803); PPV - 0.719 (95%CI: 0.673, 0.785); NPV - 0.668 (95%CI: 0.606, 0.736); PSI - 0.387 (95%CI: 0.279, 0.521) |
| GAF ≥65 at 1 year | 0.731 (95%CI: 0.697, 0.765) | N.R. | N.R. | Classification Accuracy - 0.687 (95%CI: 0.657, 0.718); Balanced Accuracy - 0.691 (95%CI: 0.541, 0.825); Sensitivity - 0.722 (95%CI: 0.487, 0.778); Specificity - 0.660 (95%CI: 0.594, 0.871); PPV - 0.650 (95%CI: 0.616, 0.769); NPV - 0.726 (95%CI: 0.655, 0.766); PSI - 0.376 (95%CI: 0.271 - 0.535) | |
| Andreasen symptom remission (6 months duration) at 1 year | 0.703 (95%CI: 0.664, 0.742) | N.R. | N.R. | Classification Accuracy - 0.670 (95%CI: 0.636, 0.703); Balanced Accuracy - 0.668 (95%CI: 0.518, 0.827); Sensitivity - 0.584 (95%CI: 0.491, 0.827); Specificity - 0.751 (95%CI: 0.544, 0.827); PPV - 0.679 (95%CI: 0.601, 0.739); NPV - 0.667 (95%CI: 0.631, 0.734); PSI - 0.346 (95%CI: 0.232, 0.473) | |
| Quality of life at 1 year | 0.704 (95%CI: 0.667, 0.742) | N.R. | N.R. | Classification Accuracy - 0.668 (95%CI: 0.632, 0.704); Balanced Accuracy - 0.667 (95%CI: 0.532, 0.789); Sensitivity - 0.623 (95%CI: 0.512, 0.774); Specificity - 0.711 (95%CI: 0.551, 0.803); PPV - 0.633 (95%CI: 0.575, 0.701); NPV 0.700 (95%CI: 0.659, 0.759); PSI - 0.333 (95%CI: 0.234, 0.460) | |
| Studies Reporting External Validity | |||||
| LEIGHTON 2019 (1) | EET status at 1 year | 0.876 (95%CI: 0.864, 0.887) | N.R. | N.R. | Classification Accuracy - 0.851; Balanced Accuracy - 0.845; Sensitivity - 0.815; Specificity - 0.875; PPV - 0.815; NPV - 0.875; PSI - 0.690 |
| Andreasen symptom remission (no duration criteria) at 1 year | 0.652 (95%CI: 0.635, 0.670) | N.R. | N.R. | Classification Accuracy - 0.612; Balanced Accuracy - 0.623; Sensitivity - 0.578; Specificity - 0.667; PPV - 0.794; NPV - 0.424; PSI - 0.218 | |
| Andreasen symptom remission (6 months duration) at 1 year | 0.630 (95%CI: 0.612, 0.647) | N.R. | N.R. | Classification Accuracy - 0.625; Balanced Accuracy - 0.626; Sensitivity - 0.606; Specificity - 0.645; PPV - 0.645; NPV - 0.606; PSI - 0.251 | |
| LEIGHTON 2019 (2) - Validated in UK | EET Status at 1 year | 0.867 (95%CI: 0.805, 0.930) | N.R. | N.R. | Classification Accuracy - 0.838 (95%CI: 0.775, 0.894); Balanced Accuracy - 0.853 (95%CI: 0.740, 0.935); Sensitivity - 0.898 (95%CI: 0.780, 0.966); Specificity - 0.807 (95%CI: 0.699, 0.904); PPV - 0.766 (95%CI: 0.679, 0.867); NPV - 0.911 (95%CI: 0.840, 0.971); PSI - 0.677 (95%CI: 0.519, 0.838) |
| Andreasen symptom remission (6 months duration) at 1 year | 0.680 (95%CI: 0.587, 0.773) | N.R. | N.R. | Classification Accuracy - 0.695 (95%CI: 0.618, 0.771); Balanced Accuracy - 0.695 (95%CI: 0.535, 0.841); Sensitivity - 0.621 (95%CI: 0.455, 0.773); Specificity - 0.769 (95%CI: 0.615, 0.908); PPV - 0.729 (95%CI: 0.636, 0.854); NPV - 0.667 (95%CI: 0.593, 0.759); PSI - 0.396 (95%CI: 0.229, 0.613) | |
| Quality of life at 1 year | 0.679 (95%CI: 0.522, 0.836) | N.R. | N.R. | Classification Accuracy - 0.702 (95%CI: 0.596, 0.809); Balanced Accuracy - 0.729 (95%CI: 0.407, 0.917); Sensitivity - 0.957 (95%CI: 0.564, 1.000); Specificity - 0.500 (95%CI: 0.250, 0.833); PPV - 0.640 (95%CI: 0.561, 0.800); NPV - 0.900 (95%CI: 0.643, 1.000); PSI - 0.540 (95%CI: 0.204, 0.800) | |
| LEIGHTON 2019 (2) - Validated in Denmark | EET Status at 1 year | 0.660 (95%CI: 0.610, 0.710) | N.R. | N.R. | Classification Accuracy - 0.680 (95%CI: 0.609, 0.725); Balanced Accuracy - 0.655 (95%CI: 0.516, 0.774); Sensitivity - 0.584 (95%CI: 0.457, 0.723); Specificity - 0.726 (95%CI: 0.574, 0.824); PPV - 0.490 (95%CI: 0.421, 0.563); NPV - 0.793 (95%CI: 0.760, 0.831); PSI - 0.283 (95%CI: 0.181, 0.394) |
| GAF ≥65 at 1 year | 0.573 (95%CI: 0.504, 0.643) | N.R. | N.R. | Classification Accuracy - 0.456 (95%CI: 0.328, 0.817); Balanced Accuracy - 0.589 (95%CI: 0.234, 0.926); Sensitivity - 0.781 (95%CI: 0.233, 0.945); Specificity - 0.396 (95%CI: 0.234, 0.906); PPV - 0.179 (95%CI: 0.158, 0.333); NPV - 0.914 (95%CI: 0.876, 0.967); PSI - 0.093 (95%CI: 0.034, 0.300) | |
| Andreasen symptom remission (6 months duration) at 1 year | 0.616 (95%CI: 0.553, 0.679) | N.R. | N.R. | Classification Accuracy - 0.618 (95%CI: 0.524, 0.704); Balanced Accuracy - 0.621 (95%CI: 0.342, 0.864); Sensitivity - 0.612 (95%CI: 0.306, 0.843); Specificity - 0.629 (95%CI: 0.378, 0.885); PPV - 0.476 (95%CI: 0.412, 0.636); NPV - 0.742 (95%CI: 0.687, 0.829); PSI - 0.217 (95%CI: 0.099, 0.465) | |
| Quality of life at 1 year | 0.556 (95%CI: 0.481, 0.631) | N.R. | N.R. | Classification Accuracy - 0.589 (95%CI: 0.540, 0.637); Balanced Accuracy - 0.589 (95%CI: 0.312, 0.845); Sensitivity - 0.876 (95%CI: 0.419, 0.947); Specificity - 0.301 (95%CI: 0.204, 0.743); PPV - 0.559 (95%CI: 0.527, 0.642); NPV - 0.706 (95%CI: 0.555, 0.841); PSI - 0.265 (95%CI: 0.081, 0.483) | |
| LEIGHTON 2021 | Andreasen symptom remission (6 months duration) | 0.73 (95%CI: 0.71, 0.75) | α - 0.12 (95%CI: 0.02, 0.22); β - 0.98 (95%CI: 0.85, 1.11); Calibration plot | N.R. | N.R. |
| PUNTIS 2021 | Psychiatric hospitalisation after discharge from early intervention | 0.70 (95%CI: 0.66, 0.75) | α - -0.01 (95%CI: -0.17, 0.167); β - 1.00 (95%CI: 0.78, 1.22); Calibration plot | Brier score - 0.094 | N.R. |
N.R. – not reported; EET – employment, education or training; GAF – Global Assessment of Functioning; DAS – Disability Assessment Schedule; PPV – positive predictive value; NPV – negative predictive value; PSI – prognostic summary index; f/u – follow-up