Table 3.
Comparison of clinical vector model performance for phase 1 and phase 2.
| Clinical vector performance measures | Model performance, % (95% CI) |
|
|||||
|
|
Phase 1 | Phase 2 | P valuea | ||||
| Behavioral health services | |||||||
|
|
Sensitivity | 70.2 (68.0, 72.5) | 86.3 (83.1, 88.9) | <.001 | |||
|
|
Specificity | 78.5 (78.0, 78.9) | 99.1 (98.5, 99.7) | <.001 | |||
|
|
F1 score | 56.6 (53.6, 58.9) | 90.4 (87.4, 93.4) | <.001 | |||
|
|
Precision (positive predictive value) | 47.4 (44.2, 49.6) | 95.0 (92.0, 98.3) | <.001 | |||
|
|
AUROCb | 88.3 (87.4, 89.2) | 98.0 (97.6, 98.5) | .01 | |||
| Social work services | |||||||
|
|
Sensitivity | 67.0 (63.4, 72.2) | 72.4 (69.1, 75.6) | .07 | |||
|
|
Specificity | 79.6 (79.1, 79.8) | 99.3 (99.2, 99.6) | <.001 | |||
|
|
F1 score | 48.6 (45.0, 52.5) | 82.5 (79.7, 85.3) | <.001 | |||
|
|
Precision (positive predictive value) | 38.2 (34.8, 41.2) | 95.8 (93.8, 97.8) | <.001 | |||
|
|
AUROC | 87.6 (86.1, 89.2) | 93.7 (92.5, 95.0) | .03 | |||
| Dietitian counseling services | |||||||
|
|
Sensitivity | 60.7 (56.5, 64.7) | 73.6 (70.5, 77.0) | .02 | |||
|
|
Specificity | 73.2 (71.9, 74.9) | 93.3 (90.8, 94.6) | <.001 | |||
|
|
F1 score | 61.5 (57.3, 66.0) | 76.4 (73.3, 80.4) | .001 | |||
|
|
Precision (positive predictive value) | 62.2 (58.1, 67.4) | 79.4 (76.4, 84.2) | .02 | |||
|
|
AUROC | 82.5 (81.5, 83.6) | 91.5 (90.3, 92.6) | .001 | |||
| Other wraparound services | |||||||
|
|
Sensitivity | 44.5 (42.7, 46.1) | 59.2 (56.5, 63.8) | .002 | |||
|
|
Specificity | 78.5 (77.5, 79.3) | 92.9 (89.7, 96.1) | <.001 | |||
|
|
F1 score | 43.2 (40.0, 45.7) | 65.5 (62.9, 67.6) | .01 | |||
|
|
Precision (positive predictive value) | 41.9 (37.7, 45.2) | 73.4 (70.5, 77.7) | <.001 | |||
|
|
AUROC | 77.2 (76.2, 78.1) | 85.3 (84.4, 86.0) | .02 | |||
aP values were calculated using confidence intervals [29].
bAUROC: area under the receiver operating characteristic curve.