Table 4.
HOMER Model Performance at Different Institutions
| Population | ROC-AUC prN1|2|3 (95% CI) | ROC-AUC prN2|3 (95% CI) | Brier Score | Observed prN1|2|3 (vs N0) | Forecast prN1|2|3 (vs N0) | Observed prN2|3 (vs N0|1) | Forecast prN2|3 (vs N0|1) | HL P |
|---|---|---|---|---|---|---|---|---|
| Entire non-MDACC cohort (n = 1,244) | 0.837 (0.814-0.859) | 0.876 (0.855-0.897) | 0.349 | 0.357 | 0.339 | 0.259 | 0.261 | .002 |
| Non-MDACC data excluding centers with <50 patients (n = 1,146) | 0.837 (0.813-0.860) | 0.878 (0.86-0.900) | 0.341 | 0.335 | 0.325 | 0.233 | 0.249 | <.001 |
| Cleveland Clinic Foundationa (n = 308) | 0.851 (0.803-0.912) | 0.912 (0.879-0.945) | 0.283 | 0.263 | 0.290 | 0.182 | 0.218 | .098 |
| Henry Ford Hospitala (n = 108) | 0.812 (0.729-0.897) | 0.849 (0.773-0.924) | 0.345 | 0.389 | 0.361 | 0.269 | 0.289 | .008 |
| Johns Hopkinsa (n = 193) | 0.803 (0.739-0.867) | 0.838 (0.777-0.900) | 0.363 | 0.363 | 0.380 | 0.285 | 0.302 | .038 |
| Royal Melbourne Hospitalb (n = 133) | 0.878 (0.819-0.938) | 0.919 (0.864-0.974) | 0.370 | 0.504 | 0.402 | 0.338 | 0.288 | .027 |
| Michael E. DeBakey VAb (n = 117) | 0.912 (0.857-0.966) | 0.911 (0.845-0.978) | 0.263 | 0.265 | 0.208 | 0.188 | 0.169 | .148 |
| St. Luke’s of Kansas Cityb (n = 102) | 0.867 (0.798-0.937) | 0.887 (0.813-0.962) | 0.332 | 0.333 | 0.276 | 0.245 | 0.205 | .354 |
| VA Portlandb (n = 54) | 0.886 (0.786-0.985) | 0.911 (0.832-0.990) | NA | NA | NA | NA | NA | NA |
| Stanford Universityb (n = 64) | 0.711 (0.581-0.841) | 0.834 (0.736-0.932) | NA | NA | NA | NA | NA | NA |
| University of Chicagob (n = 67) | 0.672 (0.535-0.809) | 0.809 (0.691-0.927) | NA | NA | NA | NA | NA | NA |
Brier scores range from 0 (perfect) to 2 (worst) when there are three possible outcomes, so the interpretation is different for HOMER and HAL. The lower Brier scores in HAL do not imply the HAL is better than HOMER. H-L = Hosmer-Lemeshow; MDACC = MD Anderson Cancer Center; N0|1 = N0 or N1; NA = not applicable because enrollment did not reach the required n ≥ 100 for these centers; prN1|2|3 = probability of N1 disease or higher; prN2|3 = probability of N2 or N3 disease; ROC-AUC = area under the receiver operator characteristic curve.
Hospital where HOMER was previously validated, new data here tests temporal external validation of HOMER.
Hospital where HOMER was never tested before, external validation of HOMER.