Table 3.
HAL Model Performance at Different Institutions
| Population | ROC-AUC (95% CI) | Brier Score | Observed prN2|3 (vs N0|1) | Forecast prN2|3 (vs N0|1) | Hosmer-Lemeshow P |
|---|---|---|---|---|---|
| Entire cohort (n = 1,799) | 0.873 (0.856-0.891) | 0.117 | 0.239 | 0.251 | .034 |
| Cohort excluding centers with <50 patients (n = 1,701) | 0.875 (0.856-0.894) | 0.112 | 0.220 | 0.242 | .675 |
| MD Anderson Cancer Centera (n = 555) | 0.871 (0.835-0.949) | 0.109 | 0.193 | 0.228 | .0467 |
| Cleveland Clinic Foundationb (n = 308) | 0.913 (0.880-0.945) | 0.092 | 0.182 | 0.216 | .704 |
| Henry Ford Hospitalb (n = 108) | 0.848 (0.772-0.924) | 0.136 | 0.269 | 0.290 | .293 |
| Johns Hopkinsb (n = 193) | 0.838 (0.776-0.899) | 0.141 | 0.285 | 0.304 | .280 |
| Royal Melbourne Hospitalc (n = 133) | 0.917 (0.862-0.973) | 0.110 | 0.338 | 0.230 | .010 |
| Michael E. DeBakey VA Hospitalc (n = 117) | 0.912 (0.847-0.977) | 0.083 | 0.188 | 0.165 | .768 |
| St. Luke’s of Kansas Cityc (n = 102) | 0.887 (0.813-0.962) | 0.120 | 0.245 | 0.204 | .459 |
| VA Portland Healthcare Systemc (n = 54) | 0.909 (0.828-0.990) | NA | NA | NA | NA |
| Stanford Universityc (n = 64) | 0.839 (0.742-0.937) | NA | NA | NA | NA |
| University of Chicagoc (n = 67) | 0.810 (0.692-0.989) | NA | NA | NA | NA |
Brier scores range from 0 (perfect) to 1 (worst) for a binary outcome. EBUS-TBNA = endobronchial ultrasound guided-transbronchial needle aspiration; NA = Not applicable because enrollment did not reach the required n ≥ 100 for these centers; N0|1 = N0 or N1; prN2|3 = probability of N2 or N3 disease; ROC-AUC = area under the receiver operator characteristic curve.
a
Hospital where HAL was developed; data tests temporal validation.
b
Hospital where HAL was previously tested; new data tests temporal external validation of HAL.
c
Hospital where HAL was never tested before; external validation of HAL