Table 2.

Diagnostic performance of routine clinical breast diagnosis, the three ultrasound experts, the unimodal ultrasound machine learning algorithms, and the multi-modal ultrasound machine learning algorithms in the validation set

	AUROC – value (95% CI)	Sensitivity – % (95% CI); no	Specificity – % (95% CI); no	Negative predictive value – % (95% CI); no	Positive predictive value – % (95% CI); no
Clinical routine	0.95 (0.93 – 0.97)	100 (97.1 – 100), 126 of 126	35.6 (29.7 – 41.9), 88 of 247	100 (95.9 – 100), 88 of 88	44.2 (38.6 – 50.2), 126 of 285
US expert 1	0.82 (0.77 – 0.87)	88.1 (81.1 – 93.2) 111 of 126	49.4 (43.0 – 55.8), 122 of 247	89.1 (82.6 – 93.7), 122 of 137	47.0 (40.5 – 53.6), 111 of 236
US expert 2	0.82 (0.77 – 0.87)	96.0 (91.0 – 98.7), 121 of 126	24.3 (19.1 – 30.1), 60 of 247	92.3 (83.0 – 97.5), 60 of 65	39.3 (33.8 – 45.0), 121 of 308
US expert 3	0.84 (0.79 – 0.89)	91.3 (84.9 – 95.6), 115 of 126	31.2 (25.4 – 37.4), 77 of 247	87.5 (78.7 – 93.6), 77 of 88	40.4 (34.6 – 46.3), 115 of 285
Unimodal ultrasound ML algorithms*
Logistic regression with elastic net penalty	0.83 (0.78 – 0.87)	100 (97.1 – 100), 126 of 126	9.3 (6.0 – 13.6), 23 of 247	100 (85.2 – 100), 23 of 23	36.0 (31.0 – 41.3), 126 of 350
XGBoost tree	0.82 (0.77 – 0.86)	100 (97.1 – 100), 126 of 126	18.2 (13.6 – 23.6), 45 of 247	100 (92.1 – 100), 45 of 45	38.4 (33.1 – 43.9), 126 of 328
Multi-modal ultrasound ML algorithms**
Logistic regression with elastic net penalty	0.90 (0.87 – 0.93)	100 (97.1 – 100), 126 of 126	27.1 (21.7 – 33.1), 67 of 247	100 (94.6—100), 67 of 67	41.2 (35.6 – 46.9), (126 of 306)
XGBoost tree	0.89 (0.85 – 0.92)	100 (97.1 – 100), 126 of 126	19.0 (14.3 – 24.5), 47 of 247	100 (92.5 – 100), 47 of 47	38.7 (33.3 – 44.2), 126 of 326

^* Trained on ultrasound features

^** Trained on ultrasound features as well as patient age and palpability

AUROC, area under the receiver operating characteristic curve; CI, confidence interval; ML, machine learning; US, ultrasound