Table 3. Performance measures of machine learning candidate tools applied using leave-one-subject out cross validation and performance measures of quantitative (analog) risk assessment from the four human readers.

Assessment Method	Expert	AUC-ROC	Youden	Specificity	Sensitivity
ML-tool candidate	Nodule	0.78	0.36	0.61	0.90
	Margin	0.86	0.46	0.77	0.85
	Immediate	0.79	0.38	0.65	0.80
	Extended	0.88	0.41	0.81	0.88
	Extended+	0.89	0.60	0.84	0.83
Observer (Analog)	Reader 1	0.76	0.63	0.62	0.88
	Reader 2	0.80	0.67	0.73	0.79
	Reader 3	0.74	0.74	0.88	0.65
	Reader 4	0.65	0.75	0.31	0.94
	Average reader	0.74	NA	0.63	0.82

ML, machine learning; AUC-ROC, area-under-curve of receiver-operator characteristic; NA, not applicable.