Table 2.

Performance of different models in training set and validation set.

Method	Training						Validation
	AUC (95%CI)	ACC (p)	Sensitivity	Specificity	PPV	NPV	AUC (95%CI)	ACC (p)	Sensitivity	Specificity	PPV	NPV
LR	0.885	0.814 (0.810)	0.800	0.829	0.824	0.806	0.821	0.774 (0.470)	0.800	0.750	0.750	0.800
	(0.816–0.942)						(0.711–0.939)
RF	0.998	0.957 (0.810)	0.943	0.971	0.971	0.944	0.740	0.710 (0.149)	0.800	0.625	0.667	0.769
	(0.995,1.0)						(0.577,0.880)
SVM	0.930 (0.877–0.974)	0.829 (0.231)	0.857	0.800	0.811	0.848	0.829 (0.709–0.946)	0.742 (0.071)	0.867	0.625	0.684	0.833
DT	0.923	0.900 (0.151)	0.971	0.829	0.850	0.967	0.605	0.581 (0.470)	0.600	0.562	0.562	0.600
	(0.874–0.967)						(0.492–0.727)
Naive Bayes	0.796	0.714 (0.632)	0.771	0.657	0.692	0.742	0.644	0.516 (0.988)	0.467	0.562	0.500	0.529
	(0.705–0.870)						(0.475–0.803)
KNN	0.852	0.771 (0.031)	0.629	0.914	0.880	0.711	0.819	0.774 (0.988)	0.733	0.812	0.786	0.765
	(0.777–0.914)						(0.699–0.932)
Radiologist1	-	0.743 (0.810)	0.714	0.771	0.758	0.730	-	0.742 (0.718)	0.733	0.750	0.733	0.750
Radiologist2	–	0.714 (0.810)	0.686	0.743	0.727	0.703	-	0.710 (0.470)	0.733	0.688	0.688	0.733
Radiologist3	–	0.729 (0.339)	0.771	0.686	0.711	0.750	–	0.710 (0.470)	0.600	0.813	0.750	0.684
Radiologist4	–	0.843 (1.000)	0.829	0.857	0.853	0.833		0.839 (1.000)	0.800	0.875	0.857	0.824
MLP	0.957	0.886 (0.473)	0.914	0.857	0.865	0.910	0.908	0.903 (0.988)	0.867	0.937	0.928	0.882
	(0.923,0.980)						(0.885,0.941)
CNN	0.957	0.957 (0.632)	0.971	0.943	0.944	0.971	0.840	0.839 (0.470)	0.867	0.813	0.813	0.867
	(0.928–0.979)						(0.797–0.900)

AUC, area under curve; ACC, accuracy; PPV, positive predictive value; NPV, negative predictive value.

Radiologist4 is a senior neuroradiologist with 17 years' experience while the other three are senior neuroradiologists with less than 5 years' experience.

P is significant difference in accuracy compared to the senior radiologist4 using chi-squared test.