Table 4.

Results obtained by Auto-WEKA for the 3 settings studied

Setting	Time limit	Algorithm	Accuracy	Sensitivity	FP-rate	Precision	F-score	MCC	ROC AUC	PR AUC	Kappa
Setting 1 (clinical and Rx data)	5 minutes	Bagging	80.3738	0.804	0.25	0.802	0.8	0.575	0.861	0.859	0.57
	15 minutes	Random Commitee	86.4486	0.864	0.174	0.864	0.863	0.709	0.903	0.896	0.706
	30 minutes	Multilayer Perceptron	80.3738	0.804	0.34	0.802	0.802	0.577	0.864	0.863	0.575
	60 minutes	Multilayer Perceptron	80.3738	0.804	0.34	0.802	0.802	0.577	0.864	0.863	0.575
	Overnight	Multilayer Perceptron	93.9252	0.939	0.08	0.94	0.939	0.87	0.915	0.913	0.869
Setting 2 (only clinical data)	5 minutes	LMT	87.3832	0.874	0.173	0.876	0.871	0.73	0.908	0.918	0.723
	15 minutes	REP Tree	81.7757	0.818	0.222	0.816	0.816	0.608	0.822	0.79	0.606
	30 minutes	REP Tree	81.7757	0.818	0.222	0.816	0.816	0.608	0.822	0.79	0.606
	60 minutes	J48	79.9065	0.799	0.262	0.798	0.794	0.564	0.786	0.754	0.557
	Overnight	Random Tree	84.1121	0.841	0.222	0.845	0.836	0.659	0.898	0.891	0.647
Setting 3 (only Rx data)	5 minutes	SMO	71.9626	0.72	0.378	0.715	0.705	0.379	0.735	0.751	0.364
	15 minutes	Multilayer Perceptron	70.5607	0.706	0.392	0.699	0.706	0.691	0.346	0.737	0.755
	30 minutes	SMO	70.0935	0.701	0.39	0.693	0.689	0.338	0.741	0.756	0.329
	60 minutes	AdaBoost	70.5607	0.706	0.368	0.699	0.698	0.355	0.717	0.699	0.351
	Overnight	Bagging	70.5607	0.706	0.406	0.701	0.686	0.343	0.741	0.734	0.324

Accuracy = (TP + TN)/(TP + TN + FP + FN); Sensitivity = TP/(TP + FN); FP-rate = FP/(FP + TP); Precision = TP/(TP + FP);

$\text{F}-\text{scoreβ}=(1+{\text{β}}^2)\times \frac{\mathit{\text{precision}}\times \mathit{\text{Sensitivity}}}{{\text{β}}^2+\mathit{\text{precision}}+\mathit{\text{Sensitivity}}};\text{ MCC = }\frac{TP\times TN-FP\times FN}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TNFN\right)}}$

Kappa is the measure of how closely the instances were correctly classified by the algorithm, comparing it’s accuracy with that of a random classifier.

TP, true positive; TN, true negative; FP, false positive; FN, false negative; MCC, Matthews correlation coefficient; AUC-ROC, area under the receiver operating characteristic curve; AUC-PR, area under the Precision-Recall (sensitivity) curve; Rx, radiographic; LMT, logistic model tree; REP, reduced error pruning; SMO, sequential minimal optimization.