Table 3.

The metrics for training and cross-validation on the 2012 to 2017 data set.

Model	AUCa (95% CIb)	Accuracy (95% CI)	Precision	Recall	F1 score
NBc	0.908 (0.882-0.934)	0.870 (0.839-0.898)	0.734	0.865	0.794
DTd	0.870 (0.839-0.901)	0.865 (0.833-0.893)	0.715	0.885	0.791
RFe	0.961 (0.944-0.978)	0.896 (0.867-0.921)	0.794	0.865	0.828
SVMf	0.947 (0.925-0.969)	0.900 (0.872-0.924)	0.859	0.782	0.819
MLPg	0.957 (0.938-0.976)	0.917 (0.890-0.939)	0.828	0.897	0.862
CNNrh	0.984 (0.972-0.995)	0.946 (0.924-0.964)	0.938	0.872	0.904
CNNwi	0.988 (0.977-0.999)	0.959 (0.939-0.974)	0.947	0.910	0.928
LSTMrj	0.982 (0.972-0.992)	0.943 (0.920-0.961)	0.919	0.878	0.898
LSTMwk	0.975 (0.960-0.990)	0.937 (0.913-0.956)	0.918	0.859	0.887

^aAUC: area under the receiver operating characteristic curve.

^bCI: 95% confidence intervals for the AUC.

^cNB: naïve Bayes.

^dDT: decision tree.

^eRF: random forest.

^fSVM: support vector machine.

^gMLP: multilayer perceptron.

^hCNNr: convolutional neural network with randomly initialized word embeddings.

ⁱCNNw: convolutional neural network with Word2vec word embeddings.

^jLSTMr: long short-term memory with randomly initialized word embeddings.

^kLSTMw: long short-term memory with Word2vec word embeddings.