Table 3.

The best performing results for each combination of feature types on the restricted dataset

Feature set	Test	wth	ucth	TP	FP	FN	TN	P	R	NPV	Spec	Acc	F1
Baseline				17	6	27	186	73.90	38.60	87.32	96.90	86.00	50.70
concepts	χ2	–	300	37	15	7	177	71.15	84.09	96.20	92.19	90.68	77.08*
concepts+assert	χ2	–	300	37	12	7	180	75.51	84.09	96.26	93.75	91.95	79.57**
Words	t	10 000	–	34	6	10	186	85.00	77.27	94.90	96.88	93.22	80.95**
words+assert	χ2+t	10	–	40	12	4	180	76.92	90.91	97.83	93.75	93.22	83.33**
words+concepts	t	10 000	50	37	8	7	184	82.22	84.09	96.34	95.83	93.64	83.15**
words+concepts+assert	t	10 000	5000	36	4	8	188	90.00	81.82	95.92	97.92	94.92	85.71**

*p<0.01, **p<0.001; statistically significant differences in performance between the system configurations considered and the baseline.

Acc, accuracy; F₁, F1-measure; FN, false negatives; FP, false positives; NPV, negative predictive value; P, precision; R, recall; Spec, specificity; TN, true negatives; TP, true positives.