Table 9.

Comparison of the inter-annotator and self-agreement over four evaluation measures

	Inter-annotator agreement	Annotators’ self-agreement
No. of overlapping examples	6807	2316
$Accuracy$ ($-,0,+$)	$77.2\%$	$86.4\%$
$\overline{F_1}(-,+)$	$74.6\%$	$87.1\%$
$Accuracy\pm 1$ ( $-,+$)	$96.0\%$	$97.4\%$
Alpha	$67.7$ (4604 examples)	82.6 (1969 examples)