Table 3.

Performance of our hate speech classification model on the training set (cross validation results) and the out-of-sample evaluation set, in comparison to the inter-annotator agreement on the same datasets. The overall performance is measured by Krippendorff’s $\mathit{Alpha}$ and accuracy ($\mathit{Acc}$), and performance for individual classes by $F_1$. Note that the performance of our model is comparable to

the annotator agreement, except for the Violent class, indicated by lower $F_1$.

Performance and agreement	Overall		Acceptable	Inappropriate	Offensive	Violent
	$\mathit{Alpha}$	$\mathit{Acc}$	$F_1$	$F_1$	$F_1$	$F_1$
Model
Training	0.59	0.79	0.87	0.54	0.64	0.52
Evaluation	0.55	0.84	0.91	0.59	0.58	0.39
Inter-annotator
Training	0.59	0.77	0.86	0.52	0.63	0.63
Evaluation	0.56	0.82	0.90	0.53	0.57	0.55