Table 14.

Inter-annotator agreement results

Category	Average	Range	Average	Range
	CHQA-email		CHQA-web
	Unadjudicated
Avg. # of questions shared	81.9	36-120	540	540-540
Named entity	-	-	0.72	0.66-0.78
Question trigger	0.37	0.18-0.52	0.60	0.46-0.66
Question type	0.58	0.39-0.69	0.74	0.65-0.81
Full frame w/ trigger	0.22	0.08-0.34	0.41	0.33-0.48
Core frame w/ trigger	0.27	0.11-0.41	0.48	0.38-0.56
Full frame w/ type	0.32	0.15-0.46	0.54	0.47-0.58
Core frame w/ type	0.41	0.22-0.56	0.64	0.55-0.70
Question topic	0.71	0.61-0.87	-	-

Inter-annotator agreement is calculated as the micro-average F1 score when one set of annotations is taken as the gold standard