Table 3.

Inter-annotator agreement during training

Agreement Type	C1	C2	C3	C4	C5
Event identification	58.35	56.01	68.26	77.07	71.94
Argument identification (relaxed span match)	80.45	85.05	91.45	89.39	91.09
Argument identification (exact span match)	61.92	63.98	73.96	79.84	79.17
Semantic role assignment	67.27	75.21	93.91	84.89	86.59
Bio-concept identification	71.35	78.65	78.29	88.55	82.36
Bio-concept category assignment (exact category)	72.34	72.05	71.61	68.84	59.76
Bio-concept category assignment (including parent)	77.53	76.74	75.11	71.58	63.65
Bio-concept supercategory assignment	89.21	89.32	93.45	90.57	84.09

Each numbered column (C1 to C5) displays the IAA results calculated after a particular cycle of training, for a number of separate annotation subtasks. Agreement was calculated between each pair of annotators, and the figures shown in the table are averages amongst all pairs of annotators. Training cycles C1 to C4 were concerned with E. coli abstracts, whilst cycle C5 concerned human abstracts