Table 5.

Inter-annotator agreement statistics.

Agreed by	Gene/Protein	Disease	Organism	Overall
1 annotator	270	178	319	767
2 annotators	480	309	216	1005
3 annotators	35934	14237	21298	71469

We evaluated annotation agreement using SemEval-2013 Task9.1 strict rule. According to the strict evaluation rule, an annotation agreement is reached only when two annotators agree on the term span and the annotation type. We achieved an overall agreement of 0.99. The first row of this table shows the entity-level breakdown of annotations that were rejected due to the voting system, i.e. at least two annotators must agree on the annotation term, boundary and entity type. Some of these entities were annotated by the other annotators with different entity boundaries.