Table 3.

Performance of humans versus automated systems (per-tag and overall precision (P), recall (R) and F value (F))

	Performance of humans versus automated systems
	Annotator 3			Annotator 4			MIST2			MCRF2
	P	R	F	P	R	F	P	R	F	P	R	F
AGE	99.51	91.93	95.57	94.59	94.17	94.38	95.50	95.07	95.28	97.21	93.72	95.43
DATE	98.17	97.78	97.97	98.56	97.62	98.09	96.73	98.57	97.65	97.86	97.78	97.82
ID	93.67	85.06	89.16	88.37	87.36	87.86	88.89	88.89	88.89	95.45	93.33	94.38
INST	78.33	85.98	81.98	84.52	86.59	85.54	90.12	89.02	89.57	93.33	85.37	89.17
LOC	97.78	93.62	95.65	86.67	82.98	84.78	66.67	55.32	60.47	82.86	61.70	70.73
NAME	98.92	94.93	96.88	99.08	97.92	98.50	93.52	95.71	94.60	95.49	96.36	95.92
OTH	68.77	65.55	67.12	47.28	81.27	59.78	83.27	76.59	79.79	87.70	71.57	78.82
PH	96.92	98.44	97.67	96.92	98.44	97.67	88.41	95.31	91.73	95.31	95.31	95.31
All	93.95	92.15	93.04	88.45	94.55	91.40	93.31	93.66	93.49	95.73	92.91	94.30

	Anno3 vs Anno4	Anno3 vs MCRF2	Anno4 vs MCRF2	Anno3 vs MIST2	Anno4 vs MIST2	Anno3 vs gold standard	Anno4 vs gold standard	MCRF2 vs gold standard	MIST2 vs gold standard
	p Value	p Value	p Value	p Value	p Value	p Value	p Value	p Value	p Value
AGE	0.5226	0.9628	0.6493	0.9368	0.6516	*0.0001	*0.0001	*0.0001	*0.0001
DATE	0.881	0.7514	0.7	0.4288	0.555	*0.0001	*0.0001	*0.0001	*0.0001
ID	0.7526	0.1885	0.1428	0.9377	0.8021	*0.0001	*0.0001	*0.0001	*0.0001
INST	0.2785	*0.0107	0.3347	*0.0078	0.2949	*0.0001	*0.0001	*0.0001	*0.0001
LOC	0.1579	*0.0078	0.1891	*0.0001	*0.021	*0.0001	*0.0001	*0.0001	*0.0001
NAME	*0.0221	0.2246	*0.0026	*0.0093	*0.0002	*0.0001	*0.0001	*0.0001	*0.0001
OTH	0.0506	*0.0029	*0.0001	*0.0006	*0.0001	*0.0001	*0.0001	*0.0001	*0.0001
PH	1	0.1299	0.1231	*0.0414	*0.0492	*0.0001	*0.0001	*0.0001	*0.0001
All	*0.0169	0.054	*0.0002	0.5088	*0.0105	*0.0001	*0.0001	*0.0001	*0.0001

Statistical significance tests between F values obtained by humans versus systems (*indicates statistical significance (p<0.05), Anno3=annotator 3, Anno4=annotator 4).

INST, institution; LOC, location; OTH, other; PH, phone.