Table 2.

Estimated agreement across various levels of analysis

Unit of Analysis	Kappa (95% CI)	F-measure
Document	0.87 (0.82,0.93)
Concept
Signs and Symptoms		0.61 (0.57, 0.86)
Diagnoses		0.83 (0.80, 0.87)
Procedures		0.63 (0.56, 0.68)
Medications		0.82 (0.76, 0.86)
all classes		0.72 (0.70, 0.74)
Attribute
Granularity	0.34 (0.28,0.41)
Ambiguity	0.08 (0.04,0.13)
Relevance	**
Negation	0.67 (0.60,0.74)
Temporality	0.67 (0.61,0.73)
Experiencer	1
Reason for Service	0.06 (0.03,0.09)
Note Section	0.54 (0.50,0.59)

** Kappa for relevance could not be estimated

*** Only 1 concept was annotated as describing an experiencer other than the patient.