Table 4.

Interannotator Agreement for Outcome Only Annotation (Outcome) and Annotation of All Clinically Relevant Elements (All Elements) in Outcome Identification

Set	Annotators	Annotation	All Annotators	Clinicians	Best Pairwise
2	RN1, MS	Outcome	0.42	0.42	0.42
3	RN1, RN2, PhD, MD	All elements	0.65	0.63	0.75
3	RN1, RN2, PhD, MD	Outcome	0.81	0.8	0.98
4	RN1, RN2, PhD, MD	All elements	0.63	0.77	0.84
4	RN1, RN2, PhD, MD	Outcome	0.78	0.94	0.97
5	PhD, MD	Outcome	0.75	—	0.75