Table 1.

F₁ measures for all participants and NLP system across all scenarios and overall.

Annotator	Scenario 1a	Scenario 2b	Scenario 3c	Scenario 4d	Overall
Expert	0.80	0.79	0.54	0.66	0.68
Nonexpert 1	0.79	0.83	0.59	0.53	0.66
Nonexpert 2	0.76	0.68	0.57	0.70	0.67
Nonexpert 3	0.74	0.62	0.53	0.62	0.61
NLP	0.58	0.40	0.41	0.46	0.46

^aNo assistance.

^bPreannotation of drug mentions.

^cPreannotation of drug mentions and PDDIs.

^dNo assistance.