Table 4.

Comparison of agreement between the participants, NLP preannotation, and PDDI annotations (N=151) in the reference standard during the scenario with NER and NLP preannotation assistance (Scenario 3).

NLP Result	No mention found		Mention
Participant	No mention	Mentiona	No mentionb	Mention
	NLP FNf User FN n (%)	NLP FN User TP n (%)	NLP TPc User FN n (%)	NLP TP User TP n (%)
Expert	59 (39.1)	50 (33.1)	23 (15.2)	19 (12.6)
Nonexpert 1	46 (30.5)	63 (41.7)	11 (7.3)	31 (20.5)
Nonexpert 2	43 (28.5)	66 (43.7)	11 (7.3)	31 (20.5)
Nonexpert 3	49 (32.5)	60 (39.7)	13 (8.6)	29 (19.2)

^aIndicates case where the user corrected an NLP error.

^bIndicates cases where the NLP was correct and the user was incorrect.

^cFN: false negative

^dTP: true positive