. 2020 Feb 4;3(1):16–20. doi: 10.1093/jamiaopen/ooz072

Table 2.

Error analysis of 100 FNs by the best model

Main category	Subcategory	Count	Subtotal^a
Unanswerable	(a) Vague question	6	14
Unanswerable	(b) Expert deemed unanswerable using only text	8	14
System answered	(c) Expert judged the system acceptable as the gold	6	18
	(d) Expert sided with the system against the gold	12	18
	(e) Real FN	18	68
	(f) Expert disagreed with both the system and gold	7
System refrained	(g) Real FN	24
System refrained	(h) Correct answer ranked second place	19

^aThis column stands for a “redemption” perspective: 14% that the system was not supposed to make it, 18% where the system answer was actually right, and 68% that the system was truly attributed for the FN.