Table 6.

An example from the medical abstract dataset for task 1: inferring true labels. Out of 5 annotations, only 2 have identified a positive span (the other 3 are empty). Dawid-Skene is able to assign higher weights to the minority of 2 annotations to return a part of the correct span. HMM-Crowd returns a longer part of the span, which we believe is due to useful signal from its sequence model.

Gold	… was as safe and effective as … for the empiric treatment of acute invasive diarrhea in ambulatory pediatric patients requiring an emergency room visit
Annotations (2 out of 5)	… was as safe and effective as … for the empiric treatment of acute invasive diarrhea in $\begin{array}{|c|}\hline \begin{array}{|c|}\hline \text{ambulatory pediatric patients}\\ \hline\end{array}\text{requiring an emergency room visit}\\ \hline\end{array}$
Dawid-Skene	… was as safe and effective as … for the empiric treatment of acute invasive diarrhea in ambulatory pediatric patients requiring an emergency room visit
HMM-Crowd	… was as safe and effective as … for the empiric treatment of acute invasive diarrhea in ambulatory pediatric patients requiring an emergency room visit