Table 4.
The original and final numbers both in total count and percentage, for Tasks 1 and 2, of true, false, excluded and total relations, considering the majority consensus and the domain expert numbers separately
| Dataset | Relations | ||||
|---|---|---|---|---|---|
| True | False | Excluded | Total | ||
| Task 1 (70%) | Original | 1751 (31.41%) | 3823 (68.59%) | – | 5574 (100%) |
| Amazon workers | 4220 (75.71%) | 283 (5.08%) | 1071 (19.21%) | 4503 (80.79%) | |
| Task 2 (30%) | Original | 729 (30.51%) | 1660 (69.49%) | – | 2389 (100%) |
| Amazon workers + extra rater (on-site) (after reaching consensus) | 1179 (49.35%) | 613 (25.66%) | 597 (24.99%) | 1792 (75.01%) | |
| Expert | 1281 (53.62%) | 343 (14.36%) | 765 (32.02%) | 1624 (67.98%) | |