Table 2.
Examples from the crowdsourced data set.
| News title | Ground truth | Voter soft labela | Voter label | Total votes | Results | |
| Concordant human votes | ||||||
|
|
COVID-19 pandemic derails Germany’s push for migrant integration-Reuters | 1 | 1.0 | 1 | 3 | Correctly classified by humans |
|
|
Photo shows the last meeting of a Turkish doctor who died due to COVID-19 with his child in Munich | 0 | 1.0 | 1 | 4 | Misclassified by humans |
|
|
3M brings on another lobbying firm | 1 | 1.0 | 1 | 5 | Correctly classified by humans |
|
|
Video shows that the Italian government/Brisbane police used zombie robots/drones to chase their citizen and make them stay home | 0 | 0.0 | 0 | 4 | Correctly classified by humans |
|
|
British vaccine provokes immune response in first human studies | 1 | 0.0 | 0 | 3 | Misclassified by humans |
|
|
This video shows a woman eating a bat soup in Wuhan | 0 | 0.0 | 0 | 5 | Correctly classified by humans |
| Discordant human votes | ||||||
|
|
An emergency department closed in a Spanish hospital | 0 | 0.5 | 1 | 6 | Misclassified by human |
|
|
Majority of Caledonian hotel jobs under review in Edinburgh | 1 | 0.5 | 1 | 4 | Correctly classified by humans |
|
|
England v Ireland: Captain Eoin Morgan relishes 'new journey' in ODI series | 1 | 0.6 | 1 | 5 | Correctly classified by humans |
|
|
Panic scene in Germany with people rushing into a supermarket | 0 | 0.4 | 0 | 5 | Correctly classified by humans |
aVoter soft label is calculated by the number of true labels/total votes.