Fig. 4.

Users’ learning curves stratified by question level of clarity. Legend. “E-IRA” = Expert Interrater Reliability. Candidate IED images were grouped by agreeance of experts to show the learning curve of users for candidate IEDs of different clarities using the same equation used in Fig. 1 (a(n/N)^b). Colors assigned to users in Fig. 1 based on quantile stayed the same across all plots in Fig. 2. [A, B, C] Show false positive rating assigned by crowd vs the number of questions answered for spikes that received 8/8, 7/8, and 6/8 votes by experts to be non-spikes (clear non-IEDs). [D, E, F] Show sensitivity of ratings assigned by crowd vs the number of question answered for spikes that received 8/8, 7/8, and 6/8 votes by experts to be spikes (clear IEDs).