Table 2.

MTurk Cohen Kappa Summary.

Cohen Kappa	Normal	Abnormal	SE(Normal)	SE(Abnormal)	P value
Top 10	0.838	0.725	0.019	0.023	< 0.001
Top 20	0.743	0.604	0.016	0.018	< 0.001
Top 30	0.643	0.507	0.015	0.016	< 0.001
Top 40	0.557	0.443	0.013	0.014	< 0.001
Top 50	0.495	0.399	0.012	0.012	< 0.001
Top 100	0.339	0.291	0.009	0.009	< 0.001
Top 200	0.203	0.196	0.006	0.006	0.401
Top 300	0.129	0.139	0.005	0.005	0.101