Table 4.

Coverage Rates of 95% Confidence Intervals When Prevalence Rate is 0.75, and the Number of Categories $q=5$.

		Sample size (n)
$\kappa$ a	Agreement coefficient	10	20	30	40	50	80	100
0.50	Cohen’s kappa	0.923	0.935	0.932	0.936	0.940	0.945	0.948
	Scott’s pi	0.931	0.940	0.933	0.937	0.940	0.945	0.948
	Gwet’s $\mathrm{AC}{}_1$	0.976	0.962	0.943	0.946	0.947	0.949	0.952
	Brennan–Prediger	0.786	0.931	0.932	0.948	0.952	0.947	0.949
	Krippendorff’s alpha	0.928	0.937	0.932	0.935	0.938	0.944	0.945
0.65	Cohen’s kappa	0.897	0.948	0.944	0.939	0.937	0.942	0.948
	Scott’s pi	0.903	0.951	0.946	0.940	0.937	0.942	0.948
	Gwet’s $\mathrm{AC}{}_1$	0.946	0.979	0.961	0.953	0.950	0.951	0.953
	Brennan–Prediger	0.678	0.887	0.930	0.946	0.948	0.952	0.952
	Krippendorff’s alpha	0.907	0.951	0.943	0.939	0.935	0.941	0.946
0.85	Cohen’s kappa	0.649	0.901	0.962	0.972	0.973	0.954	0.948
	Scott’s pi	0.653	0.903	0.962	0.973	0.974	0.954	0.949
	Gwet’s $\mathrm{AC}{}_1$	0.685	0.914	0.971	0.982	0.980	0.958	0.953
	Brennan–Prediger	0.388	0.624	0.766	0.848	0.900	0.937	0.946
	Krippendorff’s alpha	0.673	0.904	0.965	0.975	0.972	0.953	0.947

^a $\kappa$ = hypothesized agreement level, representing the nominal percent of subjects on which the raters agree for cause, as opposed to by chance.