Table 3.

Coverage Rates of 95% Confidence Intervals When Prevalence Rate is 0.75, and the Number of Categories $q=4$.

		Sample size (n)
$\kappa$ a	Agreement coefficient	10	20	30	40	50	80	100
0.50	Cohen’s kappa	0.916	0.932	0.930	0.938	0.942	0.941	0.945
	Scott’s pi	0.922	0.935	0.933	0.940	0.944	0.941	0.946
	Gwet’s $\mathrm{AC}{}_1$	0.968	0.959	0.943	0.948	0.950	0.948	0.951
	Brennan–Prediger	0.821	0.942	0.932	0.949	0.955	0.945	0.948
	Krippendorff’s alpha	0.925	0.936	0.933	0.938	0.942	0.940	0.944
0.65	Cohen’s kappa	0.888	0.938	0.938	0.940	0.942	0.945	0.941
	Scott’s pi	0.897	0.942	0.941	0.942	0.942	0.946	0.942
	Gwet’s $\mathrm{AC}{}_1$	0.942	0.971	0.957	0.951	0.952	0.953	0.946
	Brennan–Prediger	0.726	0.914	0.940	0.949	0.953	0.952	0.942
	Krippendorff’s alpha	0.904	0.946	0.942	0.939	0.942	0.944	0.939
0.85	Cohen’s kappa	0.652	0.886	0.948	0.961	0.964	0.949	0.946
	Scott’s pi	0.656	0.888	0.950	0.964	0.965	0.950	0.946
	Gwet’s $\mathrm{AC}{}_1$	0.690	0.901	0.962	0.971	0.970	0.955	0.950
	Brennan–Prediger	0.438	0.685	0.809	0.881	0.919	0.945	0.948
	Krippendorff’s alpha	0.677	0.891	0.954	0.966	0.965	0.950	0.945

^a $\kappa$ = hypothesized agreement level, representing the nominal percent of subjects on which the raters agree for cause, as opposed to by chance.