Table A.1.

Seven Test Formulas for Assessing Agreement From a Judgment Matrix.

Name and formula	Null hypothesis	Reference
k-to-z tests
$z_{\kappa 1}=\kappa /\sqrt{\mathrm{va}{r}_1(\kappa )}$	H 0: κ = 0	Cohen (1960)
$z_{\kappa 2}=\kappa /\sqrt{\mathrm{va}{r}_2(\kappa )}$	H 0: κ = 0	Fleiss et al. (1969)
Sum-of-z tests
$z_{\Sigma 1}=\sqrt{\frac{N}{k}}(k·r-1)$	$H_0:{\displaystyle \sum _{i=1}^k}{o}_{i,i}=N/k$	von Eye et al. (2006) a
$z_{\Sigma 2}=\sum _i{z}_{i,i}\sqrt{k}$	$H_0:{\displaystyle \sum _{i=1}^k}{o}_{i,i}=\sum _i{e}_{i,i}$	This article
Tests on the structure of the judgment matrix
${\chi }^2={\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{j=1}^k}{z}_{i,j}^2$	$H_0:o_{i,j}=o_{i,*}·o_{*,j}/N$	Pearson (1932)
$Q_A=\frac{{\displaystyle \sum _{i=1}^k}{(z_{i,i}^{+})}^2+{\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{j=1,j\ne i}^k}{(z_{i,j}^{-})}^2}{{\displaystyle \sum _{i=1}^k}{(z_{i,i}^{-})}^2+{\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{j=1,j\ne i}^k}{(z_{i,j}^{+})}^2}$	$H_0:Q_A=1$	This article
$P_A=\frac{Q_A}{1+Q_A}$	$H_0:P_A=1/2$	This article

Note. k is the number of categories and N is the number of observations classified.

^aThis test, based on the indifference principle, is not included in the simulations reported.