Table 5.

Inter-rater agreement for real-time assessment: Cohen’s kappa with 95% confidence intervals and percentage agreement for criteria (n = 20 except * n = 18)

Test	Criteria	Left-hand side	Kappa (95% CI)	Right-hand side	Kappa (95% CI)	Criteria agreement (%)
1	A	.86	(.59-1.00)	.88	(.64-1.00)	95.0
	B	.55	(.19-.90)	.38	(-.02-.78)	75.0
	C	.74	(.40-1.00)	.74	(.40-1.00)	90.0
	D	.58	(.19-.98)	.70	(.39-1.00)	85.0
	E	.66	(.31-1.00)	.43	(.01-.85)	80.0
2	A	.22	(-.33-.76)	.00	N/A	82.5
	B	.57	(.14-1.00)	.32	(-.27-.90)	85.0
	C	.12	(-.28-0.52)	.27	(-.17-.71)	75.0
	D	.88	(.64-1.00)	.76	(.45-1.00)	92.5
	E	.76	(.45-1.00)	.38	(-.07-.82)	82.5
3	A		.77 (.35-1.00)			95.0
	B		.80 (.54-1.00)			90.0
	C	.00	N/A	.34 *	(-.17-0.85)	84.2
4	A		1.00 (1.00-1.00)			100.0
	B	N/A	N/A	N/A	N/A	100.0
	C	1.00	(1.00-1.00)	.64	(.01-1.00)	97.5
	D	.69	(.38-1.00)	.68	(.35-1.00)	85.0
5	A	N/A	N/A	.00	N/A	97.5
	B	.63	(.24-1.00)	.74	(.40-1.00)	87.5
	C	.38	(-.07-.82)	.39	(-.03-.81)	75.0
	D	.24	(-.17-.65)	.13	(-.33-.58)	67.5
7	A	.00	N/A	.00	N/A	95.0
	B	.77	(.35-1.00)	.44	(-.20-1.00)	92.5
	C	.77	(.35-1.00)	.86	(.59-1.00)	95.0
	D		.66 (.308-1.00)			85.0
	E	.89	(.67-1.00)	.58	(.22-.95)	87.5
8	A		N/A N/A			100.0
	B	.80	(.54-1.00)	.89	(.67-1.00)	92.5
	C	N/A	N/A	N/A	N/A	100.0
	D	.69	(.29-1.00)	.78	(.50-1.00)	90.0
9	A	.29	(-.14-.73)	.63	(.24-1.00)	77.5
	B	.68	(.35-1.00)	.60	(.26-.94)	82.5
	C	.55	(.19-.90)	.89	(.67-1.00)	87.5
	D	.38	(-.12-.87)	.15	(-.28-.57)	72.5
	E	.35	(-.07-.76)	.30	(-.06-.66)	67.5
10	A	.44	(-.20-1.00)	.000	N/A	92.5
	B	.39	(-.09-.86)	.615	(.15-1.00)	85.0
	C	.50	(.12-.88)	.600	(.25-.95)	77.5
	D	.64	(.01-1.00)	1.000	(1.00-1.00)	97.5
	E	.60	(.24-.95)	.634	.28-.99)	82.5

NB. Test 6 from the Performance Matrix was not included in the study, for reasons explained in the methodology section of the text