Table 5:

Inter-Rater Reliability

	Observed Agreement	Cohen’s Kappa (95% CI)	Prevalence Adjusted Bias Adjusted Kappa
Inter-Rater Reliability	0.71	0.45 (0.41–0.50)	0.43
IRR STARD Questions 1
1	0.80	0.59 (0.36–0.82)	0.60
2	0.96	0.00 (0–0)	0.91
3	0.76	0.25 (−0.03 to 0.53)	0.51
4	0.71	0.32 (0.09–0.55)	0.42
5	0.29	0.05 (−0.01 to 0.10)	−0.42
6	0.71	0.06 (−0.15 to 0.27)	0.42
7	0.53	0.23 (0.08–0.38)	0.07
8	0.49	0.10 (−0.01 to 0.21)	−0.02
9	0.51	0.13 (0.0–0.26)	0.02
10	0.56	0.21 (0.20–0.39)	0.11
11	0.71	0.41 <0.15–0.67)	0.42
12	0.69	0.39 (0.14–0.64)	0.38
13	0.82	0.39 (0.05–0.74)	0.64
14	0.89	0.71 (0.47–0.95)	0.78
15	0.67	0.36 (0.16–0.56)	0.33
16	0.64	0.28 (0.06–0.50)	0.29
17	0.78	0.26 (−0.06 to 0.58)	0.56
18	0.71	0.43 (0.20–0.67)	0.42
19	0.69	0.36 (0.09–0.64)	0.38
20	0.93	−0.03 (−0.07 to 0.01)	0.87
21	0.80	0.61 (0.39–0.83)	0.60
22	0.62	0.23 (0.05–0.42)	0.24
23	0.76	0.35 (0.08–0.63)	0.51
24	0.80	0.30 (−0.04 to 0.63)	0.60
25	1.00	1.00 (1–1)	1.00