Table 3.

Intra-rater-, Inter-rater reliability and accuracy measured by two questionnaire rounds of 20 case vignettes each

		Years of working experience in infectious disease surveillance	Intra-rater reliability (κ, 95%-CI)	Inter-rater reliability per hospital (κ, 95%-CI)#	Accuracy (%, First round/Second round)	Accuracy in determination of presence of anastomotic leakage, n = 4. (%, First round/Second round)	Accuracy in determination of absence of anastomotic leakage, n = 16 (%, First round/Second round)
Hospital A	Rater 1	4–5	0.78 (0.46–1.00)	0.61 (0.23–0.83)	95/85	75/100	93/87
Hospital A	Rater 2	2–3	0.95 (0.74–1.00)		85/80	100/75	93/93
Hospital B	Rater 1	11–15	0.83 (0.49–0.99)	0.72 (0.42–1.00)	80/85	75/100	93/93
Hospital B	Rater 2	6–10	0.73 (0.44–1.00)		95/90	100/100	93/93
Hospital C	Rater 1	11–15	1.00 (1.00–1.00)	0.94 (0.75–1.00)	90/90	75/75	93/93
Hospital C	Rater 2	11–15	0.94 (0.76–1.00)		90/95	75/75	93/93
Hospital D	Rater 1	0–1	0.75 (0.47–1.00)	0.69 (0.36–0.92)	90/85	100/100	93/87
Hospital D	Rater 2	4–5	0.89 (0.72–1.00)		90/95	100/100	93/87
Hospital E*	Rater 1	2–3	0.89 (0.59–1.00)	0.65 (0.38–0.92)	80/80	100/100	93/93
Hospital E*	Rater 2	4–5	0.73 (0.46–1.00)		85/70	100/100	93/81
Hospital F*	Rater 1	2–3	0.79 (0.57–1.00)	0.69 (0.34–0.92)	90/90	100/100	87/81
Hospital F*	Rater 2	11–15	0.89 (0.59–1.00)		90/90	100/100	87/87
Hospital G*	Rater 1	0–1	0.79 (0.55–1.00)	0.84 (0.61–1.00)	90/90	100/100	87/93
Hospital G*	Rater 2	> 25	0.94 (0.75–1.00)		95/90	100/100	93/93

κ, Cohen’s Kappa coefficient; 95% CI, 95% confidence interval, n, number

^*Hospitals that started surveillance for the purpose of this study

^#Inter-rater reliability was calculated from the second round questionnaire case vignettes