Table 3.

Agreement between evaluators on the value of the explanatory dialog. The values in parentheses are the results obtained when the five categories are collapsed to three.

CBR DST explanatory dialog: Did you find the supporting dialog useful?
Evaluator 5 point scale (3 point scale)	Observed Agreement	Agreement expected by chance alone	κ	95% C.I.	Interpretation
End users & Expert 1	87.2% (66.0%)	83.6% (56.7%)	0.21 (0.21)	0.03 – 0.40 (0.05 – 0.38)	Fair (Fair)
End users & Expert 2	84.1% (62.0%)	83.4% (58.7%)	0.04 (0.08)	(0.13 – 0.22) (−0.10 – 0.26)	Poor (Poor)
Expert 1 & Expert 2	78.3% (60.3%)	79.2% (56.9%)	−0.04 (0.08)	−0.20 – 0.12 (−0.09 – 0.25)	None (Poor)