Table 2. Inter-rater agreement and reliability of the assessment of lithium response in the two-stage case-vignette rating procedure: kappa and intra-class correlation analysis.

Assessment of lithium response	First stage of ratings§				Second stage of ratings∼
Dichotomous	κ (95% CI)	z		p	κ (95% CI)	z		p
TS cut off of 8	0.61 (0.33–0.83)	103.50		<0.00001	0.54 (0.28–0.76)	68.06		<0.00001
TS cut off of 7	0.66 (0.38–0.86)	112.18		<0.00001	0.54 (0.31–0.76)	68.71		<0.00001
TS cut off of 6	0.65 (0.36–0.85)	110.52		<0.00001	0.51 (0.29–0.73)	64.54		<0.00001
TS cut off of 5	0.58 (0.29–0.81)	99.23		<0.00001	0.48 (0.25–0.71)	61.25		<0.00001
TS cut off of 4	0.51 (0.20–0.78)	86.83		<0.00001	0.42 (0.18–0.67)	52.94		<0.00001
TS cut off of 3	0.40 (0.10–0.73)	68.25		<0.00001	0.37 (0.13–0.66)	47.46		<0.00001
Continuous	ICC1 single measure (95% CI) *		ICC1 average measure (95% CI) *		ICC2 single measure (95% CI) *		ICC2 average measure (95% CI) *
TS	0.74 (0.59–0.89)		0.99 (0.98–1.00)		0.55 (0.36–0.80)		0.98 (0.96–0.99)
A score	0.66 (0.49–0.85)		0.99 (0.98–1.00)		0.52 (0.33–0.78)		0.98 (0.96–0.99)
Total B score	0.59 (0.41–0.81)		0.99 (0.98–1.00)		0.34 (0.19–0.64)		0.96 (0.92–0.99)
A score if total B score ≤4	0.71 (0.51–0.91)		0.99 (0.99–1.00)		0.75 (0.51–0.96)		0.99 (0.98–1.00)

TS: total score.

ICC: intra-class correlation.

CI: confidence interval.

^*Mixed and random effects models.

^§70 raters.

^¶48 raters.