Table 4.

Reliability of assessment of insightful practice (AIP) questions 1–5

Raters	AIP questions 1–3 (engagement, insight and action) 1–7 scale reliability (G)		AIP question 4 (global assessment) 1–7 scale reliability (G) (ICC)*		AIP question 5 (binary yes/no recommendation on revalidation) reliability (G) (ICC)*
	Internal consistency	Inter-rater	Inter-rater†	Inter-rater (95% CI)‡	Inter-rater	Inter-rater (95% CI)*
1	0.94	0.71	0.66	–	0.54	–
2	0.96	0.83	0.79	(0.68 to 0.88)	0.7	(0.54 to 0.83)
3	0.96	0.88	0.85	(0.78 to 0.91)	0.78	(0.69 to 0.86)
4	0.97	0.91	0.89	(0.84 to 0.93)	0.83	(0.75 to 0.89)
5	0.97	0.92	0.91	(0.87 to 0.94)	0.86	(0.80 to 0.91)
6	0.97	0.94	0.92	(0.89 to 0.95)	0.88	(0.83 to 0.92)

Reliabilities greater than 0.8, as required for high-stakes assessment, are given in bold.⁹

^*Intraclass correlation coefficients (ICCs) are G coefficients when you have a one facet design (rater).

^†Inter-rater reliability is the extent to which one rater's assessments (or when based on multiple raters, the average of raters' assessments) are predictive of another rater's assessments.

^‡95% CIs for reliabilities (ICCs) were calculated using Fisher's Z_R transformation which is dependent on raters (k) with a denominator value of (k-1), and so cannot be calculated when there is only one rater.⁹