Table 3.

Results of inter-coder reliability.

	Pilot study				Replication
	C1–C2		C1–C2		C3–C4		4C
Items	Agreement	95% CI	Agreement	95% CI	Agreement	95% CI	Agreement	95% CI
(1) Inclusion/exclusion criteria	a0.684∗∗	[0.292, 1]	a0.798∗∗	[0.533, 1]	a0.9∗∗	[0.71, 1]	a0.851∗∗	[0.707, 0.995]
(2) Methodology/design	a0.252	[-0.062, 0.566]	a0.861∗∗	[0.277, 1]	a1∗∗	[1, 1]	a0.931∗∗	[0.639, 1]
(3) Attrition	b0.505	[0.078, 0.898]	b0.653∗	[0.463, 0.962]	c0.943∗∗	[0.772, 0.986]	d0.79∗∗	[0.617, 0.963]
(4) Attrition between groups	c0.952∗∗	[0.618, 0.994]	b0.866	[0.326, 1]	c0.991∗∗	[0.629, 1]	d0.849∗∗	[0.478, 1]
(5) Exclusions after	b-0.206	[-0.871, 0.551]	b1∗∗	[0.137, 0.998]	b1∗∗	[0.476, 1]	d0.775∗∗	[0.306, 1]
(6) Follow-up	b0.522	[-0.133, 0.802]	b0.783∗∗	[0.949, 0.994]	b0.963∗∗	[0.976, 0.997]	d0.76∗∗	[0.5, 1]
(7) Occasions of measurement	a0.486∗	[0.131, 0.841]	a0.653∗∗	[0.32, 0.986]	a1∗∗	[1, 1]	a0.827∗∗	[0.66, 0.994]
(8) Pre/post measures	a0.592∗∗	[0.173, 1]	a0.714∗	[0.212, 1]	a1∗∗	[1, 1]	a0.73∗∗	[0.606, 0.854]
(9) Dependent variables	a0.577∗∗	[0.25, 0.904]	a0.512∗∗	[0.038, 0.986]	a0.857∗∗	[0.588, 1]	a0.745∗∗	[0.313, 1]
(10) Control techniques	a0.706∗∗	[0.323, 1]	a0.667	[0.104, 1]	a0.744∗	[0.281, 1]	a0.767∗∗	[0.192, 1]
(11) Construct definition	a0.047	[-0.18, 0.274]	a0.861∗∗	[0.277, 1]	a0.77∗∗	[0.271, 1]	a0.772∗∗	[0.438, 1]
(12) Imputing missing data	a0.571∗	[0.081, 1]	a0.5	[0.014, 0.986]	a1∗∗	[1, 1]	a0.841∗∗	[0.581, 1]

Items appear in abbreviated form; the whole and final version (after including improvements derived from the pilot study) can be found in the Appendix. C1–C2 = reliability results between coders 1 and 2; C3–C4 = reliability results between coders 3 and 4; 4C = reliability results across the four coders combined.

^aCohen’s κ coefficient; ^bSpearman’s ρ coefficient; ^cPearson’s r coefficient; ^dKrippendorff’s α coefficient.

^∗p < 0.05; ^∗∗p < 0.01.