Table 1.

Designs for assigning coders to subjects IRR studies.

	All subjects rated by multiple coders				Subset of subjects rated by multiple coders
		Coder A	Coder B	Coder C		Coder A	Coder B	Coder C
Design fully crossed	Subject 1	X	X	X	Subject 1	X	X	X
	Subject 2	X	X	X	Subject 2	X
	Subject 3	X	X	X	Subject 3	X	X	X
	Subject 4	X	X	X	Subject 4		X
		Coder A	Coder B	Coder C		Coder A	Coder B	Coder C
Design not fully crossed	Subject 1		X	X	Subject 1	X	X
	Subject 2	X		X	Subject 2	X
	Subject 3		X	X	Subject 3		X	X
	Subject 4	X	X		Subject 4		X

Note: “X” indicates that the ratings were provided by a given coder to the corresponding subject.