Table 3. Intra-expert agreement in plus disease diagnosis by 6 retinopathy of prematurity (ROP) experts from reviewing 7 wide-angle retinal images.

Each expert was initially asked to provide a diagnosis (plus, pre-plus, or neither) while “thinking aloud” to explain their underlying rationale. Experts were subsequently asked to provide a diagnosis after responding to a series of questions about specific features of each image. Intra-expert agreement is represented as absolute agreement and weighted kappa (κ) statistic.

	Intra-expert agreement metric
Expert	Absolute	Weighted κ
1	6/7 (86%)	0.78
2	7/7 (100%)	1.00
3	5/7 (71%)	0.36
4	4/7 (57%)	0.36
5	5/7 (71%)	0.30
6	6/7 (86%)	0.78