Figure. Example of differences in diagnostic process among experts reviewing the same image independently.

Experts were asked to provide a diagnosis (plus, pre-plus, or neither) and annotate key findings, while being videotaped “thinking aloud” to describe their reasoning. Videotapes were transcribed, coded, and analyzed to examine qualitative diagnostic process. (A) was diagnosed as plus disease by Expert #1: “…looks like a very low gestational age baby, it's taken quite a long time to get to this stage. There is a lot of arterial tortuosity [annotated], there is a little bit of venous congestion in the superior temporal and superior nasal quadrant, more in the superior half of the retina [annotated]. By definition I think this has to be plus, because it's two quadrants at least, and even the other quadrants aren't normal…” (B) was diagnosed as pre-plus disease by Expert #2: “…there is a lot of tortuosity of the arteries, the veins are about 2 to 1. This could either be a pre-plus eye or a normal variant, depending on a quick look to the periphery. Curiously, there is a lot of tortuosity down here [annotated], it looks like there is disease up here [annotated].” (C) was diagnosed as neither by Expert #4: “…vessels seem to be branching excessively in that region [superonasal area annotated] and some increased tortuosity [superotemporal area annotated] as well, and this vein looks too fat [superotemporal area annotated]. If all the quadrants were like this quadrant [superotemporal] then it would be at least pre-plus and verging on plus, but since it's only one quadrant that's highly questionable. Would not classify it as plus, I could see why some would call it pre-plus … I would call it no plus.”