Agreement Among Glaucoma Specialists in Assessing Progressive Disc Changes From Photographs in Open-Angle Glaucoma Patients

Abstract

PURPOSE

To determine the agreement among glaucoma specialists in assessing progressive disc changes from photographs in a cohort of patients with glaucomatous visual field loss.

DESIGN

Retrospective cohort study.

METHODS

Three glaucoma specialists, masked to chronological sequence, examined pairs of optic disc stereophotographs to determine whether the appearance of the optic disc had changed. Eyes for which the observers disagreed were adjudicated to reach a consensus about which discs had changed over time.

RESULTS

Sequential stereophotographs, separated in time by a median of 26 months (range, five to 50), from 164 eyes of 111 patients were analyzed. Among the three observers, the number of interpretable discs judged to have changed was 11 of 155 (7%) for Observer 1, 17 of 155 (11%) for Observer 2, and 44 of 155 (28%) for Observer 3 (κ = 0.20). Sixty-six eyes (43%) required adjudication. After adjudication, the consensus was that 10 discs had changed, six eyes in which the disc was worse in the later photograph and four eyes in which the disc was judged to appear more glaucomatous in the earlier photograph.

CONCLUSION

Interobserver agreement among glaucoma specialists in judging progressive optic disc change from stereophotographs was slight to fair. After masked adjudication, in 40% of the cases in which the optic disc appeared to have progressed in glaucoma severity, the photograph of the “worse” optic disc was in fact taken at the start of the study. Caution must be exercised when using disc change on photographs as the “gold standard” for diagnosing open-angle glaucoma or determining its progression.

Determining when Visual Field (VF) or optic disc progression has occurred is a major clinical challenge in the management of the open-angle glaucoma (OAG) patient. One of the common methods recommended for detecting progression is the observation of the optic disc over time for changes in its appearance. The American Academy of Ophthalmology’s Preferred Practice Pattern for Primary Open-Angle Glaucoma¹ recommends that optic disc evaluation and documentation be performed at frequent intervals. The Preferred Practice Pattern states that the optic disc appearance can be recorded photographically or using an image analyzer. The presumption is that a change in the optic disc appearance over time signifies deterioration. Some investigators have even argued that progressive disc worsening should be a required criterion to define OAG in clinical research.²

The utility of using optic disc deterioration as a criterion for OAG progression is dependent upon the reliability with which the deterioration can be assessed. In the absence of a corresponding change in the VF, there is no way to externally validate the deterioration. However, if different observers routinely concur on the presence or absence of optic disc deterioration, this agreement may provide support for a finding of progression based on disc assessment.

Previous studies have varied in their assessment of observer agreement regarding changes in the optic disc from photographs. Some investigators have found high interobserver correlation.³ Observers reading photographs in the context of major clinical trials are generally reported to have low interobserver variability,^4,5 whereas others have reported much greater variability.⁶

The Glaucoma Imaging Longitudinal Study (GILS) was conducted to determine whether macular and nerve fiber layer thickness could predict progressive VF loss in OAG patients. As part of the GILS protocol, annual stereophotographs of the optic discs were obtained for the study cohort.We used these data to address the degree of expert agreement on the change in the optic disc in the context of a longitudinal study.

METHODS

Annual photographs of the optic discs of subjects enrolled in the GILS study were taken with a Zeiss fundus camera (Zeiss Inc, Jena, Germany) as 30-degree images and printed on color transparencies. The photographer captured a stereopair by aligning the camera at the maximal separation possible between two sequential photographs. All eyes with at least two sets of stereophotographs were selected for inclusion. For each eye, the pairs of optic disc photographs most separated in time were arranged in plastic sheets, with the earlier and later photographs placed randomly (using a random number table) as the “top” and “bottom” photo. All photographs were devoid of other clinical information and when two eyes of the same subject were enrolled in the study, the photographs from each of the two eyes were viewed separately.

Three glaucoma specialists (H.Q., H.J., and D.F.) independently viewed each sheet containing the earliest and latest pairs of stereo disc photographs. The slide set was read once by each expert. The observer was asked to comment on whether the quality of the photos was sufficient to make an assessment of change. For pairs judged to be gradable, two choices were allowed: one pair worse than the other (for either top/bottom or vice versa) or no difference. When one pair looked worse than the other, the observer specified which pair was worse (top or bottom) and whether the nature of the difference was a generalized enlargement of the cup, a focal rim thinning, a deepening of the cup, or a change other than one of these choices. The observers also noted the presence and location by clock hour of optic disc hemorrhages. No firm criteria for the definition of optic disc hemorrhage were given to the observers prior to grading.

The findings of the three observers were tabulated and all eyes in which the assessment had not been unanimous were adjudicated. For adjudication, the three observers viewed the photographs together, still masked to the temporal sequence, and concluded whether change in the disc photos had or had not occurred and, where applicable, which pair was worse (top or bottom). In addition, all disc photographs in which one or more observers had noted a disc hemorrhage were adjudicated for confirmation.

The code for the temporal sequence of the photographs was broken, and from the clinical records of the patients with disc change the following data were abstracted: age, gender, race, intraocular pressure (IOP) on the day of photography, VFs (Humphrey Field Analyzer II [HFA II], Zeiss Humphrey, San Leandro, California, USA; SITA [Swedish interactive threshold algorithm] Fast algorithm), and scanning laser tomography (Heidelberg Retina Tomograph II [HRT II], Heidelberg Engineering, Heidelberg, Germany) done on all visits between (and including) the day of the two photos.

When there were at least three follow-up VFs, the sequence of SITA Fast VFs performed concurrently with the disc photographs was analyzed with the glaucoma progression analysis (GPA) incorporated into the software of the HFA II. Those eyes with a GPA determination of “possible progression” or “likely progression” were considered to have VF progression. The sequences of HRT II images obtained concurrently with the disc photographs were analyzed by the method of Fayers and associates.⁷ This technique uses the interobserver, intervisit rim area repeatability coefficients to define the criterion for change in each HRT disc sector. For progression to have occurred, a change has to be present in at least two sectors, and this change has to be confirmed in at least one of the next two imaging sessions.

The κ statistic was computed to determine the degree of agreement among all three observers.⁸ Pairwise κ values were not computed.

RESULTS

Stereophotographs from at least two different time points were available for 164 eyes of 111 subjects from the entire GILS cohort of 312 eyes of 205 patients. The quality of at least one of the two stereopairs of nine (5.5%) of the 164 eyes was deemed insufficient for analysis by at least two of the three readers, leaving 155 eyes of 103 subjects that were graded by the three glaucoma specialists.

The age of the subjects whose disc photographs were analyzed was 68 ± 12 years with a range of 36 to 92 years. The linear cup-to-disc ratio was 0.69 ± 17 as assessed on the HRT II. The median interval between the sets of photographs was 26 months, with a range of five to 50 months. All three observers agreed that there was no change in 89 eyes (57.4%) of 45 patients and that there was a definite change in two eyes (1.3%) of two patients; in the remaining 64 eyes (41.3%) of 56 patients there was disagreement (Figure).

Figure. 1.

Flow chart describing characterization of photographs of glaucomatous discs in the Glaucoma Imaging Longitudinal Study.

There was a large difference in the proportion of discs assessed to have changed, ranging from 11 of 155 (7%) for Observer 1, 17 of 155 (11%) for Observer 2, and 44 of 155 (28%) for Observer 3. Furthermore, in a substantial proportion of the photographs (36% to 63%) deemed to have changed, the earlier photograph had been judged to show more severe damage (ie, the disc appearance improved over time). The κ value comparing the initial (pre-adjudication) agreement of all three observers was 0.20 (poor to fair) (Table 1). To determine if the agreement varied with the amount of disc damage, we divided the discs into four groups based upon their linear cup-to-disc ratio measured by the HRT II. The κ values were similar for the four groups (0.17, 0.21, 0.22, 0.24 for linear cup-to-disc ratios of 0 ≤ 0.6, 0.6 ≤ 0.7, 0.7 ≤ 0.8, and >0.8, respectively). When change was believed to have occurred, the vast majority of the changes were listed as either concentric or focal rim thinning, at about equal frequency, and few changes consisted of cup deepening (Table 2).

TABLE 1.

Observation of Change in 155 Photographs of Glaucomatous Discs in the Glaucoma Imaging Longitudinal Study

	Observer 1	Observer 2	Observer 3
Later photo worse	7	9	16
Earlier photo worse	4	8	28
Total rated “changed”	11	17	44

TABLE 2.

Morphologic Description of Change in Photographs of Glaucomatous Discs in the Glaucoma Imaging Longitudinal Study (Number of Eyes)

	Observer 1	Observer 2	Observer 3	Total
Concentric enlargement	5	5	22	32
Focal rim thinning	4	8	13	25
Deepening cup	2	2	3	7
Cannot specify	0	2	6	8
Total	11	17	44	72

After adjudication of the 64 eyes with disagreement, the observers reached a consensus that in 56 of the eyes there was no change and that in the other eight eyes, one of the two pairs of stereophotographs showed more damage than the other. We examined these eight eyes plus the two originally deemed to show change by all three observers. Of these 10 eyes (6.4% of 155) the “worse” disc was in the later photograph (ie, true disc worsening) for six eyes of six patients and in the earlier photograph (ie, disc improvement) for four eyes of four patients. The incidence of disc worsening was 1.8% per year (six of 155 eyes in 26 months). The incidence of disc improvement was 1.2% (four of 155 eyes in 26 months). The clinical courses of the 10 eyes with consensus change are shown in Table 3.

TABLE 3.

Characteristics of Eyes in Which There was Consensus of Disc Change by Photography in the Glaucoma Imaging Longitudinal Study

Age	Race	Gender	IOP First Photo (mm Hg)	IOP Second Photo (mm Hg)	Time Between Photos (mos)	VF Worse?a	HRT Worse?b
Disc worsening
71	White	Male	17	21	25	No	No
78	White	Male	22	22	37	No	Yes
68	White	Female	10	17	41	No	Yes
67	White	Male	11	18	26	No	No
60	White	Male	15	12	24	No	No
69c	South Asian	Male	18	12	30	Possible	Yes
Disc improving
70	White	Male	14	14	24	No	No, improved
68	White	Male	11	17	35	No	No
54d	White	Male	21	10	32	No	No
63e	White	Female	13	10	22	Insufficient data	No

HRT = Heidelberg Retinal Tomography; IOP = intraocular pressure; mos = months; VF = visual field.

^aBy glaucoma progression analysis.

^bBy confirmed changes in rim area in at least two sectors.

^cUnderwent trabeculectomy, revision, and second trabeculectomy between the two photos. All three readers independently judged the discas worse.

^dThis eye had a trabeculectomy between the two photographs.

^eA disc hemorrhage was seen on the earlier photo.

Of the six eyes with disc worsening, two showed progressive worsening by HRT II criteria, one showed progressive worsening by both HRT II and GPA analysis, and the remaining three had neither HRT II nor GPA analysis worsening. This can be compared to the HRT II and VF findings in the 145 eyes in which the disc was not assessed to have changed (Table 4). In those 114 eyes with good-quality HRT II studies and sufficient follow-up for analysis (leftmost three columns of Table 4), 14 (12%) had worsening, six (5%) showed improvement, and 94 (82%) were unchanged. In the 83 eyes with analyzable GPA (first three rows of Table 4), 13 (16%) had possible progression, three (4%) had likely progression, and 67 (80%) were unchanged. It should be noted that GPA does not have a classification of “improved.”

TABLE 4.

Heidelberg Retina Tomograph II and Swedish Interactive Threshold Algorithm Fast Glaucoma Progression Analysis Results in Eyes Without Disc Changes by Photography in the Glaucoma Imaging Longitudinal Study, N (% with reference to HRT II, columns)

HRT II
	No Change	Worsening	Improvement	Insufficient # of Studies (<4)	Insufficient Quality (TSD ≥50)	Total
GPA
No change	45 (48)	8 (57)	2 (33)	5 (26)	7 (58)	67 (46)
Possible progression	8 (9)	4 (29)	1 (17)	0 (0)	0 (0)	13 (9)
Likely progression	2 (2)	0 (0)	0	0 (0)	1 (8)	3 (2)
GPA results not available	39 (41)	2 (14)	3 (50)	14 (74)	4 (33)	62 (43)
Total	94 (100)	14 (100)	6 (100)	19 (100)	12 (100)	145 (100)

GPA = glaucoma progression analysis; HRT II = Heidelberg Retina Tomograph II; SITA = Swedish interactive threshold algorithm; TSD = topographical standard deviation.

Among the four eyes graded as disc improvement, one eye had undergone trabeculectomy between photographs with a dramatic IOP lowering from 21 to 10 mm Hg (Table 3). However, in the other three “improved” discs, IOP was not dramatically lower at the time of the second photograph. One of the four “improved” eyes was also classified as “improved” by the HRT II, but this was not the eye with the IOP lowering.

Optic disc hemorrhages were noted in seven eyes by Observer 3, in five of the same seven eyes by Observer 1, and in six of the same seven eyes by Observer 2. The κ value was 0.89. After adjudication, consensus was reached that disc hemorrhages were present in all seven eyes, for a prevalence of seven of 310 (2.3%). Of four eyes with disc hemorrhage present at baseline, one subsequently had worsening by HRT only, one had worsening by both HRT and GPA, one had improvement by disc photography, and one had no change on disc photography, HRT, or GPA (Table 5).

TABLE 5.

Disc Photograph, Heidelberg Retina Tomograph II, and Visual Field Findings in Eyes With Disc Hemorrhages Present in Disc Photographs in the Glaucoma Imaging Longitudinal Study

ID Number	Eye	Time of Disc Hemorrhage (months after baseline)	Disc Change on Photos (months after baseline)	HRT II Change (months after baseline)	Visual Field Progression by GPA (months after baseline)
1112	OD	0	No	Worsened (24)	No
2110	OS	0	No	Worsened (5)	Yes (31)
4214	OS	0	Improved (22)	No	No
4218	OS	0	No	No	No
4221	OS	12	No	No	No
221	OD	12	No	No	No
313	OS	12	No	No	No

GPA = glaucoma progression analysis; HRT II = Heidelberg Retina Tomograph II; ID = identification; OD = right eye; OS = left eye.

DISCUSSION

We found a lack of agreement among three glau-coma experts when they independently assessed disc change over time in a cohort of OAG patients with established VF loss. Not only was the κ value for agreement poor, but the proportion of discs deemed changed varied widely among the three observers. With adjudication, many discs that one or more observers had considered changed were classified as unchanged, suggesting that many of the changes observed were “soft calls.” It should be noted that, unlike reading centers for clinical trials, the observers in this study were not trained on a standard set of photographs of progressing and nonprogressing discs, which could have improved the concordance. However, clinicians do not routinely train in this regard, and so our experience is likely closer to what occurs in clinical practice.

Coleman and associates reported similarly discouraging data in a study in which they studied intraobserver and interobserver variability in assessing optic disc changes in OAG suspects using nonsimultaneous stereoscopic disc photographs.⁹ When a third glaucoma specialist examined sets of photographs of 18 discs judged progressive by two glaucoma specialists, he conceded progression in only 13. Although the study of Coleman and associates was limited to OAG suspects, in whose optic discs it would arguably be easier to detect change than in our more advanced cohort, the investigators still concluded that examination of stereophotographs was no better than a comparison of drawings of the optic disc.

One can reach similar conclusions about the difficulty of assessing change from the Ocular Hypertension Treatment Study (OHTS).⁴ The OHTS employed a reading center to assess disc changes and the protocol involved confirmation of disc changes in a repeated set of nonsimultaneous stereophotographs. Despite the requirement for high-quality photographs in the OHTS, and despite the potential to observe changes in these relatively healthy optic discs with much rim to lose, the ability of the graders to reproducibly grade discs as progressed ranged from 64% to 81% from year to year; ie, one-fifth to one-third of the time, an optic disc that had been judged progressive on one reading was subsequently judged to be nonprogressive. On the other hand, Caprioli and associates³ and the European Glaucoma Prevention Study Group,⁵ both using nonsimul-taneous stereophotography, reported much higher agreement among observers, expressed as a κ value, than the previously cited studies. It should be noted, however, that in the former study, only two out of three readers were required to be in agreement, and the proportion of discs about which all three readers agreed is not given. In the latter study, the rate of agreement between three readers is between 65% and 75%. In studies like these it is important to present not only the κ value, but also the proportion of eyes for which there was any disagreement among readers.

Four of the 10 disc changes observed represented improvement, rather than worsening. One of the four discs that improved was in an eye with substantial IOP lowering following a trabeculectomy, a well-documented phenomenon in which apparent topographic return of disc tissues toward the normal position occurs when IOP is lowered.^10,11 Similar topographic improvement has been seen in experimental monkey eyes subjected to IOP change and judged by disc photography or laser imaging.¹² We found no explanation for the improvement observed in the other three improved eyes, but apparent disc improvement over time is not a new finding. Azuara-Blanco and associates⁶ reported that 13% of all changes that they observed were improvements. Caprioli and associates³ reported that four of 33, or 12%, of changes were improvements. In a study comparing masked to unmasked reading of sequential, stereoscopic disc photographs, Altangerel and associates¹³ reported that three glaucoma experts judged 22 of 76 changes (29%) as improved when masked to chronology as opposed to nine of 110 (8%) when the chronology was known.

The interobserver agreement for the detection of disc hemorrhage was high, but some disc hemorrhages were missed by two of the observers. The OHTS investigators reported that many disc hemorrhages identified by graders on stereophotographs were not reported on study exam forms by clinicians caring for the patient during dilated fundus exams.¹⁴ The interobserver agreement in the detection of disc hemorrhage in the OHTS was not reported. It is generally accepted clinically that image analyzers of the optic nerve and retina nerve fiber layer are poor at detecting disc hemorrhages.

One potential limitation of our study is the use of nonsimultaneous, as opposed to simultaneous, stereophotographs. It is possible that if the observers had viewed these discs with a consistent level of stereopsis, they might have had higher concordance. However, we are not aware of any published reports comparing the assessment of optic disc change in nonsimultaneous vs simultaneous disc photography. Our experience in the Baltimore Eye Survey was that simultaneous fundus photography achieved a lower rate of gradable photographs than did nonsimultaneous methods because of the need for a larger pupil diameter to obtain simultaneous images.¹⁵

Another limitation is the relatively short interval between the two photographs evaluated in this study. With a longer time period there might have been larger disc changes that might have resulted in better agreement among the specialists. Another caveat about interpreting our results is that we studied a population of patients who already had established VF loss. This population had relatively severe optic nerve damage at baseline and hence it might have been particularly difficult to detect change in these discs. Although our results might have differed had we examined photographs from eyes with less optic disc damage, we found no difference in interobserver agreement when we stratified our cohort by amount of disc damage.

Finally, we did not assess the intraobserver variability by having each grader regrade the photographs. While regrading may have provided additional information to explain some cases of disagreement, the overall degree of intergrader disagreement would have remained the same. It was not possible to conduct a regrading after the adjudication had been performed, since a valid adjudication process can only be carried out if the graders are unaware of the expected rate of disc change from the cohort.

A single image of the optic disc has limited predictive power to identify glaucomatous optic neuropathy. For this reason Coleman and associates¹⁶ and Medeiros and associates² have suggested that progressive glaucomatous change in the optic disc appearance is the best reference standard for glaucoma diagnosis at present. Although documented progressive disc change seems more likely to be a valid measure of glaucoma than a single disc evaluation, our work suggests that it is a less-than-perfect reference standard, since interobserver agreement is far from ideal, and rates of adjudicated change in a direction considered unlikely are not infrequent. Until computerized imaging systems are shown to be superior to serial disc photographs, serial disc photographs are probably the best standard that we have, but we must be aware of their limitations.

Biography

Henry D. Jampel, MD, MHS, is the Odd Fellows and Rebekahs Professor of Ophthalmology at the Wilmer Eye Institute of the Johns Hopkins University School of Medicine. Dr Jampel is the Associate Editor-in-Chief of Ophthalmology, serves on the Glaucoma Preferred Practice Pattern committee of the American Academy of Ophthalmology, and is Co-Director of the Glaucoma Subspecialty Day 2008 at the annual meeting of the Academy.