Comparison of two reference standards in validating two field mydriatic digital photography as a method of screening for diabetic retinopathy

Abstract

Aim: To compare two reference standards when evaluating a method of screening for referable diabetic retinopathy.

Method: Clinics at Oxford and Norwich Hospitals were used in a two centre prospective study of 239 people with diabetes receiving an ophthalmologist’s examination using slit lamp biomicroscopy, seven field 35 mm stereophotography and two field mydriatic digital photography. Patients were selected from those attending clinics when the ophthalmologist and ophthalmic photographer were able to attend. The main outcome measures were the detection of referable diabetic retinopathy as defined by the Gloucestershire adaptation of the European Working Party guidelines.

Results: In comparison with seven field stereophotography, the ophthalmologist’s examination gave a sensitivity of 87.4% (confidence interval 83.5 to 91.5), a specificity of 94.9% (91.5 to 98.3), and a kappa statistic of 0.80. Two field mydriatic digital photography gave a sensitivity of 80.2% (75.2 to 85.2), specificity of 96.2% (93.2 to 99.2), and a kappa statistic of 0.73. In comparison with the ophthalmologist’s examination, two field mydriatic digital photography gave a sensitivity of 82.8% (78.0 to 87.6), specificity of 92.9% (89.6 to 96.2), and a kappa statistic of 0.76. Seven field stereo gave a sensitivity of 96.4% (94.0 to 98.8), a specificity of 82.9% (77.4 to 88.4), and a kappa statistic of 0.80. 15.3% of seven field sets, 1.5% of the two field digital photographs, and none of the ophthalmologist’s examinations were ungradeable.

Conclusion: An ophthalmologist’s examination compares favourably with seven field stereophotography, and two field digital photography performs well against both reference standards.

A mobile digital photographic screening programme was introduced in Gloucestershire in October 1998 covering all 85 general practices. A study was designed to formally evaluate the introduction of the screening programme, which used an ophthalmologist’s examination as a reference standard. There have been very few studies that have compared an ophthalmologist’s examination using slit lamp biomicroscopy with seven field stereophotography. To the authors’ knowledge, no published data exist to show that slit lamp biomicroscopy by an ophthalmologist experienced in retinal examination can compare favourably with seven field stereophotography as a reference standard, when assessing different methods of screening for diabetic retinopathy. The current study, on a preselected group of patients with a higher prevalence of referable retinopathy, was designed to validate the ophthalmologist’s reference standard and two field digital photography against seven field stereophotography. The results are reported in this paper.

SUBJECTS AND METHODS

Study design

A two centre prospective evaluation study of 239 people with diabetes was carried out between December 2000 and July 2001. Subjects were recruited from the Oxford Eye Hospital diabetic retinopathy clinic, or the Bertram Diabetes Centre diabetic and eye clinic in Norwich. A patient information sheet was posted to all patients 1 week before, and informed consent was obtained at the time of their booked outpatient appointment. Ethics committee approval was obtained from the Oxford and Norwich ethics committees.

Screening process

All individuals with diabetes attending the above clinics were considered eligible for inclusion except if they were pregnant, under 18 years of age, known to have learning or significant physical disabilities, or unwell. On arrival, visual acuity was tested using a Snellen chart at 6 metres before dilating both pupils using one drop of tropicamide 1% and phenylephrine 2.5%, repeated not more than three times. In the Norwich clinic phenylephrine 2.5% was restricted to patients with blood pressure less than 180/90 and only given once. Patients were then reviewed and examined in the clinic by their ophthalmic or diabetological team as part of their booked outpatient visit. This was followed by an ophthalmic examination by indirect slit lamp biomicroscopy using a 78D lens and direct ophthalmoscopy, performed by an experienced ophthalmologist (PS). Patients then underwent two field mydriatic digital photography using a 45 degree Canon CR5 retinal camera in Oxford and Canon CR6 camera in Norwich. Both cameras had a Sony three chip video camera capturing an image of 768×568 pixel resolution at 24 bits colour depth. Patients then underwent seven standard field stereoscopic 35 mm slide photography using a Zeiss 30 degree retinal camera in Oxford and a Topcon 50X 35 degree retinal camera in Norwich.

Grading

All three methods of examination were graded independently. The ophthalmoscopic assessments were completed at the time of examination by PHS, the two field digital images by RM, and the seven field stereoscopic 35 mm slide photographs by the Retinopathy Grading Centre, London. The examiner (PHS) was masked regarding each subject’s history of diabetes mellitus, and findings of current or previous ophthalmoscopic examinations although he did, on occasion, have access to the patient’s current visual acuity measurement. The digital and film graders were masked regarding each subject’s history of diabetes mellitus, and findings of current or previous ophthalmoscopic examinations.

The first two methods were graded using the Gloucestershire grading form to define referable diabetic retinopathy (Table 1), and the seven field photographs were graded according to the Modified Airlie House final classification¹ with a comparison table developed for analysis (Table 2). Referable DR was defined as maculopathy, moderate to severe non-proliferative, proliferative and advanced retinopathy as defined by categories 3–6 on the Gloucestershire grading form and/or categories D–G in the comparison table.

Table 1.

The Gloucestershire grading form

Grade	Right eye	Description	Left eye	Grade	Outcome
0		No diabetic retinopathy		0	12/12
1		Minimal non-proliferative DR		1	12/12
2		Mild non-proliferative DR		2
3		Maculopathy			Referral
3a		Haemorrhage <1 DD from foveal centre		3a	“Routine”
3b		Exudates <1 DD from foveal centre		3b, c, d	“Soon”
3c		Groups of exudates (including circinate and plaques within the temporal arcades >1 DD from foveal centre
3d		Reduced VA not corrected by a pinhole likely to be caused by a diabetic macular problem and/or suspected clinically significant macular oedema
4		Moderate to severe non-proliferative DR		4
4a		Multiple cotton wool spots (>5)			Refer “soon”
		And/or multiple haemorrhages
4b		And/or intraretinal microvascular abnormalities (IRMA)
		And/or venous irregularities (beading, reduplication, loops)
5		Proliferative DR		5	Refer “urgent”
		New vessels on the disc (NVD) or
		New vessels elsewhere (NVE)
		Preretinal haemorrhage and/or fibrous tissue
6		Advanced DR		6	Refer “immediate”
		Vitreous haemorrhage, and/or traction/traction detachment, and/or rubeosis iridis
PC		Treated diabetic retinopathy
		Photocoagulation scars anywhere
		Focal
		Sectoral
		Panretinal
		Grid

Table 2.

Comparison table in validation study

7-F	7-F description	Category	2-F/exam	Description	Category
10	No retinopathy	A	0	No retinopathy	A
20	Minimal	B	1a	Minimal	B
35a	Mild with loops	F	1b	Minimal with haem	B
35b	Mild with Quest CWS/VB/IRMA	C	2a	Mild with haem/HE/CWS	C
35c	Mild with haem	C	2b	Mild with Ma/HE	C
35d	Mild with HE = 2	C	3a	Maculop with haem<1DD	D
35e	Mild with HE>3 in 1+	C	3b	Maculop with HE<1DD	D
35f	Mild with CWS	C	3c	Maculop with HE groups	D
43a	Moderate with HMA>3 in 4/5	E	3d	Maculop with reduced VA/CSME	D
43b	Moderate with IRMA = 2 in 1–3	E	4a	Mod/severe with CWS/HMA	E
47a	Mod/severe with both 43	E	4b	Mod/severe with IRMA/VB	F
47b	Mod/severe with IRMA = 2 in 4–5	F	5	Prolif	G
47c	Mod/severe with HMA = 4 in 2–3	E	6	Advanced	G
47d	Mod/severe with VB = 2 in 1	F	U	Unassessable	U
53a	Severe with 2 or more 47s	F
53b	Severe with HMA>4 in 4–5	E	2c	Treated DR	C
53c	Severe with IRMA>3 in 1+	F
53d	Severe with VB = 2 in 2–3	F
61a	Mild prolif with FPE or FPD	G
61b	Mild prolif with NVE = 2 in 1+	G
65a	Mod/prolif with NVE>3 in 1 or NVD = 2	G
65b	Mod/prolif with VH or PRH = 2 in 1	G
71a	HRC with VH or PRH>3 in 1+	G
71b	HRC with NVE>3 in 1+ and VH/PRH	G
71c	HRC with NVD = 2 and VH/PRH	G
71d	HRC with NVD>3	G
75	HRC with NVD>3 and VH/PRH	G
81	Advanced DR	G
88	Unasessable	U
99	Unassessable	U

Image quality for the two field digital was determined using the following criteria:

• Fully assessable—possible to see the small vessels of the temporal arcades with reasonable clarity

• Partially assessable—possible to see the large vessels of the temporal arcades with reasonable clarity

• Not assessable—the large vessels of the temporal arcades are blurred or more than one third of the picture is blurred unless sight threatening retinopathy is detected in the remainder.

The evaluation of the quality of the seven field images was performed to determine gradeability based on strict definitions of field definition, focus/clarity and stereoscopic effect as outlined in the Early Treatment Diabetic Retinopathy Study² (ETDRS) manual of procedures. Seven field sets, including those defined as “ungradeable” were regraded to include the presence of haemorrhage or exudates less than 1 disc diameter from the foveola and any proliferative or advanced retinopathy, identified in the available images. This was carried out to provide comparative data with the Gloucestershire grading procedures for these features.

Statistical methods

The study was designed to include 250 patients, 100 patients having referable retinopathy, in order to achieve a standard error in the estimate of sensitivity no wider than 4%; 150 patient controls were chosen in order to prevent bias in the ophthalmologist’s examination result and in the photographic grading of the seven field photographs. The ophthalmologist and the retinopathy grading centre did not know the percentage of referable retinopathy in the study population but RM, who graded the two field digital images was aware of this percentage, but not the identity of individual patients.

Data were entered onto an Excel spreadsheet and downloaded into SPSS version 10 for data analysis.

Sensitivity, specificity, together with their 95% confidence intervals and kappa values were calculated for the following comparisons:

• The ophthalmologist’s examination compared with seven field stereophotography

• Two field digital photography compared with seven field stereophotography

• Two field digital photography compared with the ophthalmologist’s examination.

Calculations were based on assessable images from the appropriate reference standard method. Unassessable images were then included in the sensitivity calculations when comparing other methods with the reference standard used.

RESULTS

Technical failure rate and image quality

Ophthalmoscopic examination was technically possible in all patients. Determination of retinal status was not possible in six eyes of three patients (1.3%) from the two field mydriatic digital images. A total of 151 eyes (31.6%) of seven field stereo photosets were technically unassessable using the strict quality criteria, hence were not suitable for assignment of ETDRS retinopathy level. However, when the criteria were supplemented by the additional assessment of lesions lying within one disc diameter of the foveola or presence of proliferative retinopathy, the technical failure rate reduced to 73 eyes (15.3%). All of the six eyes that were unassessable on two field digital photography had assessable seven field stereophotography using the above criteria. Five out of the six eyes showed referable features both on seven field stereophotography and the ophthalmologist’s examination.

The reasons that 73 eyes were unassessable when grading the seven field stereophotographs was:

• 13 eyes—fields 1 and/or 2 were of insufficient quality for assessment

• 50 eyes—fields 3–7 were of insufficient quality for assessment

• 5 eyes—fields 1 or 2 and fields 3–7 were of insufficient quality for assessment

• 4 eyes—images were too dark for assessment

• 1 eye—images were absent

Detection of referable diabetic retinopathy

Comparison of the two reference standard methods, an ophthalmologist’s examination against seven field stereo photography, in assessable eyes (Table 3) gave a sensitivity of 87.4% (confidence interval 83.5 to 91.5) and a specificity of 94.9% (CI 91.5 to 98.3). The measure of agreement was a kappa value of 0.80.

Table 3.

Comparison of examination versus seven field

		Ophthalmologist’s examination findings
		No DR or non-referable DR	Referable DR	Total
Reference standard result (seven field stereophotography)	No DR or non-referable DR	150	8	158
		94.9%	5.1%	100%
	Referable DR	31	216	247
		12.6%	87.4%	100%
	Total	181	224	405
		44.7%	55.3%	100%

These calculations were based on 405 eyes that had assessable seven field stereophotogaphs.

A more detailed analysis of this result, which includes the 73 eyes that were not assessable using seven field stereo photography is shown in Table 4.

Table 4.

Detailed comparison of examination versus seven field

	Ophthalmologist’s examination findings
Reference standard result (seven field stereophotography)	Not Gr	No DR	Non-referable DR	Maculopathy	Mod NPDR	Severe NPDR	Prolif and adv	Total
Not gradable		29	30	6		3	5	73
		39.7%	41.1%	8.2%		4.1%	6.8%	100%
No DR		68	2					70
		97.1%	2.9%					100%
Non-referable DR		35	45	4		2	2	88
		39.8%	51.1%	4.5%		2.3%	2.3%	100%
Maculopathy			13	53	3	7	6	82
			15.9%	64.6%	3.7%	8.5%	7.3%	100%
Mod NPDR			13	17	7	9	1	47
			27.7%	36.2%	14.9%	19.1%	2.1%	100%
Severe NPDR			1	7	4	15	3	30
			3.3%	23.3%	13.3%	50.0%	10.0%	100%
Proliferative and			4	14	8	11	51	88
advanced			4.5%	15.9%	9.1%	12.5%	58.0%	100%
Total	0	132	108	101	22	47	68	478
		27.6%	22.6%	21.1%	4.6%	9.8%	14.2%	100%

Comparison of the two field digital photography against seven field stereophotography, in assessable eyes (Table 5) gave a sensitivity of 80.2% (CI 75.2 to 85.2) and a specificity of 96.2% (CI 93.2 to 99.2). The measure of agreement was a kappa value of 0.73.

Table 5.

Comparison of two field versus seven field

		Two field digital photography
		No DR or non-referable DR	Referable DR	Subtotal	Unassessable
Reference standard result (seven field stereo photography)	No DR or non-referable DR	151	6	157	1
		96.2%	3.8%	100%
	Referable DR	48	194	242	5
		19.8%	80.2%	100%
	Total	199	200	399	6
		49.9%	50.1%	100%

These calculations were based on 399 eyes that were gradable by both methods of examination.

Comparison of the two field digital photography against the ophthalmologist’s examination findings, in assessable eyes (Table 6) gave a sensitivity of 82.8% (CI 78.0 to 87.6) and a specificity of 92.9% (CI 89.6 to 96.2). The measure of agreement was a kappa value of 0.76.

Table 6.

Comparison of two field versus examination

		Two field digital photography
		No DR or non-referable DR	Referable DR	Subtotal	Unassessable
Reference standard result (ophthalmologist’s examination findings)	No DR or non-referable DR	222	17	239	1
		92.9%	7.1%	100%
	Referable DR	40	193	233	5
		17.2%	82.8%	100%
	Total	262	210	472	6
		55.5%	44.5%	100%

These calculations were based on 472 eyes that were gradable by both methods of examination.

Comparison of the two reference standard methods with the ophthalmologist’s examination as the main reference standard, in assessable eyes (Table 2) gave a sensitivity of 96.4% (CI 94.0 to 98.8) and a specificity of 82.9% (CI 77.4 to 88.4). The measure of agreement was a kappa value of 0.80.

These calculations were based on 405 eyes that had assessable seven field stereo photographs.

Retrospective examination of the seven field stereophotographs in which there was a difference in grading/classification between the two reference standards

There were eight eyes which the ophthalmologist’s examination had classified as referable retinopathy that had been graded as non-referable by seven field stereophotography.

On looking retrospectively at the photographs of these eight eyes five eyes were classified by the ophthalmologist as referable in whom the appropriate abnormalities were detected retrospectively on the photographs—four with maculopathy and one with referable non-proliferative retinopathy.

There were three patients who had abnormalities noted by the ophthalmologist in the superior retina, two with new vessels and one with IRMA that were not detected retrospectively on the photographs, as these features lay outside the standard fields.

There were 31 eyes that had been graded as referable retinopathy by seven field stereophotography and the ophthalmologist’s examination had classified as non-referable.

Of these, 12 eyes had been graded as having IRMA present on seven field stereophotography. Of these 12 eyes, six had received panretinal photocoagulation and the IRMA had therefore been graded as present between panretinal photocoagulation scars. Of these six eyes, IRMA were retrospectively confirmed on the photographs in five eyes. Of the six eyes that had not received panretinal photocoagulation IRMA were retrospectively confirmed in all six on one of the peripheral fields (two eyes field 6, two field 7, one field 5, and one field 3).

There were a further three eyes in whom the seven field stereophotographs had noted small fibrotic NVE following extensive panretinal photocoagulation (two in field 6 and one in field 3) that the ophthalmologist had classified as stable treated retinopathy (that is, non-referable). These were retrospectively confirmed from the photographs.

There were eight eyes that had been graded as having a haemorrhage <1DD from the foveal centre on seven field stereophotography. In all eight of these eyes the ophthalmologist had classified these as more than two microaneurysms <1DD from the foveal centre. Looking at the photographs retrospectively we encountered difficulties in interpretation of the ETDRS definition of haemorrhage/microaneurysm, which will be discussed below.

There were three eyes that had been graded as having a hard exudate <1DD from the foveal centre on seven field stereophotography. Retrospectively, examination of the photographs confirmed that hard exudate was present in one of these eyes, a single probable drusen was present in one and no hard exudate was seen in the third, although the image quality of this third image was only just within ETDRS standard 14.

There were two eyes that had been graded as having a group of exudates >1DD from the foveal centre on seven field stereophotography. In both of these patients the ophthalmologist had graded non-grouped exudate >1DD from the foveal centre.

Looking at the photographs retrospectively we encountered difficulties with the definition “group of exudates” as discussed below.

There was one eye that had been graded as having a multiple haemorrhage on seven field stereophotography. The ophthalmologist had classified this eye as >two haems >1DD from the foveal centre. Retrospectively it was agreed that multiple haemorrhages were present.

There was one eye that had been graded as having NVD present (ETDRS level 65a) on seven field stereophotography and classified as non-referable on the ophthalmologist’s examination. In retrospect no NVD were visible on the photographs and this patient should have been included within the non-referable group.

DISCUSSION

No previous studies have included a complete set of patients in whom a comparison has been made between seven field stereophotography and an ophthalmologist’s examination using slit lamp biomicroscopy, as illustrated in the examples below.

Comparison with other studies in the literature

Moss et al³ compared ophthalmoscopy with fundus photography in determining the severity of diabetic retinopathy. The ophthalmoscopy method was direct ophthalmoscopy supplemented by indirect ophthalmoscopy if the examiner felt this was necessary. Slit lamp biomicroscopy was not used.

Kinyoun et al⁴ compared indirect ophthalmoscopy by a retina specialist with seven standard field fundus photos but slit lamp biomicroscopy was not used.

Schachat et al⁵ compared the use of clinical examination and fundus photographs in detecting diabetic retinopathy in a population that included people with and without diabetes. Slit lamp biomicroscopy with a 78 dioptre lens, three mirror, or both was performed on 1168 individuals, of whom 9.5% had definite diabetes. The photographic method was two 30 degree fields using 35 mm film (Diabetic Retinopathy Study Standard fields 1 and 2 centred on the disc and macula). They concluded that the clinical examination detected most of the cases of diabetic retinopathy identified by disc and macular photographs read by skilled graders, although there would be an underestimate of prevalence using the clinical examination.

Pugh et al⁶ compared four screening methods in 352 subjects, which included an ophthalmologist’s examination through dilated pupils using direct and indirect ophthalmoscopy and seven field stereophotography. Two centres were used, 10 ophthalmologists performed the examinations, and one of the two centres used slit lamp biomicroscopy with a 90 dioptre lens. The number of subjects who had an examination that included slit lamp biomicroscopy was not stated, nor whether this affected the examination results. Sensitivity, specificity, and positive and negative likelihood ratios were calculated after dichotomising the retinopathy levels into none and mild non-proliferative versus moderate to severe non-proliferative and proliferative. Overall the examination results were poor with sensitivity of 0.33, specificity of 0.99, positive and negative likelihood ratios of 72 and 0.67. Of a total of seven cases of proliferative retinopathy, the ophthalmologist’s examination only detected three.

Reasons for differences between the ophthalmologist and seven field stereophotography in this study

A small number of differences were explained by errors being made by both reference methods.

Definitions of referable retinopathy accounted for a significant number of differences. Particular sources of difficulty were:

• Haemorrhage <1DD from the central fovea was a common source of confusion with microaneurysms <1DD often being graded instead of haemorrhage and vice versa

• The definition of a group of exudate >1DD from the central foveola needs to be clearly defined.

The ophthalmologist in this study differed from seven field stereophotography much more commonly in patients who had received extensive laser treatment. Although the grading form did not differentiate between IRMA in patients who had received panretinal photocoagulation and those who had not, he had not looked for IRMA in the former group. This was because he had considered the lack of new vessels to be a stable treated retina and not a referable eye. The grading form used did not allow for this difference. It perhaps illustrates the difference in performing studies to one’s routine clinical practice.

Performance of two field digital photography

Two field digital photography performed well with sensitivities of >80% and specificities of >92% against both reference standards. The technical failure rate was low at 1.5%.

Reasons for unassessable images in this study

During the clinical examination by PS, no patient was recorded as being unassessable. However, patients were excluded from the study by clinic doctors if they had media opacities. These might otherwise have been technical failures for the ophthalmologist.

A strict evaluation of the quality of the seven field images was performed to determine assessability based on field definition, focus/clarity and stereoscopic effect as outlined in the Early Treatment Diabetic Retinopathy Study (ETDRS) manual of procedures (ETDRS Chapter 18). However, seven field retinal photography in the ETDRS and similar major research trials was only carried out by extensively trained, certified, and constantly monitored photographers. Even within the context of such tightly controlled retinal photography protocols it is not unusual to experience cases where the seven field imaging fails to meet the required quality levels, with 10% technical failure rates being reported within WESDR.⁷ Indeed, authors suggested in the same article that use of such a relatively complicated and difficult protocol may not be entirely necessary and fewer retinal fields may be appropriate in the context of diabetic retinopathy imaging. Technical failure rates for studies that have used a seven field stereo protocol are not routinely reported in the literature, nor do studies report how many attempts were made to achieve an assessable seven field set of photographs.

Within the context of the current study the patients had a much higher prevalence of retinopathy than generally among people with diabetes, and consequently were more likely to have media opacities or to dilate poorly. In addition, the photographers in this study perform this technique relatively infrequently compared to the photographers in the Wisconsin studies,⁷ and only one chance was given to obtain a seven field set (no photographs were repeated). Hence, it proved difficult to obtain consistently high quality results in seven field stereo imaging, with approximately 15% being technically unassessable, even when applying less strict definitions for assessability.

A review of these technically unassessable images indicated that in the majority of cases, there were sufficient images present, and of sufficient photographic quality, for the presence of sight threatening retinopathy features to have been detected, should they have been present. seven field sets, including those defined as “ungradeable,” were therefore regraded to include the presence of haemorrhage or exudates less than one disc diameter from the foveola and any proliferative or advanced retinopathy identified in the available images. This was carried out to provide comparative data with the Gloucestershire grading procedures for these features.

CONCLUSION

The current study has shown that slit lamp biomicroscopy by an ophthalmologist, experienced in retinal examination, can compare favourably with seven field stereophotography as a reference standard when assessing different methods of screening for diabetic retinopathy. There are advantages and disadvantages with both reference methods. There is no hard copy for the ophthalmologist’s slit lamp biomicroscopy and one cannot necessarily conclude that this examination, with different ophthalmologists, will produce consistently high quality results. However, this paper has highlighted the high technical failure rate of seven field stereophotography, which even with the most experienced photographers⁷ can be 10%, and the technical failure rate for this procedure has not often been reported in previous literature.