Accuracy of Screening for Diabetic Retinopathy by Family Physicians

Abstract

BACKGROUND We wanted to examine the accuracy of family physicians’ screening for diabetic retinopathy using standardized criteria and a nonmydriatic ophthalmoscope.

METHODS Eleven family physicians assessed 28 standardized patients with diabetes mellitus using the PanOptic ophthalmoscope. Their assessments of whether the patients required referral to an ophthalmologist were compared with the reference standard of retinal diagrams.

RESULTS The mean sensitivity for the family physicians was 87% (95% confidence interval [CI], 83%–91%) with a specificity of 57% (95% CI, 46%–68%). Overall agreement was moderate, with a mean κ = .43 (95% CI, 0.39%–0.47%).

CONCLUSIONS Using standardized criteria and a nonmydriatic ophthalmoscope, family physicians were fairly accurate in screening patients for diabetic retinopathy. Whereas this technique is not sufficiently accurate to replace routine referral for all patients with diabetes, it can be used to improve care for those patients who fail to get routine eye screenings.

INTRODUCTION

It is widely recommended that all persons with diabetes mellitus should be regularly screened for diabetic retinopathy.¹ Most adults with diabetes, however, do not receive this screening.^2–4 One reason is that most persons with diabetes receive their care in primary care settings,⁵ and most primary care physicians have neither the expertise nor the equipment to screen accurately for retinopathy.⁶ Two recent developments have increased the potential for primary care physicians to screen for diabetic retinopathy with greater accuracy and efficiency. First, a simple prediction rule has been developed that accurately predicts vision-threatening diabetic retinopathy by viewing lesions in a limited number of retinal fields.⁷ Second, a new-generation ophthalmoscope has been developed that allows viewing of these retinal fields without dilation. The purpose of this study was to test the accuracy of screening for diabetic retinopathy by family physicians using these new tools.

METHODS

This study compared the predictive accuracy of screening by family physicians against an ophthalmologist’s assessment using retinal diagrams.

The study was conducted in New Castle County, Del. Eleven family physicians were recruited from the Delaware Academy of Family Physicians. Twenty-eight standardized patients from a local ophthalmologist’s practice were selected to represent a spectrum of retinal abnormalities from no disease to severe retinopathy.

Training

The family physicians first participated in a 4-hour training program given by an education specialist and a retinal specialist (DRC, HML). The physicians were trained in the eye watch screening criteria (EWSC), which are based on examination of 2 standard retinal fields.⁷ A positive screening test is defined as finding a hard exudate within one disc diameter of the macula (field 2) or 3 or more hemorrhages or microaneurysms temporal to the macula (field 3). The physicians were also trained to use the PanOptic scope, which provides a view of the retina 3 to 5 times that of a standard direct ophthalmoscope without dilation.⁸ This view would theoretically allow the physician to see both fields 2 and 3, which are usually not visible using a standard direct ophthalmoscope without dilation. Finally, physicians used the EWSC and PanOptic scope to assess 7 patients with diabetic retinopathy. This assessment was based on the decision of whether the patient should be referred to an eye specialist for further evaluation. The assessments were categorized into 1 of 3 categories: (1) refer because the criteria for a positive screening test (as defined above) are met, (2) do not refer because the criteria for a positive screening test are not met, or (3) refer because the physician was unable to evaluate the field (eg, because of small pupils or lens opacity). After the workshop the family physicians were given a PanOptic ophthalmoscope to use on their own clinic patients and asked to use the EWSC criteria they had learned.

Evaluation

All 11 family physicians returned 4 weeks later for an evaluation session. Each family physician assessed each of 28 standardized study patients (none of whom overlapped with the 7 patients used in the training session). The physicians, who were unaware of the patients’ history of retinopathy, were asked to make a screening assessment, which was the same as that used in the training program. The training ophthalmologist also assessed each patient using the same screening algorithm and PanOptic ophthalmoscope.

One month after the evaluation session, the ophthalmologist was asked to review the medical charts of the 28 study patients. Based on the most recent retinal diagrams in their medical record (all of which were within the previous 12 months), he made a screening assessment for each retinal field using the EWSC. All diagrams were based on comprehensive retinal examinations that included indirect ophthalmoscopy.

Data Analysis

For this analysis, the screening assessments were combined into 2 categories: refer (either because of a positive screening test or inability to evaluate), or do not refer (a patient was considered to be referred if any of the retinal fields was assessed as “refer”). These categories were chosen because the clinical decision of whether to refer a patient for further evaluation was considered to be the critical outcome of primary care screening. For each family physician, the sensitivity and specificity were measured against the reference standard of the retinal diagrams. Overall agreement was measured for each physician against the reference standard, using the kappa statistic. Because not all physicians rated all patients (1 physician missed 1 patient, and a second physician missed 2 patients), statistics were weighted accordingly to the number of patients rated by each physician. The weighted mean sensitivity, specificity, and kappa were calculated for the group of 11 family physicians, with 9% confidence intervals (CIs).

RESULTS

According to the reference standard of retinal diagrams and using the EWSC, 75% of patients should have been referred for further assessment. The mean referral rate by the family physicians was 76%, with a range of 41% to 93% (Table 1 ▶). The weighted mean sensitivity of the family physicians’ referral assessments was 87% (95% CI, 83%–91%). Specificity was lower, with a weighted mean of 57% (95% CI, 46%–68%).

Table 1.

Accuracy of Physician’s Assessments Compared With Reference Standard

Family Physician	Sensitivity	Specificity	Overall Agreement	κ	Referral Rate
100	85.7	71.4	82.1	0.55	71.4
101	100.0	28.6	82.1	0.38	92.9
102	95.2	71.4	89.3	0.70	78.6
103	90.0	16.7	73.1	0.08	88.5
104	55.0	100.0	66.7	0.39	40.7
105	95.2	28.6	78.6	0.29	89.3
106	71.4	100.0	78.6	0.57	53.6
107	90.5	14.3	71.4	0.06	89.3
108	100.0	28.6	82.1	0.34	92.9
109	85.7	71.4	82.1	0.55	71.4
110	85.7	85.7	85.7	0.65	67.9
Ophthalmologist	81.0	71.4	78.6	0.48	67.9

The overall agreement of each physician with the reference standard is shown in the table. Kappa statistic for the 11 physicians ranged from 0.06 to 0.70, with a weighted mean of 0.43 (95% CI 0, 0.39–0.47). Overall agreement between the ophthalmologist’s assessment using the PanOptic ophthalmoscope and the reference standard retinal diagrams was similar, with a κ = 0.48.

DISCUSSION

The nonmydriatic ophthalmoscope and standardized screening criteria resulted in a relatively high sensitivity for family physicians screening for diabetic retinopathy. The mean sensitivity for a correct decision to refer for further evaluation was 87% when measured against an ophthalmologist’s assessment using retinal diagrams, with a specificity of 57%. That the sensitivity was higher than the specificity is expected for a screening test, since false-negative examinations would cause more problems than false-positive examinations.

This level of accuracy represents an improvement when compared with techniques that are currently used by primary care physicians. Previous studies have shown that primary care physicians using standard direct ophthalmoscopy correctly identify less than 50% of serious retinopathy even when they dilate the eyes.⁶ When using standard direct ophthalmoscopy without dilation, even those experienced in ophthalmoscopy have rates for correct assessment of only about 50%.⁹

There are several limitations that must be considered in interpreting the results of this study. First, we evaluated accuracy after 1 training workshop and 1 month of practice; it is likely that physicians would probably be more accurate after using the tools for a longer period. That the ophthalmologist (who was also new to the tools) had a sensitivity result similar that of the family physicians supports this possibility. Also, while we examined the accuracy of a single examination, most diabetic patients see their primary care physician several times a year; it is likely that the test sensitivity would be higher if multiple examinations are conducted during the course of a year. Additionally, the standardized patients in this study might not represent typical patients in primary care; they were selected from an ophthalmology practice and would likely have more retinal disease, as well as more abnormalities that would lead to an assessment of unable to evaluate (eg, cataracts). Finally, we examined a small group of self-selected family physicians in Delaware; future studies would be needed to determine whether the results could be replicated in a larger population of physicians who are not self-selected and who represent other specialties.

Despite these limitations, this study shows one promising way to improve screening for diabetic retinopathy. Currently, many diabetic patients do not get screened adequately for retinopathy, partly because screening by primary care physicians is neither accurate nor efficient using currently available techniques. Our study suggests one technique that primary care physicians can use to screen for diabetic retinopathy with greater accuracy and efficiency. Using this technique, family physicians can correctly refer most diabetic patients who are likely to have serious retinopathy. This technique will probably not replace the current standard of having all diabetic patients evaluated by an eye specialist, because a false- negative rate of 13% might not be acceptable. The technique, however, might at least improve care for those who currently do not regularly see eye specialists; having an abnormality identified by their primary care physician may motivate patients to seek further evaluation by an ophthalmologist. If the results of this study can be replicated in larger populations, then this technique may be one way for primary care physicians to improve care for their patients with diabetes.