Validity, Usefulness and Cost of RETeval System for Diabetic Retinopathy Screening

Abstract

Purpose

We studied the validity, usefulness, and relative cost to detect diabetic retinopathy (DR) and sight-threatening DR (STDR) by using a hand-held electrophysiologic tool compared to digital fundus photography.

Method

Patients with diabetes attending the screening unit of King Khaled Eye Specialist Hospital, Riyadh, Saudi Arabia were evaluated by “RETeval”, Amsler grid, and digital dilated fundus photography. Fundus images were evaluated by a retina specialist to determine grade of DR. The sensitivity and specificity of STDR and DR screening compared to photography were calculated, as well as “RETeval” combined with Amsler grid testing. The area under the curve (AUC) of “RETeval” screening outcome was calculated.

Result

We analyzed data of 400 diabetic patients. The prevalence of DR of any grade was 48.8% (95% confidence interval [CI], 43.9–53.7) while the prevalence of STDR was 27% (95% CI, 22.6–31.4). The outcome of RETeval test was “fail” (based on 20 μV or more amplitude of electrophysiologic spikes) in 351 (87.8%; 95% CI, 84.5–91.0) eyes. The sensitivity of the device was 95.4% and the specificity was 17.5%. Thus, the sensitivity of sequential testing with RETeval and Amsler grid test was 30.1% and the specificity was 80.1%. The AUCs for STDR and DR in general were 76.6% and 50.6%, respectively.

Conclusions

“RETeval” is a rapid screening device with excellent sensitivity for detecting STDR. It has potential as a first level screening tool to detect patients who require further evaluation.

Translational Relevance

Retinal function, such as electrophysiology, can be used as a new concept for screening for DR.

In 2014, 387 million people were estimated to be living with diabetes, an alarming number that is predicted to rise to 592 million within the next 20 years.¹ This projected epidemic of diabetic retinopathy (DR) needs a public health approach in the coming years to prevent visual disabilities.² Early detection and prompt action delay development as well as progression of DR.³ Unfortunately, many cases of DR are detected in later stages after the development of symptoms; to attain early diagnosis and treatment of sight-threatening DR (STDR), annual DR screening of patients with type II diabetes is recommended.⁴ The existing work force is unable to undertake annual DR screening and, hence, innovative manners of DR screening are recommended. Telemedicine via transfer of digital images of the retina is one of the gold standards for remote screening of DR.⁵ However, action-oriented DR screening models must be developed for the unique environments of underprivileged and developing countries.

The cost and portability of digital fundus cameras limit their application for DR screening. Although a number of models for mobile DR screening are available, the current coverage is limited in large countries, such as India and China.^6–8 A cost-effective, portable instrument that can be used by nonophthalmic personnel to detect STDR through an undilated pupil would be welcomed by public health authorities tasked with preventing visual disability due to DR.

Different aspects of retinal function, such as microperimetry and electrophysiologic responses, are altered at different stages of DR. The electroretinography (ERG) full-field flicker systems measure response from the cone system and are representative of the whole retina. They are useful primarily in differentiating STDR from nonproliferative stages.^9,10

A new handheld electrophysiologic flicker ERG recording device, RETeval (LKC Technologies, Inc., Gaithersburg, MD; Welch Allyn, Inc. Skaneateles Falls, NY) has been developed.¹¹ It simultaneously measures the full-field flicker ERG and pupillary response to light. It has been shown to perform well in nonmydriatic eyes¹² and cataracts of less than Grade 2 on the Emery-Little classification.^13,14 To the best of our knowledge, only one study has been published to validate this new tool for DR screening.¹⁵ The purpose of our study was to evaluate the RETeval as a screening tool in (sight-threatening) DR compared to conventional digital retinal photographs. We further evaluated the ease of use and cost-effectiveness of DR screening using RETeval. We also studied the added predictive value of the Amsler central field of vision chart in enhancing the validity of DR screening.

Methods

Subjects

This cross-sectional, single-site, noninterventional study of consecutive patients with diabetes mellitus was undertaken at the screening clinic of the King Khaled Eye Specialist Hospital (KKESH), a tertiary eye care hospital of central Saudi Arabia. The Institutional Research Board approved this study (#1497-P) and the study adhered to the tenets of the Declaration of Helsinki. The study was undertaken between January 2015 and April 2016. Diabetic patients with eye problem(s) seeking ophthalmic advice in our hospital were the potential study population. Written informed consent was obtained from all patients who elected to participate in this research project. Personnel trained in fundus photography and the use of the RETeval device were the testers. The eyes with severe corneal opacities and cataracts grade 2 and above were excluded from the study, as well as eyes with poor quality of retinal image obtained by fundus camera.

At the time of enrollment, we obtained demographic- and diabetes-related information, including age, sex, duration of diabetes, glycemic control based on last hemoglobin A1c (HbA1c) report and the year of last DR screening.

Testing

A nonmydriatic digital funds camera (TRC NW 300; Topcon, Livermore, CA) was used to obtain three images (macula including the optic disc, nasal, and temporal). The images were reviewed by retina specialist (who was masked to ERG results) to grade the DR. The grading followed the guidelines of the American Academy of Ophthalmology.¹⁶ We further grouped the severe nonproliferative DR (sNPDR), proliferative DR (PDR), and diabetic macular edema (DME) together as STDR. The elapsed time for obtaining the digital images was noted.

The technical description of the RETeval device has been reported previously.^12–15 The testing was performed by a medical student trained in the use of the device. Following the manufacturer's recommendations, we used the following steps for DR screening using RETeval (Fig. 1).¹⁷ The spectacles of the patient were removed. A sensor strip was applied on the cheek inferior to the lateral half of the lower eyelid. The patient was instructed not to talk during the test. The lead was connected to the sensor strip. The machine was switched on and with the fellow eye covered, the patient was asked to focus on the red beam projected from the device. The elapsed time from placing the strip until the result was displayed on the LCD was noted. The automated display information on test “pass” or “fail,” as well as amplitude of the wave was noted. The right eye was tested first followed by the left eye. Testing took place in a room with no background light, which has been shown to influence detection of STDR.¹⁸ The sensor strips were disposed of to avoid rescreening of other patients using the used strips. The calibration phase followed prior reports with this device.¹⁵

Figure 1.

Picture of electrophysiologic tool RETeval for DR screening.

Based on a 20 mV amplitude with a constant implicit time, as fixed by the manufacturer, the machine was set to determine the test as passed (≥20 mV) or failed (<20 mV). To compare the validity of RETeval to the gold standard (interpretation of digital retinal image by retina specialist to grade DR) we used “pass” and “fail” results from RETeval to the presence and absence of STDR as diagnosed by a retina specialist review of fundus photography. We also used amplitude with a constant implicit time, as fixed by the manufacturer, as a continuous outcome variable to study the validity of RETeval by plotting separate curves for eyes with STDR as identified by the retina specialist using digital images.

We then tested the presence of metamorphopsia using reverse Amsler grid test. Any abnormality noted was defined as a “failed” test while no abnormality perceived in viewing the grid was considered as a “passed” test. The time taken to perform this test also was noted.

In patients with false-positive results by RETeval, a secondary chart review was performed to determine the presence of other ocular diseases that might explain an abnormal ERG, such as glaucoma, optic atrophy, or chorioretinal abnormalities.^19,20

Outcomes

The primary outcome of the study was assessment of validity of the instrument as a screening tool in DR and STDR along with assessment of the instrument's sensitivity, specificity, and positive and negative predictive values. This was calculated for the instrument alone and in combination with Amsler grid testing. We also evaluated the time required to perform the testing, the subjects' preferred screening device, and relative absolute cost of this instrument in DR screening.

Samples Size Calculation

We assumed that the sensitivity of RETeval in detecting STDR is 83% compared to digital photography.¹⁵ To have 5% error margin, 95% confidence interval (CI) of a cross-sectional validity study with a clustering effect of 1.5, we estimated a sample size of 326 diabetic persons to include in our study. In view of possible media opacities that prevent photographic assessment of the DR stage, we increased the sample by 33% and the final sample was planned as 400 eyes of 400 diabetic persons. The eye with worse stage of DR from each participant was included in the study.

Statistics

The data were collected in a Microsoft XL spreadsheet (Microsoft, Corp., Redmond, WA). Data were converted into a spreadsheet of Statistical Package for Social Studies (SPSS 22) (IBM, Chicago, IL). For qualitative outcomes, like pass and failed tests, we calculated the frequencies and the percentage proportions. For continuous variables, like amplitude of ERG, we calculated the mean and standard deviation provided the distribution of the variable was normal. The validity of the DR and STDR screening by RETeval was assessed by comparing its results to the outcomes of a retina specialist's evaluation of presence and absence of STDR and staging of DR. The sensitivity, specificity, and positive and negative predictive values of DR screening by RETeval, were calculated. The amplitude of the ERG also was used as a continuous outcome variable and the area under the curve (AUC) was plotted to determine the test validity for STDR and DR screening. We added the outcomes of Amsler screen test into the respective RETeval screening results to calculate the validity of sequential screening by two methods.²¹ The independent factors, such as duration of diabetes, sex, age, and glycemic control, were tested for correlation. We calculated 95% CIs of different validity parameters. Two sided P values were used for statistical validation.

Results

Subjects

We enrolled 400 eyes of diabetic patients in this study. Their demographic and diabetic profile is given in Table 1. The glycemic control was poor in two-thirds of diabetic patients (n = 293, 73%). The median duration of diabetes was >10 years. Slightly more than two-thirds of them had undergone DR screening at least once in the past (n = 279, 69.8%).

Table 1.

Demographic Profile of Diabetic Patients in the Study

Testing

There were no technical failures of RETeval data acquisition in any of the eyes tested. Table 2 shows the magnitude of different stages of DR per eye and in the worse eye based on digital fundus camera photography interpreted by a retina specialist. The prevalence of DR of any grade was 48.8% (95% CI, 43.9–53.7) while the prevalence of STDR was 27% (95% CI, 22.6–31.4).

Table 2.

Status of DR Based on Digital Photography Evaluation by Retina Specialist

The outcome of RETeval test was “fail” (based on 20 μV or more amplitude of electrophysiologic spikes) in 351 (87.8%; 95% CI, 84.5–91.0) eyes. The amplitude of electrophysiologic spikes (μV) as measured by RETeval in eyes with worse stage DR of diabetic patients by different grades of DR is presented in Figure 2. If a “fail” test was defined as 27 μV or more amplitude of electrophysiologic spikes, all cases of severe NPDR, PDR, DME, and STDR would be included in the failed test group.

Figure 2.

Amplitude of ERG spikes as displayed by RETeval while screening of eyes with different stages of DR. In x axis different stages of DR are given while y axis shows amplitude of electrophysiologic spike (mV) as measured by RETeval.

The mean Reteval measured amplitude was 25.1 ± 4.9 mV among 263 males and 25.5 ± 5.1 mV among 135 females. The difference of means was 0.34 mV (95% CI, −0.7; 1.39; P = 0.53). The information on glycemic control was available in 400 participants. The mean Reteval measured amplitude was 23.4 ± 4.3 mV in 107 cases with HbA1c <7 and 25.9 ± 5.1 mV among 293 diabetics with HbA1c >7. The difference of means was 2.44 in the two groups (95% CI, 1.4; +3.4), which was statistically significant (P < 0.001).

Validity

Table 3 presents validity parameters of STDR screening by RETeval. The sensitivity was 95.4% and the specificity was 17.5%. The addition of Amsler grid testing in the validity parameters of RETeval STDR screening revealed 34 cases that failed the Amsler test among 108 with STDR based on digital fundus photo evaluation. There were 44 cases that failed Amsler test among 190 without STDR based on the digital fundus photo evaluation. Thus, the sensitivity of sequential testing with RETeval and Amsler grid test was 30.1% and the specificity was 80.1%.

Table 3.

Validity of RETeval-DR in Screening for STDR Compared to Digital Fundus Photographs Evaluated by Retina Specialist

Table 4 presents validity parameters of overall DR screening by RETeval. The positive predictive value was 60.7% for any stage of DR. The areas under the receiving operating curves (ROC) for diabetics who had a positive or negative RETeval and digital fundus outcome were plotted for STDR (Fig. 3), DR (Fig. 4), and PDR + SNPDR (Fig. 5). The ROCs were 76.6%, 50.6%, and 74.4%, respectively (Figs. 3–5).

Table 4.

Validity of RETeval-DR in Screening for DR of All Grades Compared to Digital Fundus Photographs Evaluated by Retina Specialist

Figure 3.

The ROC curve for eyes with STDR (proliferative DR and/or diabetic macular edema) as diagnosed by RETeval screening and confirmed by digital fundus photography. The x axis shows fraction of 1-specificity while y axis shows fraction of sensitivity. The area under the red line depicts AUC.

Figure 4.

The ROC for eyes with nonproliferative and proliferative DR as diagnosed by RETeval screening and confirmed by digital fundus photography. The x axis shows fraction of 1-specificity while y axis shows fraction of sensitivity. The area under the red line depicts AUC.

Figure 5.

The ROC for eyes with any stage of DR as diagnosed by RETeval screening and confirmed by digital fundus photography. The x axis shows fraction of 1-specificity while y axis shows fraction of sensitivity. The area under the red line depicts AUC.

Usefulness

RETeval was the preferred method of DR screening among 250 (63.5%) diabetic patients whereas 91 (22.8%) preferred digital fundus photography. A total of 59 (14.5%) patients could not decide their preference for DR screening procedures. In 56 (14%) cases that could not be evaluated using a digital fundus camera due to media opacities, poor cooperation, or a small pupil, RETeval was useful for DR screening. Of the 156 false-positive cases, documentation of subsequent complete examination was available to determine the presence of other posterior segment pathologies in 119 (76%). Of the 119 cases with available information on other ophthalmic conditions, 17 (15%) were diagnosed with glaucoma, scars consistent with inactive chorioretinitis, optic atrophy, or retinal dystrophies.

Testing Time

The mean time of testing DR status of two eyes in a diabetic person using RETeval was 5.3 ± 2.1 minutes, compared to a median time of 15 minutes (25% quartile of 10 minutes) by digital photography. Amsler grid testing is instantaneous.

Relative Cost

The capital cost of RETeval was approximately US $4000, and the recurrent cost of 400 strips for testing diabetics was US $4800. The time spent in performing this test was 10 minutes per patient. The capital cost of a digital fundus camera was US $29,333. The time spent by a medical student to take retinal images was 20 minutes per patient, and the retina specialist required approximately 1 minute to evaluate six images (three for each eye) of a diabetic person.

Discussion

This study aimed to independently investigate the usefulness of RETeval for DR screening with evidence of validity, patient acceptance, and cost compared to the conventional screening method using color fundus photography. To our knowledge, this is the first study to use this device in the Middle Eastern diabetic population. Our study demonstrates that the RETeval portable ERG device may provide effective first level DR screening with a high sensitivity of disease detection. This is similar to results of a previous report that compared the device with Early Treatment of Diabetic Retinopathy Study 7 (ETDRS 7)–standard field photography.¹⁵ Additionally, we reported that the device is easy to use by a nonophthalmologist, is accepted by patients, requires half of the time of fundus photography, and, depending upon the use may be cost-effective.

As a stand-alone test we observed a high sensitivity and low specificity for DR. We attributed the low specificity to higher false-positive rates, which occur with screening devices. Of the false-positives detected by RETeval screening, 15% included other posterior segment pathologies in addition to early stages of DR; it can be proposed that these patients also may benefit from comprehensive ophthalmic evaluation for diseases, such as glaucoma, and, therefore, they are not false-positives from a holistic patient care standpoint. For comparison, frequency doubling technology (FDT) in glaucoma screening has been noted to have good sensitivity but with a high rate of false-positives.²² Among populations with diabetes, FDT has been demonstrated to be useful not only for glaucoma detection but for other posterior segment pathologies, including DR.²³ Similarly, outcomes of RETeval screening for STDR may be altered by the presence of other pathologies that decrease the specificity for the disease for which we are primarily screening.

Regarding the manufacturer's recommendations and a prior report¹⁵ for the cut-off value for determining positive or negative results of the test, our sub-analysis suggested that if an amplitude of 27 MV (rather than 20 MV per the manufacturer) is chosen as the pass/fail parameter, STDR could be detected with increased specificity and minimal effect on sensitivity. By combining ERG screening with the Amsler grid test, the sensitivity of the test declined and specificity increased.

Digital images demonstrate anatomic derangement of retinal tissue due to DR, while RETeval screening provides information on functional abnormalities of the retina (and pupillary response). Thus, the outcomes of both DR screening methods could be considered as complementary to each other. Compared to the previous report,¹⁵ we found no technical failures during the screening process in any of the eyes. It performed well in nonmydriatic eyes and eyes with early cataracts. We have not addressed the reproducibility of ERG testing in our cohort. However, a prior study using the same instrument reported high intra-class correlation of measurements with the device.¹⁵

Although the capital cost of this DR screening procedure is low, recurrent costs should be considered before accepting it as a universal screening procedure in developing countries. We propose that it could be used by nonophthalmic personnel at diabetes centers or primary health centers where diabetic patients visit their physician periodically, with referral for fundus photography or ophthalmic examination for patients with an abnormal screening result. A feasibility study to undertake DR screening using RETeval at primary health centers by nonophthalmic providers is further recommended.

While the study was performed at a tertiary eye care facility, the testing itself was performed at the screening clinic where eligibility for tertiary care is determined. It is likely that this population includes a bias towards patients with retinopathy and other ocular conditions, but it still is plausible that the outcomes of the study can be applicable to the general diabetic population in the Middle East. Additionally, the patients with ocular comorbidities other than DR and including refractive errors, for instance, also are part of the screening system. Thus, our population of diabetics includes a large variety of patients representing a diabetic population. As such, truly asymptomatic diabetic patients may be underrepresented.

In summary, we found the RETeval device to be a time-efficient and sensitive first-line tool to screen for DR in a cohort with diabetes mellitus. Further studies are warranted to determine the cost-effectiveness of this tool as part of a comprehensive DR screening program.