Interdevice Agreement between a Smartphone and a Commercial Pupillometer

Abstract

Background:

The reliability of dynamic pupillometry parameters varies from one pupillometer to another, making it difficult to standardize the values for any particular device. Hence, further studies are required to evaluate the agreement of various pupillometer devices and explore their utility in routine clinical settings.

Aim:

This study sought to evaluate the agreement between smartphone and commercial pupillometer measurements in routine clinical settings.

Methods:

The study included pupillary measurements obtained by a single investigator from 100 healthy participants (200 eyes) with each pupillometer. Pupillary measurements taken by a smartphone pupillometry application (reflex pupillary light reflex analyzer by Brightlamp [Indianapolis, IN, USA]) were compared with a commercial pupillometer (neurological pupil index-200, NeurOptics Inc., Irvine, USA).

Results:

The comparison of descriptive statistics revealed a statistically significant difference between the smartphone and commercial pupillometers for various parameters, including maximum diameter, minimum diameter, constriction velocity (CV), maximum CV, and dilatation velocity (P < 0.05), except for latency (P = 0.36). The intraclass correlation coefficient revealed poor agreement between the two devices (<0.50).

Conclusion:

The measurements by smartphone pupillometry application were found to be unreliable, indicating that they may not be an ideal substitute for commercial pupillometers in their present form in the Indian population. Further studies with larger sample size as well as improvements in the processing and interpretation of the measurements by the software, are needed to determine its utility in routine clinical settings.

Introduction

Pupil evaluation is an important part of neuropsychologic and ophthalmologic evaluation. Pupillary reactions are controlled by the opposing actions of the sphincter and dilator muscles of the iris, providing an indirect measure of both central and autonomic nervous systems. Conventional assessment of pupil is done manually using a penlight, which can be quite subjective and qualitative with several shortcomings such as a lack of standardization, the requirement for deliberate training, and poor interobserver reliability.

Recently, automatic portable pupillometers have been introduced, which enable clinicians to objectively and quantitatively examine the dynamics of pupil response. An image or sequence of images of the iris and pupil are captured and measured using a noninvasive device called a pupillometer. It assists in capturing the pupillary light reflex (PLR) or the pupil’s initial constriction and subsequent dilation in response to a light stimulus. Various commercially available pupillometers have been shown to provide useful information in a variety of clinical situations, which include Alzheimer’s and Parkinson’s disease,[1] isolated third nerve palsy,[2] traumatic brain injury,[3,4,5,6] autism spectrum disorder,[7,8,9,10] evaluation of alcohol and drug intoxication,[11,12,13,14] Horner’s syndrome,[15] and diabetic retinopathy.[16,17] Commercial pupillometers use infrared light to assess dynamic pupillary characteristics, which broadens their applicability to patients with a range of iris colors and lighting conditions. They can be used in both routine clinical settings and bedside diagnostics because they are lightweight, convenient to use, rechargeable, and allow simple data transfer. However, their availability and relatively high cost limit their use in resource-limited settings.

With the nearly universal availability of smartphones, they have found innovative uses in various clinical scenarios.[18,19,20,21] Recently, several smartphone-based pupillometer applications have been introduced to assess the absolute pupil measurements as well as the relative change in pupil size using the device’s built-in flash. Smartphone pupillometers have the advantage of being more affordable, portable, and accessible. Although previous studies have compared the smartphones and commercial infrared pupillometers in different population groups, including the USA,[22,23] Italy,[24] and Chicago;[25] there are no such data available for the Indian population. There have been conflicting results about the agreement between the two devices. While some studies[23,24,25] have reported a high degree of similarity, the study done by McKay et al.[22] demonstrated poor agreement. The current study was conducted with the aim of evaluating the agreement between a smartphone and a commercial pupillometer by comparing their measurements in the Indian population in routine clinical settings.

Methods

After approval by the institutional ethics committee, this observational study was conducted in the ophthalmology outpatient department of the institute in accordance with the tenets of the Declaration of Helsinki. One hundred healthy, adult participants (200 eyes) were consecutively recruited. Written informed consent was obtained from all the participants, following which they underwent routine ophthalmic examination that included best-corrected visual acuity (BCVA) testing using the Snellen chart, intraocular pressure measurement, eye movements, slit-lamp examination, and fundus examination. Participants were considered eligible, if they had a BCVA ≥6/6 according to Snellen’s chart and were free of any physical, mental, neurological, or ophthalmological disease other than spherical or cylindrical refractive errors. Participants with a history of use of any systemic or topical medications affecting pupil size, iris and/or pupil abnormalities, head or orbital trauma, or previous ocular or orbital surgery were excluded. The participants were advised to come for pupillometer measurements 2 days after the initial screening.

For pupillometry measurements, participants were examined on a comfortable chair in routine clinical settings and asked to keep their eyes closed for 5 min. A single investigator took pupillary measurements of both eyes with the help of a smartphone pupillometer. Subsequently, the participants were asked to close their eyes for about 5 min again, following which the same investigator took measurements of both eyes with the help of a commercial pupillometer.

Smartphone pupillometry in the current study was performed using the Reflex PLR analyzer application by Brightlamp Version 3.12.4 (313) (Indianapolis, IN, USA), which was downloaded from the Apple Store on an iPhone 12 mini. After creating a patient profile, the application was ready for testing [Figure 1a]. Despite the wide range of available settings, flash mode with the illuminance set at 3% and the duration set at 3 s was chosen for the current study as per the manufacturer’s recommendation, given that the majority of Indians have dark iris. The participants were asked to look straight at a distant target and keep their eyes open, whereas measurements were being taken. Initially, the Reflex application was opened, the right eye measurement was obtained by holding the smartphone at approximately 10 cm from the participant’s eye level, and a circular start button displayed on the application screen was pressed when alignment was achieved with the cutout of the eye image [Figure 1b-d]. The other eye was neither stimulated by the flash nor covered. The entire process was then repeated on the left eye. The device records the videos of the pupillary reactions, which were subsequently reviewed, and in the case of any blink artifacts or incomplete measurements, the reading was taken again, and only high-quality images were included and analyzed.

Figure 1.

A screenshot of the patient data recording interface of the smartphone pupillometer application (a); an image of measurement being carried out (b); an image showing the precise eye cutout and the circular button as displayed on the application screen while recording (c); output of various pupillary parameters measured on the result screen by the device (pupillogram) (d)

The participant was then tested using a commercial pupillometer, the neurological pupil index (NPi)-200 (NeurOptics Inc., Irvine, CA, USA) [Figure 2a]. This device employs an infrared camera to record the pupil’s dynamic parameters over the course of 3.2 s, whereas integrating a calibrated light stimulus with a fixed intensity of 1000 lux. The device was held at a right angle to the patient’s axis of vision. To measure the pupil of the right eye, the pupillometer device’s smart guard was first placed on the right cheekbone, as shown in Figure 2b. The right button of the device was pressed, and the image of the eye displayed on the device screen was noted. The button would be released when the circle representing the pupillary margin on the screen turned green, indicating proper alignment. Subsequently, the device flashed a white light to induce a pupillary reaction, after which the different parameters measured would be presented on the screen [Figure 2c]. The entire process was then repeated on the left eye in a similar manner. According to the manufacturer’s statement, the device automatically calibrates, focuses, regulates the vertex distance, and omits outliers. In the event of any artifacts brought on by tracking issues due to blinking, the measurements were deleted, and the scan was redone, and only the high-quality measurements were included for further analysis.

Figure 2.

An image of neurological pupil index-200 (NeurOptics Inc., Irvine CA, USA) pupillometer device along with smart guard and charging station (a); an image of measurement being carried out (b); output of various pupillary parameters measured on the result screen by the device (c)

The pupil images on the Reflex application are captured at 12 frames per second for 3 s, whereas on the NPi-200 pupillometer, the images are captured at 30 frames per second for 3.2 s. A total of 100 paired scans (right and left eyes) were performed for each participant with the smartphone pupillometer and the commercial pupillometer. The measurements were carried out in routine clinical settings with an illumination of 64.9 lux.

Statistical analysis

Immediately after processing, the smartphone and commercial pupillometers displayed the results of various parameters on their screens. For statistical comparison in the current study, only those common pupillary parameters, including size (maximum diameter [MAX]), minimum diameter, constriction velocity (CV)(average constriction speed), maximum CV (maximum constriction speed), latency (LAT), and average dilatation velocity, which are recorded by both devices were included, whereas other parameters such as NPi and % change in pupil size (CH), which are exclusive to the NeurOptics, were excluded. The raw data were captured from each device and transferred to an Excel sheet for comparative analysis using the Statistical Package for Social Sciences (SPSS) software, (IBM manufacturer, Chicago, USA, ver 20.0). Descriptive statistics were presented as mean ± standard deviation. The statistical significance was set at P < 0.05 at the 95% confidence level. The Kolmogorov–Smirnov test was used to check for normality. For normally distributed parameters, paired t-test was used, whereas for other parameters that were not normally distributed, the Wilcoxon signed-rank test was used for the analysis. The intraclass correlation coefficient (ICC) was used to evaluate the agreement between the two devices. ICC values <0.50 were interpreted as poor, those between 0.50 and 0.75 as moderate, those between 0.75 and 0.90 as good, and those >0.90 as having excellent agreement. As positive correlation between amplitude with constriction and dilation velocity in healthy subjects has been previously reported to be an indicator of the reliability of pupillometer devices,[22,26,27] scatter plots of these variables were constructed to compare the values of positive correlation between the two pupillometer devices [Figures 3 and 4].

Figure 3.

Scatter plot between amplitude and constriction velocity using the reflex application (a); using neurological pupil index-200 pupillometer (b)

Figure 4.

Scatter plot between dilation velocity and amplitude using the reflex application (a); using neurological pupil index-200 pupillometer (b)

Results

Of the total 100 participants (41 males and 59 females) in the age range of 18–60 years, 10 were in the age group of 18–20 years, 37 in the age group of 21–30 years, 32 in the age group of 31–40 years, 15 in the age group of 41–50 years, and 6 in the age group of 51–60 years. A comparison of descriptive statistics revealed a statistically significant difference between the two devices for all measured pupillary parameters except LAT [Table 1]. The ICC was found to be <0.50 for all dynamic pupillometry parameters except for MAX (ICC = 0.58), indicating poor agreement between the smartphone and commercial pupillometer [Table 2]. In the current study, scatter plot graphs of amplitude with constriction and dilation velocities using the Reflex application showed wider scatter and lower positive correlation values (r = 0.33 and 0.14, respectively) as compared to those obtained through the NPi-200 pupillometer (r = 0.69 and 0.48, respectively) [Figures 3 and 4].

Table 1.

Dynamic pupillometry parameters (200 eyes) and their descriptive statistics

Parameter	Mean±SD (range)		P
	By neurological pupil index-200	By Reflex
Maximum diameter (mm)	4.55±0.83 (2.76–6.99)	5.36±0.84 (3.78–8.81)	0.00ǂ*
Minimum diameter (mm)	3.01±0.55 (1.73–5.15)	3.68±0.45 (2.7–5.21)	0.00ǂ*
CV (mm/s)	2.91±0.73 (0.93–4.79)	1.89±1.31 (0.03–8.58)	0.00Ɨ*
MCV (mm/s)	4.39±1.05 (1.45–7.47)	13.76±11.6 (0.38–74.4)	0.00Ɨ*
LAT (s)	0.22±0.02 (0.13–0.33)	0.22±0.06 (0.1–0.43)	0.36Ɨ
DV (mm/s)	1.2±0.28 (0.3–2.01)	0.54±0.59 (0.01–6.35)	0.00Ɨ*

*Statistically significant at P<0.05; ^ǂPaired t-test; ^ƗWilcoxon signed-rank test. CV: Constriction velocity; MCV: Maximum CV; LAT: Latency; DV: Dilatation velocity; SD: Standard deviation

Table 2.

Intraclass correlation coefficient for dynamic pupillometry parameters (n=200)

Parameter	ICC	95% CI	P
Maximum diameter	0.58	−0.08–0.8	0.00*
Minimum diameter	0.39	−0.18–0.68	0.00*
CV	0.09	−0.11–0.26	0.16
MCV	0.02	−0.14–0.18	0.38
LAT	−0.09	−0.44–0.17	0.72
DV	0.14	−0.09–0.34	0.01*

*Statistically significant at P<0.05. CI: Confidence interval; CV: Constriction velocity; MCV: Maximum CV; LAT: Latency; DV: Dilatation velocity; ICC: Intraclass correlation coefficient

Discussion

The current study found no agreement between the smartphone and commercial pupillometer, as indicated by the low ICC values. To the best of our knowledge, this study is the first to compare the two devices in the Indian population in routine clinical settings. Our results are in concordance with McKay et al.,[22] who in their study of dynamic pupillary measurements observed significant disagreement between the Brightlamp iPhone application and a portable infrared pupillometer (NeurOptics PLR-3000). They found iPhone pupillometry measurements to be unreliable for clinical decision-making. They included subjects with varying iris colors (blue, green, and brown) in their study, and quite significantly, all the failed measurements were observed in participants with brown-colored irises. This was attributed to the use of visible light stimuli in smartphones, which makes it difficult to measure pupillary parameters under various illumination levels, especially in individuals with darker iris colors, as compared to the infrared light used by most of the commercial pupillometers.[19,23,24,28] As the current study included participants from the Indian population who usually have brown or dark brown iris, it is likely that smartphone pupillometer measurements are not reliable in this population group.

Our results were not in concordance with studies done by Neice et al.,[23] Piaggio et al.,[24] and McAnany et al.[25] who obtained their results after pre- and postprocessing of the images and analyzing them using software. A smartphone pupillometer application (iPhone X) was created by Neice et al.,[23] who then compared it to a commercial pupillometer (NeurOptics) by scanning both eyes in 13 healthy volunteers (26 paired scans). NeurOptics pupillometer measured pupil diameter in millimeters versus time, whereas the smartphone pupillometer measured pupil diameter in pixels versus time. Before analysis, smartphone data were smoothed by averaging adjacent points, and to compare the data, two scans had to be aligned in time. They reported a substantial agreement between the two devices. However, in 23% of the captures (6/26), the smartphone pupillometer completely failed to identify the pupil and/or iris. The use of visible light imaging and related variations in iris color, which complicate the image processing methods used for pupil recognition, were probably responsible for the 23% failure rate. For all pupillometry-related measurements from 11 healthy subjects with light brown irides, Piaggio et al.[24] found that the suggested smartphone application (Samsung Galaxy A7 [2016]) outperformed the commercial infrared pupillometer (DP-2000-NeurOptics) with lower errors, higher and more significant correlations, and significantly better Bland–Altman plots. However, to improve the contrast between the pupil and iris, they converted video frames into gray scale and binarized them in accordance with empirically established thresholds for various degrees of illumination. The interpolation algorithm and normalization of the pupil diameter with the iris diameter, which minimized artifacts from hand motions and the use of visible light for pupil stimulation through cell phone flash and video acquisition, were attributed to making these results achievable. McAnany et al.[25] simultaneously recorded 15 visually normal subjects (age: 19–65 years) using an iPhone application (Sensitometer™ test) and an infrared camera (ViewPoint EyeTrack System; Arrington Research, Scottsdale, AZ). The following measurements were then calculated: dark-adapted steady-state pupil size, minimum pupil size, and pupil size during the redilation phase after the flash. Their findings showed that these two approaches had a high degree of similarity. The software used by them defined pupil size in pixels, in contrast to our study. All measurements were taken in a well-controlled laboratory environment, whereas our study was done in routine clinical settings. The small sample size, light iris color, and different methodologies and devices utilized in the abovementioned studies make them different from the current study. The results were analyzed as obtained by the devices without any pre- or post-processing of the images to evaluate whether the smartphone pupillometer can be incorporated into routine clinical settings when a commercial pupillometer is not available.

In the current study, wider scatter and lower positive correlation values were observed on scatter plot graphs using the Reflex application compared to those obtained through the NPi-200 pupillometer [Figures 3 and 4]. Therefore, the smartphone pupillometer was found to be unreliable. Disagreement between the two devices can be attributed to lower sensitivity and resolution (phone camera frame rates are lower) and a relative inability to differentiate between iris and pupillary zone colors by a smartphone pupillometer in dark irides.

The limitation of the current study is that we included participants from the Indian population only; hence, the values reported may not be representative of other population groups. Furthermore, we used only a single setting of the Reflex application, as advised by the manufacturer. Hence, further studies with different settings in different population groups may help establish the accuracy of the device. A guideline from the manufacturer’s end regarding the use of appropriate settings for different iris colors may help improve the accuracy and reliability of the application. Furthermore, a possible bias might have been introduced because the opposite eye was not covered and could have been partially stimulated by changes in ambient light. However, it was kept as constant as possible.

Conclusion

The current study reveals a significant disagreement between the measurements by the two devices, indicating that smartphone pupillometer may not be an ideal substitute for commercial pupillometers in their present form in the Indian population. Further studies with larger sample size as well as improvements in the processing and interpretation of the measurements by the software are needed to determine its utility in routine clinical settings.

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Ethical clearance

The study was approved by the institutional Ethics Committee of Sri Guru Ram Das Institute of Medical Sciences and Research, Vallah, Amritsar (Approval no: 4392).

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.