Use of the Argus II Retinal Prosthesis to Improve Visual Guidance of Fine Hand Movements

Abstract

Purpose.

We studied the capabilities of the Argus II retinal prosthesis for guiding fine hand movement, and demonstrated and quantified guidance improvement when using the device over when not using the device for progressively less predictable trajectories.

Methods.

A total of 21 patients with retinitis pigmentosa (RP), remaining vision no more than bare light perception, and an implanted Argus II epiretinal prostheses used a touchscreen to trace white paths on black backgrounds. Sets of paths were divided into three categories: right-angle/single-turn, mixed-angle/single-turn, and mixed-angle/two-turn. Subjects trained on paths by using prosthetic vision and auditory feedback, and then were tested without auditory feedback, with and without prosthetic vision. Custom software recorded position and timing information for any contact that subjects made with the screen. The area between the correct path and the trace, and the elapsed time to trace a path were used to evaluate subject performance.

Results.

For right-angle/single-turn sets, average tracing error was reduced by 63% and tracing time increased by 156% when using the prosthesis, relative to residual vision. With mixed-angle/single-turn sets, error was reduced by 53% and time to complete tracing increased by 184%. Prosthesis use decreased error by 38% and increased tracing time by 252% for paths that incorporated two turns.

Conclusions.

Use of an epiretinal visual prosthesis can allow RP patients with no more than bare light perception to guide fine hand movement visually. Further, prosthetic input tends to make subjects slower when performing tracing tasks, presumably reflecting greater effort. (ClinicalTrials.gov number, NCT00407602.)

A total of 21 blind retinitis pigmentosa patients used retinal prostheses to visually guide their hands to trace high-contrast paths of varying complexities. Prosthesis use decreased performance error by an average of 60% and increased time to complete the task by an average of 211%. Use of an epiretinal visual prosthesis can allow RP patients with no more than bare light perception to visually guide fine hand movement. Further, prosthetic input tends to make subjects slower when performing tracing tasks, presumably reflecting greater effort.

Introduction

Visual feedback is necessary for control of arm movements, such as reaching.¹ Individuals suffering outer retinal blindness, such as late stage retinitis pigmentosa (RP) or dry age-related macular degeneration, may be able to recover some visual feedback with use of a retinal prosthesis.^2–4 These devices stimulate the remaining cells in the inner retina to convey signals to the visual cortex. In theory, stimulating remaining cells in a pattern corresponding to visual stimuli could create visual percepts resembling those stimuli. With sufficient control over such percepts, a retinal prosthesis could be used to restore useful vision. In 2002, the first wireless intraocular prosthesis was implanted in a subject blind from late-stage RP.⁵ Since then, multiple studies conducted during long-term clinical trials of retinal prostheses have demonstrated that these devices can, indeed, provide some benefit in the realms of visual acuity,⁶ object and letter recognition,^7,8 and orientation and mobility.³

Research into hand-eye (or hand-visual prosthesis) coordination for subjects with retinal prostheses, however, is confined mostly to simulations^9–13 and pointing tasks.³ As demonstrated by Humayun et al., prostheses with at most 60 functioning electrodes are sufficient to improve the pointing ability of blind patients.³ While pointing demonstrates some degree of hand-eye coordination, guiding continuous hand movement based on visual input may provide a better measure of hand-eye coordination ability. For pointing, one must be able to determine the position of an object and use that information to guide a single motor command. To guide continuous movement, one must be able to extrapolate regularly distance and trajectory information from visual input, and provide that to the motor system, in addition to information on the hand's position. This kind of hand-eye coordination would be important for everyday tasks, such as placing or removing an object on a set dinner table, drawing, or moving food or beverages from one container to another.

Our study examines whether hand-movement guidance is aided by using a 6 × 10-electrode epiretinal prosthesis. The tool used for this study is similar to one first introduced for the evaluation of low-vision patients,¹ and adapted for use in simulations of prosthetic vision¹³ (Mueller VJ, et al. IOVS 2007;48:ARVO E-Abstract 2548). The test was modified further so it could be used with real prosthesis subjects and measure limitations of current prosthetic technology.

Methods

Subjects

A total of 21 subjects who were enrolled in a phase I/II clinical trial of the Argus II Retinal Stimulation System Feasibility Protocol (available online at http://www.clinicaltrials.gov: NCT00407602) participated in these experiments. All subjects suffered from late-stage RP and had been implanted with a 6 × 10-electrode epiretinal prosthesis (Argus II; Second Sight Medical Products, Sylmar, CA), but did not use any other assistive devices for this study. No assistive image processing, such as binarization or edge detection, was applied to camera input for this study. These subjects, when not activating the prosthesis, had no measurable remaining visual acuity at the 2.9 logMAR level, and no more than bare-light perception. All subjects had functioning retinal ganglion cell connections through the optic nerve, as confirmed by electrically evoked responses or documented light perception. All subjects also had some history of functional form vision in both eyes. In the United States and Switzerland, subjects were all 25 years of age or older. In France and the United Kingdom, the minimum age for inclusion was 18 years old.

All experiments were performed in accordance with the Declaration of Helsinki. Informed consent for our study was obtained from each subject after explanation of its nature and possible consequences. This research was approved by local institutional review boards at all test sites.

Experimental Setup

The concept of the tests used in these experiments is similar to that used by Carlson et al. to test the function of subjects who had regained some vision after a prolonged period of visual deprivation.^14,15 The task, originally referred to as the labyrinth, asked subjects to trace inside a path without drawing outside its borders. The experimenters concluded that rehabilitation was possible, but depended on the period of visual deprivation, duration of rehabilitation training, and the subject's personal motivation.

Subjects sat in front of a 37.6 × 30.2 cm touchscreen (Elo Touch Solutions, Menlo Park, CA) with a resolution of 1024 × 768 pixels. The distance between the prosthesis camera, located above the bridge of glasses worn by the subject, and the monitor was initially set to 30 cm. Movement was not restricted. The prosthesis system's video processing unit (VPU) was connected to a Lenovo ThinkPad notebook computer (Lenovo, Morrisville, NC) running Windows XP (Microsoft, Redmond, WA). Custom software controlled retinal stimulation by the array via the VPU, and provided the test operator with real-time displays of camera input and associated electrode stimulation patterns. A second Lenovo ThinkPad computer, also running Windows XP, was connected to the touchscreen and ran the custom Meander Maze Tracing program to test hand-eye coordination.

Meander Maze Tracing was written originally using Java by personnel at the Lions Vision Research & Rehabilitation Center of the Wilmer Eye Institute at Johns Hopkins University. It had been adapted to be usable by wearers of visual prostheses. The adapted version of the program presented a predesigned white path on a black background, and subjects attempted to trace the path without drawing outside its borders. Sets of 6–8 paths were presented in a random order.

The sequence of an individual trial is depicted schematically in Figure 1. Before each path was presented, a green screen was presented until the subject touched the screen to trigger the start of the trial. Initially, only the starting point of the path (a circle 85 pixels in diameter) was presented against the background. This stimulus, at the initial viewing distance, encompassed 3–4 stimulating electrodes. Auditory feedback informed the subject whether he or she had touched the starting point, was close to it, or touched the screen well outside the starting point. Once the subject found and made contact with the starting point, the entire path was displayed (lines 85 pixels wide). To assure that a continuous trace was obtained, the program alerted the subject through a warning tone whenever the touchscreen had stopped registering user input, at which point the subject could tap the screen or press harder to re-establish contact. Auditory feedback also was provided when the subject reached the end point of the path, signaling either a successful or unsuccessful trace of the path. For reinforcement purposes, a trace was considered successful if the number of points touched outside the path was less than half of those touched on the path. This measurement was used primarily to provide subjects with feedback on their levels of proficiency.

Figure 1.  .

The above frames show screens that were presented to the subjects in chronologic order for each trial. The first green screen appeared before any trial began. Upon touching the screen, the start point of the path would appear (center panel). Upon touching the start point, the rest of the path appeared for the subject to trace.

Each trace was recorded by the program for later comparison with its matching presented path. By constructing polygons encompassing the regions between the trace and path, the program could calculate the total area of error bounded by the trace and the path; time required for each trace also was calculated. The total error area and time were divided by the path length, to normalize across all given paths, and used as the outcome measures for a given trial.

Before testing with any paths, subjects were required to begin sessions by training on at least one set of paths. Training was very similar to testing, but incorporated auditory feedback that warned the subject whenever a trace had left the boundaries of the presented path. A constant tone, distinguishable from that signaling loss of contact, sounded until the subject's trace returned to the path. Subjects could use this feedback to “feel” the location and size of each path, and more importantly, to understand what cues from the prosthesis provided them with information useful for this task. Except for the right-angle set (for which only 8 variants exist), all paths used for training were in a separate set from those used for testing.

For each set of trials, the room lights were turned off and the subject was permitted to proceed through the trials at his or her own pace. Training was conducted with the system on, and testing called for one run each with the system on and off. For system off testing, subjects removed their glasses and the VPU was turned off, thus allowing them to rely on any remnant native vision. This experimental setup allows for a direct comparison of the subjects' native vision with and without the additional information provided by the prosthesis.

Procedures

Subjects participated in up to three separate experiments. Each successive experiment presented paths of increasing complexity and less predictability. All experiments called for system-on training, followed by system-on testing, and finally system-off testing. The size and composition of a training set was different for each test and is detailed below. If possible, subjects only received one training set per test per session. If a subject failed to trace successfully at least 75% of the paths in the training set, training was repeated. If training could not be completed with at least 75% success within two training rounds in the same session, no testing was conducted during that session. Subjects could move on to the third experiment only if they showed significant improvement with the prosthesis, over natural vision, in the second experiment or if system-off performance was comparable to average system-on performance, both greater than chance performance.

Right-Angle Tests

Subjects were introduced to the task using a set of eight basic paths that each had one right angle; an example is shown in Figure 2. Each path would start in one corner of the screen, end in the corner diagonally opposite the starting corner, and turn on a right angle in one of the remaining two corners. Since subjects were new to the task, and only eight paths could conform to these parameters, subjects trained and tested on this set of eight right-angle paths, but the order was randomized.

Figure 2.  .

This image is an example of a right-angle path.

Mixed-Angle, Single-Turn Tests

After completing tests with right-angle paths, 16 of the original 21 subjects were introduced to paths that incorporated acute and obtuse angles. The remaining five subjects were unable to continue testing for reasons unrelated to these experiments. Examples of such paths are shown in Figure 3. Paths still started and ended in corners, and had exactly one turn, but the relative locations of the start and end points varied. The location of the turn also no longer was bound to a corner, but still remained along the side the screen. Three classes of acute-angle paths and one class of obtuse-angle paths, shown in Figure 3, were used along with right-angle paths. All paths in Figure 3 started in the upper left corner, but paths in actual trials could start in any corner. Paths could be flipped horizontally, vertically, and about the diagonal to create a pool of 40 different paths, including the eight with right angles.

Figure 3.  .

The above images are examples of path classes introduced for the mixed, single-angle tests. Each path here starts in the upper left corner. Each class is defined by the position of the turn relative to the start point and the general angle of the turn. Different members of a class are generated by horizontally, vertically, and diagonally flipping configurations.

Subjects trained on only one example from each of the four new classes of paths. The subjects then tested on a set of eight paths with a mixture of angles. Paths given in the training set were not used for the testing set.

Two-Turn Tests

Based on results from the mixed, single-angle tests, a third category of paths was developed to reduce the predictability of paths further. The two-turn paths start and end at randomly chosen points on the screen, and incorporate exactly two turns. The two turns can have any combination of angles. Examples of such paths are shown in Figure 4. Only nine subjects who showed significant improvement with prosthesis use on the simpler paths, or who performed well with native vision, attempted these tests. For time considerations, the number of paths per set was reduced from eight to six. Subjects trained on one set of six paths, and then tested on a set of six different paths.

Figure 4.  .

The above images are examples of two-turn paths.

Statistical Analysis

Normalized error measurements and normalized trace times were pooled for within- and across-subject analyses. In all cases, data were evaluated using heterogeneous, unpaired Student's t-tests. Initial thresholds for significance were set at 0.05, and subsequently were adjusted using Bonferroni corrections.

Binomial tests also were used to measure performance on sets of right-angle paths, for the following reason: Subjects were given the corner containing the start location of each path through auditory feedback, and knew that the path must travel to one of two adjacent corners, and end at the corner farthest from the start point. The trajectory, therefore, only has two possibilities, and the accuracy of a subject's trace is dependent largely on correctly detecting in which direction the path travels away from the start point. Tracings for these sets were grouped into “correct” and “incorrect” categories, removing a small number of tracings that did not correspond to assumptions of right-angle paths. Data then were analyzed as though generated by a two-alternative forced-choice paradigm.

Timing of mixed-angle tests was analyzed for dependencies on angle-type. Timing data were square-root transformed and analyzed using a two-way ANOVA.

Results

On average, collapsing across all subjects and trials, prosthesis use significantly reduced the error in tracing by 60% (P < 0.001) and increased trace time by 211% (P < 0.001). Details for each individual experiment are provided below.

Right-Angle Tests

Normalized error scores for the right-angle tests are shown in Figure 5. Analyzing the data as though generated by a two-alternative forced-choice paradigm, and assuming that a subject with no intrinsic light perception was using the prosthesis, we found that the subject would choose the correct orientation of the path in 95% of trials (P < 0.001), as opposed to 60% without prosthesis use. Normalized error scores lower than 65 pixels were considered correct, while scores greater than 215 were interpreted to indicate that the opposite orientation was chosen.

Figure 5.  .

The above graph displays error scores for right-angle paths. Points represent error scores for individual trials, chronologically ordered across all non–light-perceiving subjects. The predictable nature of the paths created an approximate two-alternative forced-choice test. Subjects typically traced conforming to a right angle path, and most of the variability arose from whether they chose the correct orientation of the path. Closed symbols: system-on trials. Open symbols: system-off trials. Percent correct (<65 pixels error) given system-on: 95% (P < 0.001, binomial test).

Comparing system-on versus system-off performance directly, this experiment showed an average 63% reduction in tracing error (P < 0.001) and 156% increase in trace time (P < 0.001) when using the prosthesis. Of the subjects 43% (9 of 21) showed significant within-subject reduction of error with prosthesis use.

Mixed, Single-Angle Tests

The less predictable mixed-angles sets showed more varied results than the bimodal results of the right-angle sets. Figure 6 plots system-on performance against system-off performance, averaged within subjects. On average, prosthesis use reduced trace error by 53% (P < 0.001) and increased trace time by 184% (P < 0.001), as seen by the clustering of points below and above the identity line, respectively, in the left and right panels of Figure 6. Of the 16 subjects nine (56%) were able to demonstrate significant within-subject reduction of error relative to system-off performance.

Figure 6.  .

Left graph: normalized error scores from the mixed, single-angle tests. Right graph: normalized time to trace paths in the mixed, single-angle tests. Error areas and times were divided by path lengths (in pixels) to obtain the normalized scores shown here. Points indicate average performance for each subject. Error bars denote SEM. Diagonal lines represent the identity line. In general, system-on status reduced error and increased trace time.

Considering only system-on trials, timing data showed no correlation with subject accuracy. Figure 7 demonstrates the high degree of variation in subject performances as given by the average amount of error and time recorded for each trial; note that timing data have been square root-transformed to obtain a more Gaussian distribution.

Figure 7.  .

Average performance for each subject, system-on, for the mixed-angle paths. Error areas and times were divided by path lengths (in pixels) to normalize scores, and transformed using the square root operation. Error bars denote SEM. Subject performance with the prosthesis, in both speed and accuracy, was highly varied.

Two-Turn Tests

Averaging over the nine subjects in this experiment, prosthesis use reduced tracing error by 38% (P < 0.001) and increased tracing time by 252% (P < 0.001). Five of the nine subjects showed significant within-subject reduction of error with system on versus off, as shown in Figure 8.

Figure 8.  .

Left graph: error scores from the two-turn tests. Right graph: the time to trace paths in two-turn tests. Error areas and times were divided by path lengths (in pixels) to normalize scores. Points indicate average performance for each subject. Error bars denote SEM. Diagonal lines represent the identity line. Prosthesis use reduced error and increased time to trace paths.

Discussion

Collectively, these experiments demonstrated that patients with severe outer retinal degeneration can improve their abilities to guide fine hand movements by using a 6 × 10-electrode epiretinal prosthesis. Variation across subjects, however, did indicate that some patients may not gain this benefit, while others may see robust improvement. Factors affecting the efficacy of prosthesis use may include the condition of the prosthesis, degree and source of retinal deterioration, motivation, and effectiveness of prior training.

Right-Angle Tests

The first experiment offered an easy introduction to the tracing task. Its manifestation as a two-alternative forced-choice test, resulting from its simplicity, also provided another basic measure of the perceptual benefits of the prosthesis. Chance performance with this experiment, indicating that the subject is not perceiving the path, corresponds to scoring 50% correct. In the system-on condition, non–light-perceiving subjects selected the correct direction of the path in 95% of the trials. This indicates that they were capable of crudely locating the paths with the percepts generated by the prosthesis. The difference from 100% correct would suggest, however, that improvement in the device design and/or training regimen is required for reliably perceiving such stimuli. No conclusions can be drawn regarding hand-eye coordination, as the ease of this task limits the necessity for such processing.

Mixed, Single-Angle Tests

The mixed, single-angle experiment provided a cleaner look at the motor guidance skills of these patients. As the paths could turn in a variety of manners, the likelihood of completing a trial by chance successfully was reduced. This could be seen by the more broadly distributed error scores for this experiment. With minimal information, however, such as crudely localizing light from each line segment, a subject could make an educated guess and complete a trial with very little error. Some subjects also made larger errors using this technique, though, such as by drawing a right-angle path when an obtuse-angle path was presented.

Because types of paths and the locations of turns still were quite predictable, these tests only provided a limited demonstration of hand-eye coordination. Still, not all subjects were able to use the prosthesis effectively to improve performance above that of the system-off condition. Only subjects gaining enough benefit from the prosthesis could move on to tests more directly evaluating hand-eye coordination.

Two-Turn Tests

The two-turn paths were designed to limit the element of prediction. Because subjects were less able to extrapolate path designs, these paths required much more real-time visual guidance of fine hand movement. Subjects generally were quite successful performing these tests, although only five showed significant improvement with the prosthesis. These results demonstrated that, at least for some subjects, true continuous visual guidance of fine hand movement is possible with an epiretinal prosthesis.

Timing

In all tests, use of the prosthesis increased time to complete the tracing task by, on average, 150% or more. Among system-on trials in the second experiment, there was a significant effect of angle-type (acute, right, and obtuse) on square-root transformed tracing time (two-way ANOVA, P < 0.001). Although insufficient data were collected to analyze the effect of angle-type within subjects, tracing times for right versus acute angles were very similar, and those for obtuse angles versus the other two were 86% higher, averaging across subjects.

We interpreted these data to reflect the scarcity of information available to the subjects without the prosthesis. In the system-off condition, many of the subjects could only guess at the design of the path, while in the system-on condition, subjects were making a concerted effort to determine the trajectory of the path. Although encouraged to try to complete the task in the same manner without the prosthesis as with it, most subjects quickly lost motivation and resorted to near-random guessing. Information provided by the prosthesis is consistent enough that the status of the device (on or off) cannot be hidden from the subject. This random guessing allowed subjects to find the end of path much more quickly than deliberate use of the prosthesis, at the expense of increasing tracing error. Obtuse angles seemed to cause more difficulty than paths with right and acute angles. Presumably, the subject notices and follows a sharp break in the trajectory more easily than a more subtle break.

Future Directions

The methods described in our study can be used to study further hand-eye coordination in visual prosthesis patients. Limitations of hand-eye coordination can be evaluated by increasing the complexity of the paths or by decreasing the path widths. More complex paths will put more demand on form vision and finger tracking. Narrower paths will reduce the deviation permitted for any given trace, requiring more exact position estimation, and will test prosthesis sensitivity. As prosthesis development continues, such increases in difficulty can provide comparative measures of functional benefit.

Given the wide range of abilities among subjects performing these tests, from those who had difficulty with the basic right-angle paths to those who could trace the two-turn paths successfully, investigation into what factors may influence performance will be beneficial. Further studies will attempt to find commonalties among subjects who did not get much benefit from the prosthesis and among those who performed very well.

Appendix

Members of the Argus II Study Group

Aries Arditi, Lighthouse International, New York, NY; Rajat Agrawal, University of Southern California, Los Angeles, CA; Pierre-Olivier Barale, Centre Hospitalier National d'Ophtalmologie des Quinze-Vingts, Paris, France; David Birch, Retina Foundation of the Southwest, Dallas, TX; Susmito Biswas, Manchester Royal Eye Hospital, Manchester, UK; Gary Brown, Wills Eye Hospital, Philadelphia, PA; Artur V. Cideciyan, Scheie Eye Institute, University of Pennsylvania, Philadelphia, PA; Lyndon da Cruz, Moorfields Eye Hospital, NIHR Biomedical Research Centre for Ophthalmology, London, UK; Gislin Dagnelie, Lions Vision Research and Rehab Center, Johns Hopkins University, Baltimore, MD; Eugene de Juan, University of California-San Francisco, San Francisco, CA; Lucian Del Priore, Columbia University, New York, NY; Jacque L. Duncan, University of California-San Francisco, San Francisco, CA; Dean Eliott, Doheny Eye Institute, University of Southern California, Los Angeles, CA; Amani Fawzi, Doheny Eye Institute, University of Southern California, Los Angeles, CA; Eugene Filley, Retina Foundation of the Southwest, Dallas, TX; Farhad Hafezi, Hôpitaux Universitaires de Genève, Geneva, Switzerland; Julia Haller, Wills Eye Hospital, Philadelphia, PA; James Handa, Wilmer Ophthalmological Institute, Johns Hopkins University, Baltimore, MD; Allen Ho, Wills Eye Hospital, Philadelphia, PA; Mark Humayun, Doheny Eye Institute, University of Southern California, Los Angeles, CA; Samuel G. Jacobson, Scheie Eye Institute, University of Pennsylvania, Philadelphia, PA; Saddek Mohand-Said, Centre Hospitalier National d'Ophtalmologie des Quinze-Vingts, Paris, France; Lisa Olmos, Doheny Eye Institute, University of Southern California, Los Angeles, CA; Marco Pelizzone, Hôpitaux Universitaires de Genève, Geneva, Switzerland; Angelica Perez-Fornos, Hôpitaux Universitaires de Genève, Geneva, Switzerland; Carl Regillo, Wills Eye Hospital, Philadelphia, PA; Enrique Roig, Puerta de Hierro Centro Medico, Guadalajara, Mexico; Avinoam B. Safran, Hôpitaux Universitaires de Genève, Geneva, Switzerland; José-Alain Sahel, Centre Hospitalier National d'Ophtalmologie des Quinze-Vingts, Paris, France; Joel Salzmann, Hôpitaux Universitaires de Genève, Geneva, Switzerland; Arturo Santos, Centro de Retina Medica y Quirúrgica, SC, Escuela de Medicina, Tec de Monterrey Campus Guadalajara, Guadalajara, Mexico; Sarah Sheer, Centre Hospitalier National d'Ophtalmologie des Quinze-Vingts, Paris, France; Jörg Sommerhalder, Hôpitaux Universitaires de Genève, Geneva, Switzerland; Rand Spencer, Retina Foundation of the Southwest, Dallas, TX; Paulo E. Stanga, Manchester Royal Eye Hospital, Manchester Biomedical Research Centre, University of Manchester, Manchester, UK; George Turner, Manchester Royal Eye Hospital, Manchester, UK; Andrew Webster, Moorfields Eye Hospital, London, UK.