Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 May 14.
Published in final edited form as: Cell. 2020 May 14;181(4):774–783.e5. doi: 10.1016/j.cell.2020.04.033

Dynamic Stimulation of Visual Cortex Produces Form Vision in Sighted and Blind Humans

Michael S Beauchamp 1, Denise Oswalt 1, Ping Sun 1, Brett L Foster 1, John F Magnotti 1, Soroush Niketeghad 2, Nader Pouratian 3, William H Bosking 1,*, Daniel Yoshor 1,*,†
PMCID: PMC7331799  NIHMSID: NIHMS1586199  PMID: 32413298

Summary

A visual cortical prosthesis (VCP) has long been proposed as a strategy for restoring useful vision to the blind, under the assumption that visual percepts of small spots of light produced with electrical stimulation of visual cortex (phosphenes) will combine into coherent percepts of visual forms, like pixels on a video screen. We tested an alternative strategy in which shapes were traced on the surface of visual cortex by stimulating electrodes in dynamic sequence. In both sighted and blind participants, dynamic stimulation enabled accurate recognition of letter shapes predicted by the brain’s spatial map of the visual world. Forms could be presented and recognized rapidly by blind participants, up to 86 forms per minute. These findings demonstrate that a brain prosthetic can produce coherent percepts of visual forms.

Graphical Abstract

graphic file with name nihms-1586199-f0001.jpg

ETOC:

Dynamic stimulation of visual cortex allows both sighted and blind subjects to recognize a variety of letter shapes without training and with accuracy.

Introduction

In most participants with acquired blindness, only the eyes or optic nerves are damaged. This has inspired hope for the development of a visual cortical prosthetic (VCP), a device that would bypass the eyes and optic nerve, transmitting visual information from a camera directly into the visual cortex (Bosking et al., 2017a; Brindley and Lewin, 1968; Christie et al., 2016; Foroushani et al., 2018; Lewis et al., 2015; Lewis et al., 2016; Normann et al., 2009). VCPs rely on the fact that stimulating visual cortex with electrical current can produce a percept of a small flash of light, known as a phosphene (Penfield and Rasmussen, 1950). Because of the retinotopic organization of visual cortex, implanting an array of multiple electrodes at different locations within this cortical map allows for the creation of multiple phosphenes, with each stimulated electrode contributing a phosphene at one predictable visual field location (Bosking et al., 2017a; Tehovnik and Slocum, 2013). Recent advances in biomedical engineering have resulted in wirelessly powered and controlled devices containing dozens of electrodes that can be implanted in visual cortex, leading to a worldwide resurgence in efforts to develop a clinically useable VCP (Lowery, 2013; Mirochnik and Pezaris, 2019; Roelfsema et al., 2018; Troyk, 2017).

Despite these impressive technical advances, there have not been corresponding advances in the stimulation paradigms used to generate visual percepts. The fundamental assumption has remained that individual phosphenes are analogous to pixels in a computer display, and that they can be easily combined to generate a coherent image. Despite its intuitive appeal, there is little evidence that this assumption is correct. In early testing with VCPs, stimulation of multiple electrodes produced only percepts of multiple isolated phosphenes which did not combine into coherent forms (Dobelle et al., 1976; Schmidt et al., 1996), a finding we replicated in sighted participants implanted with intracranial electrodes for the pre-surgical evaluation of epilepsy (Bosking et al., 2018a). To overcome this obstacle, we developed an alternative approach using dynamic activation of a sequence of electrodes.

Our new stimulation paradigm can be explained by analogy to tracing letters on the palm (Figure 1). To convey the letter “Z”, one could press multiple probes arranged in a “Z” pattern into the palm (Figure 1A). However, this technique produces a percept of a touch without coherent form. Alternatively, one could dynamically move a single probe in a sequence that matches the “Z” shape (Figure 1B), which immediately produces a coherent letter percept.

Figure 1. Stimulation paradigms for visual cortical prosthetics.

Figure 1.

Open in a new tab

(A) To convey a letter through touch, one could use static tactile stimulation to press multiple probes (blue dots) into the palm of the hand, forming the shape of a static letter. However, this results in an amorphous percept (blob in thought bubble). (B) Alternately, one could use dynamic tactile stimulation to trace the shape of the letter dynamically using a single probe (single blue dot) traced across the palm in a sequence (black line with arrow) that matches the desired shape, producing a coherent letter percept (“Z” in thought bubble). (C) In a cortical visual prosthetic, an electrode grid is implanted over visual cortex. For static electrical stimulation, current is delivered concurrently to some electrodes (blue circles) but not others (gray circles), resulting in an amorphous percept. (D) For dynamic current steering, current is delivered to the electrode grid in a temporal sequence that matches the desired shape (white arrows), resulting in a coherent visual percept. Current is delivered in sequence to physical electrodes (dark blue circles) and virtual electrodes (light blue ovals) created by current steering (delivering current simultaneously to adjacent physical electrodes).

The electrical stimulation paradigm used in existing VCPs is similar to the multiple probe approach: multiple electrodes are stimulated, either at once or slightly offset in time, resulting in an incoherent percept (Figure 1C). Our new paradigm is analogous to tracing letters on the palm, except that the desired letter is formed on the surface of the cortex by stimulating electrodes in a dynamic sequence (Figure 1D). Electrodes are located at discrete locations, requiring additional machinations to stimulate cortex in a continuous trajectory. This is accomplished through a technique known as current steering: if current is passed through two adjacent electrodes, a virtual electrode is created midway between them (Firszt et al., 2007). Varying the amount of current delivered to adjacent electrodes allows the virtual electrode to be positioned at different locations along the line segment between the two physical electrodes.

Results

Applying current steering to human visual cortex

Dynamic current steering combines current steering with dynamic stimulation. In order to examine the efficacy of current steering in human visual cortex, we tested a blind participant (participant 03–281) with electrodes implanted on the medial wall of the occipital lobe near the calcarine sulcus, the location of primary visual cortex (Figure 2A). We delivered 100 ms trains containing 60 Hz electrical pulses to two nearby electrodes, varying the amount of current delivered to each electrode across five different levels (Figure 2B and 2C). The participant reported their percept by drawing the perceived location of the phosphene on a computer touchscreen.

Figure 2. Effectiveness of current steering in a blind participant.

Figure 2.

Open in a new tab

(A) A medial view of a cortical surface model of the participant’s occipital lobe (dashed line indicates calcarine sulcus). Two subdural electrodes are shown as a red and a blue disc (labeled F01 and F03). (B) Current steering was implemented by delivering varying amount of current to the two electrodes. In the first condition, maximum current was delivered to electrode F01 and no current was delivered to electrode F03; in successive conditions, the amount of current to F01 was decreased and the amount of current delivered to F03 was increased, until in the fifth condition, no current was delivered to F01 and maximal current was delivered to F03. Colored circles underneath each condition correspond to phosphene locations in (D). (C) In each condition, current was delivered in a single 100 ms pulse train with 6 pulses per train (pulse frequency of 60 Hz). The pulse train for the third stimulation condition in (B) is shown, with equivalent current amplitudes for F01 and F03. For other conditions, the current amplitude delivered to the electrodes differed, but the timing of the pulse trains was identical. (D) The participant used a touch screen to report the location of the visual percept resulting from each stimulation condition. Ten trials of each condition were performed. The colored circle shows the average location of all reports for that condition, the dashed line shows the 95% spatial confidence interval fit with an ellipse. Colors of each circle and dashed line correspond to condition colors shown in (B).

First, a brief calibration procedure (see Methods) was used to find the current required to evoke phosphenes from the two electrodes, resulting in baseline current values of 4 mA for electrode F03 and 3.6 mA for electrode F01. In the first stimulation condition, a 100 ms pulse train with a base current of 3.6 mA was delivered to the more anterior electrode (F01) with no current delivered to electrode F03 (F01: 100%; F03:0%). During delivery of the pulse train, the participant reported the percept of a single phosphene located in the upper right visual field and recorded its location on the touchscreen. Across ten trials, the mean location of the report was at co-ordinates of (azimuth 13.3°, elevation 2.1°). To characterize the variability of the location across trials, an ellipse was generated that contained all reported phosphene locations (Figure 2D). The length of the major and minor axes of the ellipse were (1.0°,0.5°).

In the fifth stimulation condition, a base current of 4 mA was delivered only to the more posterior electrode (F03) with no current delivered to F01 (F01: 0%; F03:100%). Consistent with the known retinotopic organization of visual cortex, the F03 phosphenes were located more centrally than the F01 phosphenes, with center (5.4°, 2.2°) and variability (0.9°,0.6°).

In the intervening conditions, current was delivered in varying proportions to the two physical electrodes, with the goal of using current steering to create virtual electrodes located between the two physical electrodes. In the second condition, 80% of the base current was delivered to electrode F01 and 50% of the base current was delivered to F03. As hypothesized, this generated phosphenes located intermediate to those generated by stimulating the two physical electrodes but closer to the F01 phosphenes, with center (11.4°, 2.1°) and scatter (0.5°,0.3°). In the third condition, equivalent currents were delivered to the two electrodes (F01: 70%; F03:70%) and the resulting phosphenes were located almost exactly half-way between the F01 and F03 phosphenes, with center (9.8°, 2.1°) and scatter (0.4°,0.3°). In the fourth condition (F01: 50%; F03:80%), phosphenes were intermediate between the two physical electrodes but closer to F03, with center (8.0°, 2.2°) and scatter (0.5°,0.3°).

Combining current steering and dynamic stimulation

When plotted (Figure 2D), the location of the phosphenes created by stimulating the physical and virtual electrodes lay on a horizontal line at 2° elevation with no overlap between the phosphenes created by stimulating the two physical and three virtual electrodes. This suggests phosphenes could be created at additional intermediate locations by manipulating the applied current. Specifically, if the current varied dynamically on a rapid time scale, it might result in the percept of a phosphene moving continuously along a line.

To test the efficacy of dynamic stimulation in combination with current steering, we selected 5 electrodes for stimulation located anteriorly-to-posteriorly along the lingual gyrus (Figure 3A). Over a 400 ms window, a 120 Hz pulse train was applied with gradually increasing and then decreasing currents to each electrode in sequence (Figure 3B) in order to create a virtual electrode moving continuously from anterior to posterior along the gyrus. The participant reported that this stimulation pattern produced the percept of a “like a line being drawn” with a continuity rating of 8 out of 10, where 1 would be discrete phosphenes, and 10 would be a perfectly continuous line. The participant reported that the line began at (12.9°, 2.7°) and ended at (3.2°, 1.8°) as shown in Figure 3C.

Figure 3. Effectiveness of dynamic current steering in a blind participant.

Figure 3.

Open in a new tab

(A) A medial view of a cortical surface model of the participant’s occipital lobe (dashed line indicates calcarine sulcus; same participant as Figure 2). Five subdural electrodes are shown as blue discs (letters show electrode labels). (B) Dynamic current steering was implemented by delivering time-varying amounts of current to the electrodes. (C) The participant reported the percept of a point traversing the visual field in a horizontal direction.

Using dynamic stimulation to create letter percepts in sighted participants

In sighted participants, we examined whether dynamic stimulation of visual cortex could be used to evoke form percepts. In the example illustrated in Figure 4, a high-density array of 24 electrodes was implanted on the medial face of the occipital lobe (participant YBN; Figure 4A). The RFs obtained from this array were organized in an orderly manner in the upper right visual field (Figure 4B). Using the mapped RFs, we designed dynamic stimulation sequences corresponding to four different letter-like forms (Figure 4C). Following stimulation, the participant was able to easily reproduce each of the four letters on a touchscreen, and there was a striking correspondence between the predicted and actual shape of the perceived letters (Figure 4D). The participant performed well above chance in a four alternative forced choice-task discriminating between stimulation patterns (15/23, accuracy of 66% vs. chance rate of 25%, p = 10−4 from binomial distribution). In sighted participant YAY, dynamic current steering was used to convey the letter “Z” (Figure 4H; Supplementary Video 1).

Figure 4. Dynamic stimulation produces perception of letter forms in sighted participants.

Figure 4.

Open in a new tab

(A) Medial view of the left occipital lobe of sighted participant YBN. Blue circles show the 24 electrodes contained in a grid implanted inferior to the calcarine sulcus (dashed black line). Black numbers label electrodes and correspond to phosphene numbers in (B). (B) To generate receptive field maps, the participant fixated while mapping stimuli were presented (see Supplementary Figure 1). The blue circles show the location of the receptive field centers for each electrode relative to the fixation point (+). (C) Pulse diagram for dynamic stimulation (without current steering). The timing of the pulses to the first three electrodes in a sequence are shown; pulses to successive electrodes occurred with the same timing. For each electrode, a stimulation current was used that produced a reliable phosphene when that electrode was stimulated in isolation, ranging from 1.2 to 1.5 mA for different electrodes. (D) Dynamic stimulation of selected electrodes was used to generate visual percepts of four different letter forms. For each letter form, the left panel shows the stimulated electrodes (bold circles) and the direction of the temporal sequence of stimulation (arrow). The right panel shows the participant’s actual drawing of the visual percept and the verbal label used to identify it. (E) Medial view of the left occipital lobe of sighted participant YAY. Blue circles show the location of stimulated electrodes relative to calcarine sulcus (dashed black line). (F) Location of individual phosphenes. The participant fixated while electrical stimulation was delivered to one electrode at a time. The participant drew each phosphene on a touchscreen (bold ellipses, numbered by the corresponding electrode). (G) Pulse diagram for dynamic stimulation with current steering. The timing of the pulses to the first three electrodes in a sequence are shown; pulses to successive electrodes occurred with the same timing. Baseline stimulation currents for each electrode ranged from 0.7 mA to 1.5 mA. (H) The phosphene map was used to design a stimulation sequence to produce the visual percept of the letter “Z”. The black arrows show the temporal sequence of stimulated electrodes (black numbers) and virtual electrodes located between the physical electrodes (red dots). The participant drew the pattern they perceived on the touchscreen (blue line in middle panel). Right panel shows still frame from a video of the participant drawing, see Supplementary Video 1 for full video.

Using dynamic stimulation to create letter percepts in blind participants

In blind participant BAA, five electrodes implanted on the medial wall of visual cortex were stimulated (Niketeghad et al., 2019). When stimulated individually, each electrode produced a discrete phosphene (Figure 5C). Using the phosphene map from this participant, seven different dynamic stimulation sequences were designed. Without instruction, the participant was able to reproduce letter-like shapes that corresponded to each sequence (Figure 5D; Supplementary Video 2). To assess reliability, the participant received randomly interleaved presentations of different dynamic sequences, drawing the perceived pattern following each trial. The shapes were reliable across repeated trials of the same stimulation sequence but differed for different sequences. To quantify this effect, the drawings were quantized, correlated, and subjected to multidimensional scaling followed by k-means clustering (Figure 4E). 23 out of 28 (82%) letters were placed in the correct cluster (p < 10−5 from a boot-strap analysis in which the sequences were randomly shuffled).

Figure 5. Dynamic stimulation tested in blind participant BAA.

Figure 5.

Open in a new tab

(A) Still frame from a video of the participant drawing, see Supplementary Video 2 for full video. The participant placed the index finger of the left hand on a tactile fixation point in the middle of the touch screen and used the index finger of the right hand to trace the visual percept on the touch screen. (B) Medial view of a surface model of the participant’s right occipital lobe. Dashed line shows calcarine sulcus, circles show electrode locations and numbers. (C) Blue ellipses show participant drawing of phosphenes created by stimulation of individual electrodes, with numbers corresponding to electrodes in (B). Crosshairs show location of tactile fixation point. (D) Dynamic stimulation pulse diagram. The timing of the pulses to the first three electrodes in a sequence are shown; pulses to successive electrodes occurred with the same timing. Stimulation current for all electrodes was 2 mA. (E) Seven different letter-like shapes created by seven different dynamic stimulation patterns. The left panel for each shape shows the temporal sequence of stimulation (electrodes indicated by numbers connected by black arrows). The right panel for each shape shows the participant drawings with each stimulation pattern. Each line illustrates a separate trial (randomly interleaved), colored in different shades of blue for visibility. The letter in quotation marks shows the participant mnemonic for that pattern (backwards “G”; backwards “N”; backwards “R”; upside-down “U”; upside-down “V”; “W”; “Z”). (F) Quantification of the drawings produced by the participant for each trial of each stimulation pattern using multidimensional scaling analysis. Each letter corresponds to a single trial of the corresponding stimulation pattern. Boldface letters show the centroid of each cluster from a k-means analysis.

As a more direct test of the difference in the percepts created by the different stimulation sequences, an additional experiment was conducted in which the participant was asked to verbally identify stimulation sequences presented in random order, selected from a set of five patterns. The participant correctly identified 14/15 of the patterns (93% correct vs. 20% chance rate, p = 10−8).

In blind participant 03–281, six electrodes implanted on the medial wall of visual cortex were stimulated (Figure 6). When stimulated individually, each electrode produced a discrete phosphene. Using the phosphene map from this participant, four different dynamic stimulation sequences were designed. Without instruction, the participant was able to reproduce letter-like shapes that corresponded to each sequence (Supplementary Video 3). To assess reliability, the participant received repeated presentations of different dynamic sequences, drawing the perceived pattern following each trial. The shapes were reliable across repeated trials of the same stimulation sequence but differed across sequences. The drawings were quantized, correlated, and subjected to multidimensional scaling followed by k-means clustering. 36 out of 40 (90%) letters were placed in the correct cluster (p < 10−5 from a boot-strap analysis in which the sequences were randomly shuffled). The participant also verbally identified each pattern following the trial. The participant correctly identified 37 out of 40 patterns presented in random sequence (93% vs. chance rate of 25%, p = 10−15). The participant was unequivocal in equating the perceived patterns with letters, for instance replying “N as in Nancy” for the N stimulation pattern.

Figure 6. Dynamic stimulation tested in blind participant 03–281.

Figure 6.

Open in a new tab

(A) Still frame from a video of the participant drawing, see Supplementary Video 3 for full video. The participant placed the index finger of the left hand on a tactile fixation point and used the index finger of the right hand to trace the visual percept on the touch screen. (B) Medial view of a surface model of the participant’s right occipital lobe. Dashed line shows calcarine sulcus, circles show electrode locations and labels. (C) Blue ellipses show participant drawing of phosphenes created by stimulation of individual electrodes, with labels corresponding to electrodes in (B). (D) Dynamic stimulation pulse diagram. The timing of the pulses to the first three electrodes in a sequence are shown; pulses to successive electrodes occurred with the same timing. Currents ranged between 3.5 and 5.8 mA per electrode. (E) Four different letter-like shapes created by four different dynamic stimulation patterns. The left panel for each shape shows the temporal sequence of stimulation (electrodes indicated by numbers connected by black arrows). The right panel for each shape shows the participant drawing for each stimulation pattern along with verbal label. (F) Participant drawings for ten different trials of each stimulation pattern, each trial indicated with a different color. (G) Quantification of the drawings produced by the participant for each trial of each stimulation pattern using multidimensional scaling analysis. Each letter corresponds to a single trial of the corresponding stimulation pattern. Boldface letters show the centroid of each cluster from a k-means analysis. (H) Accuracy of form identification at two different presentation rates. Left bar: two forms (a downward line and an upward line) were presented in random order at 85 forms per minute and verbally identified by the participant. Right bar: three forms (“C”, “U”, backwards “C”) were presented at 30 forms per minute. Error bars show 95% confidence interval from binomial distribution. See Supplementary Video 4 for full video.

Stimulation Rate

After demonstrating that blind participant 03–281 could reliably recognize stimulation patterns, we set out to determine how rapidly different patterns could be delivered (Figure 6I and Supplementary Video 4). In the first stimulation rate experiment, electrodes were stimulated such that they produced the percept of a line drawn in either a downward or upward direction in the visual field. The stimulation time for each sequence was 200 ms followed by a 500 ms response window to allow the participant to verbally report the form as either “down” or “up”, for a delivery rate of 86 forms per minute. In a sequence of 30 trials, the participant accurately identified 26 forms (87% correct vs. chance rate of 50%, p = 10−4).

In the second stimulation rate experiment, electrodes were stimulated such that they produced one of three forms, either “C”, “U” or backwards “C”. The stimulation time was 700 ms followed by a 1300 ms response window, for a delivery rate of 30 forms per minute. In a sequence of 36 trials, 33 forms were correctly identified (92% correct vs. chance rate of 33%, p = 10−12).

Identity of Stimulated Visual Areas

A human cytoarchitectonic atlas (Rosenke et al., 2018) was morphed to each participant’s cortical surface model and used to evaluate the probable identity of the identified visual area underlying each electrode. 35 of 48 stimulated electrodes were assigned to visual area V1 and the remaining 13 electrodes were assigned to visual area V2. These findings are consistent with an earlier study that found no qualitative difference in the phosphenes generated by V1 and V2 identified within individual subjects using BOLD fMRI retinotopic mapping (Murphey et al., 2009).

Comparison of Dynamic Current Steering with other stimulation paradigms

Across sighted and blind participants, dynamic stimulation and dynamic current steering stimulation was able to produce percepts of letter forms. We attempted to compare the efficacy of dynamic stimulation, in which a set of electrodes are stimulated in temporal sequence, with static stimulation, in which the same set of electrodes are stimulated at the same time. We were stymied by the inability of static stimulation to produce perceptible forms, an inability due to the merging of the phosphenes created by simultaneous stimulation of multiple electrodes. For instance, in participant YBH, we expected that simultaneous stimulation of 5 different electrodes would produce 5 separate phosphenes, which could conceivably be interpreted as a form. Instead, the participant reported seeing only 2 large phosphenes, too few to be recognizable as a shape. Stimulating a different set of 5 electrodes also resulted in the percept of 2 large phosphenes, albeit in a different location than the first set. We observed similar results in other participants, making it impossible to quantify the performance advantage of dynamic vs. static stimulation.

Discussion

For fifty years, attempts have been made to restore vision to blind participants with electrical stimulation of the intact visual cerebral cortex (Bosking et al., 2017a; Foroushani et al., 2018; Lewis et al., 2015; Lewis et al., 2016; Normann et al., 2009). The pioneering early attempt of Brindley and Lewin was the first to accomplish wireless stimulation, but was impeded by technological limitations, especially the inability to fabricate miniature electrode arrays and control them with portable transmitters and receivers (Brindley and Lewin, 1968). Recent advances in wireless microelectronics have renewed interest in VCPs, with at least four neuroengineering groups currently developing devices that are either in, or rapidly approaching, the clinical trial stage (Lowery, 2013; Mirochnik and Pezaris, 2019; Roelfsema et al., 2018; Troyk, 2017). Despite these advances, the stimulation paradigm underlying VCPs has not changed. The fundamental assumption remains that multiple phosphenes created by concurrent (or nearly concurrent) stimulation of multiple electrodes will combine into coherent visual forms. However, in our testing with sighted and blind participants, the conventional stimulation paradigm was never able to evoke visual percepts of coherent forms. Instead, participants reported perceiving multiple isolated phosphenes, consistent with earlier reports (Bosking et al., 2018b; Dobelle et al., 1976; Schmidt et al., 1996). Using the same number of electrodes, the dynamic stimulation paradigm was able to evoke a wide variety of patterns that could be immediately reproduced and identified by the participant without any training.

A powerful demonstration of the effectiveness of dynamic stimulation was obtained in two blind participants. With only five or six electrodes, it would have been impossible to create a set of letter-like forms using static stimulation. By comparison, with dynamic stimulation, simply changing the order in which the same electrodes were stimulated allowed for the perception of multiple letter forms. Participants were able to reliably draw, name, and discriminate these forms. While we tested only letter-like shapes, the outlines of other common objects, such as faces, bodies, houses, cars, table or chairs could also be traced using the same principles. In combination with modern machine vision algorithms that can rapidly identify objects in visual scenes (LeCun et al., 2015), dynamic stimulation could be used to give blind participants a rapid outline of salient objects in their environment or to provide cues for navigation (Liu et al., 2018).

Our work is related to previous work on form vision with retinal (Klauke et al., 2011; Rizzo et al., 2003a, b; Shivdasani et al., 2017; Zrenner et al., 2011) and cortical visual prostheses (Dobelle et al., 1976). In many of these studies, electrical stimulation was delivered in a regular sequence of stimulated electrodes, rather than to all active electrodes at once. This results in less current entering cortical tissue, reducing the likelihood of epileptic seizures due to synchronized activation. Dynamic current steering differs in that the pattern of sequential stimulation is related to the form that is being conveyed to the participant, rather than a fixed order. An analogy can be made to a cathode ray tube video display, in which the image is formed through a fixed continuous raster of scan lines from the top to the bottom of the display; this would represent the traditional method. In contrast, in a vector graphics display such as a vintage oscilloscope, the cathode ray is deflected by a voltage to draw arbitrary shapes; this is analogous to dynamic current steering.

What underlies the greater efficacy of dynamic stimulation as opposed to static stimulation? When multiple electrodes are stimulated at once, the resulting phosphenes can interact, often coalescing into a single phosphene that is not recognizable as a shape (Bosking et al., 2018b). Dynamic stimulation seems to decrease the likelihood of this happening, increasing the effective spatial resolution of the patterns that can be generated.

Natural visual stimuli evoke activity in neurons tuned for particular features. For instance, a visual “T” activates only neurons selective for vertical and horizontal orientations. In contrast, electrical stimulation activates all neurons in the immediate vicinity of the electrode, regardless of their selectivity, an unnatural activity pattern that may impair perception (Beyeler et al., 2017). Dynamic stimulation may be an effective work-around for this limitation because it activates motion-processing circuits that in turn result in form perception, much as visual motion can produce form perception (Pavan et al., 2013).

Quantitative comparison with previous studies

Static stimulation produces the percept of multiple isolated phosphenes. Participants can learn an association between these phosphene patterns and individual characters in the Latin alphabet, so-called “visual braille” (reviewed in Brindley, 1982). An individual with an implanted cortical visual prosthetic was able to recognize characters in a visual braille alphabet at a rate of 30 per minute with 85% accuracy (Dobelle et al., 1976) while an individual with a retinal prosthetic recognized 40 braille letters per minute with 89% accuracy (Lauritzen et al., 2012). Similar speeds were achieved with dynamic current steering. Blind participant 03–281 recognized 86 forms per minute with 87% accuracy when two forms were presented in random order. With three forms, accuracy was 92% at 30 forms per minute.

For blind participant BAA, technical limitations of the implanted device meant that stimulation sequences could only be delivered at a rate of ~2 seconds per electrode (~10 seconds per sequence) resulting in the percept of successive phosphenes rather than a dynamic shape. Nevertheless, even this slow rate of dynamic stimulation allowed the participant to accurately integrate discrete phosphene sequences as uniquely recognizable letter forms. In contrast, static stimulation was never able to produce a recognizable letter percept in any of our participants.

Conclusions

Our work represents the continuation of decades of stimulating human and non-human primate visual cortex towards the goal of building a cortical visual prosthetic (Bak et al., 1990; Brindley and Lewin, 1968; Davis et al., 2012; DeYoe et al., 1989; Normann et al., 2009; Schiller and Tehovnik, 2008; Schmidt et al., 1996; Torab et al., 2011). Dynamic stimulation and dynamic current steering for VCPs also build on previous efforts to create optimal neural stimulation strategies in human neuroprosthetics which aim to ‘read in’ neural information (Roelfsema et al., 2018). For instance, current steering has been applied both in retinal implants (Dumm et al., 2014; Matteucci et al., 2013), cochlear implants (Firszt et al., 2007; Kalkman et al., 2016) and deep brain stimulators (Gross and McDougal, 2013) although it has not yet found clinical applications in the cerebral cortex. All experiments in the present manuscript were conducted on participants implanted with surface electrodes, necessitating current levels on the order of milliamps. Penetrating electrodes require much less current to evoke phosphenes, on the order of microamps (Bak et al., 1990). This reduced current will activate a smaller volume of cortex, changing the likelihood of interactions between adjacent electrodes in both static and dynamic stimulation paradigms. Advances in technology, including electrical stimulation with high-density grids of electrodes placed on the cortical surface (Khodagholy et al., 2015) or penetrating into the cortex (Rousche and Normann, 1998) and non-electrical stimulation with optogenetic (Deisseroth, 2015) magnetothermal (Chen et al., 2015) or focused ultrasound techniques (Legon et al., 2014) promise dramatic increases in our ability to stimulate human cortex. As general-purpose stimulation paradigms, dynamic stimulation and dynamic current steering maybe be used in combination with any of these technologies to help restore useful visual function to blind people, and to permit more efficient transformation of information within other cortical prosthetic applications.

STAR Methods

Resource Availability

Lead Contact

Further information and requests for resources should be directed to and will be fulfilled by the Lead Contact, Daniel Yoshor (daniel.yoshor@pennmedicine.upenn.edu).

Materials Availability

This study did not generate new unique reagents.

Data and Code Availability

The datasets and code generated during this study have been deposited in the DataDryad repository with identifier doi:10.5061/dryad.gtht76hhk.

Experimental Model and Subject Details

The participants for this study consisted of three sighted participants with medically intractable epilepsy (all male; anonymized subject codes YBN, YAY, YBH, ages 20–54) tested at Baylor College of Medicine (BCM); one blind participant (female, age 35; anonymized subject code BAA) tested at University of California, Los Angeles (UCLA); and one blind participant (male, age 58; anonymized subject code 03–281) tested at Baylor College of Medicine. In all participants, subdural electrodes were implanted on the surface of the occipital lobe. The electrodes were stimulated, and the resulting percepts examined. The Institutional Review Boards of BCM and UCLA approved all research protocols, and all participants gave written informed consent.

A preliminary version of this manuscript was deposited in the bioRxiv preprint server (https://www.biorxiv.org/content/10.1101/462697v1).

Epileptic participants

Clinical electrodes were implanted for monitoring of epileptogenic activity, with electrode placement guided solely by clinical criteria. Additional research electrodes (embedded in the same silastic strips used for clinical monitoring) were implanted and stimulated for the studies described here. Clinical monitoring continued uninterrupted during experimental sessions. Participants were hospitalized in the epilepsy-monitoring unit for 4 to 14 days after electrode implantation. During experiments, the participants remained seated comfortably in their hospital bed. A ground pad was adhered to the participant’s thigh and except where noted, all electrical stimulation was monopolar. Electrical stimulation currents were generated using a 16-channel system (AlphaLab SnR, Alpha Omega, Alpharetta, GA) controlled by custom code written in MATLAB (Version 2013b, The MathWorks Inc., Natick, MA). For all participants, the epilepsy seizure focus was determined to be distant from visual cortex. For participant YBN, the research electrodes consisted of a six by four grid of electrodes (total of 24 electrodes). Each electrode was 0.5 mm in diameter with a center-to-center spacing of 2 mm. For sighted participants YAY and YBH, the research electrodes consisted of 16 electrodes. Each electrode was 0.5 mm in diameter with a center-to-center spacing of 4 mm or 6 mm.

Blind participants

Blind participant BAA acquired blindness at age 27 and had minimal residual light perception. As a component of an early feasibility study for the development of a visual cortical prosthetic, BAA underwent surgical implantation of a responsive neurostimulator developed to treat epilepsy (RNS System, Neuropace, MountainView, CA) containing electrodes located on two separate silastic strips. Each strip contained 4 electrodes. Each electrode was 3.18 mm in diameter with a center-to-center spacing of 10 mm. Participant BAA was tested as an outpatient as described in (Niketeghad et al., 2019). Blind participant 03–281 (male, age 58) acquired blindness at age 46 and had no light perception. The participant was implanted with the Orion Visual Cortical Prosthesis System (Second Sight Medical Products, Inc., Sylmar, CA) at age 57. The Orion contained 60 electrodes arranged in 10 rows, with interelectrode spacing of 4.2 mm within rows and 3.0 mm across rows.

Method Details

Electrode Localization and Visualization

Before surgery, T1-weighted structural magnetic resonance imaging scans were used to create cortical surface models with FreeSurfer (Dale et al., 1999; Fischl et al., 1999) and visualized using SUMA (Argall et al., 2006). Participants underwent a whole-head CT after the electrode implantation surgery. The post-surgical CT scan and pre-surgical MR scan were aligned using Analysis of Functional Neuroimaging (AFNI) software (Cox, 1996) and all electrode positions were marked manually on the structural MR images. Electrode positions were then projected to the nearest node on the cortical surface model using the AFNI program SurfaceMetrics.

Screening to Determine Responsive Electrodes

First, all electrodes were screened to identify responsive electrodes i.e. those that produced a phosphene when electrical stimulation was delivered. In each trial, participants verbally reported whether they experienced a localized, brief, visual percept similar to a flash of light. During each trial, an auditory warning tone cued the participants to fixate visual crosshairs. This was followed by a second tone that indicated the beginning of the electrical stimulation period. Electrical stimulation consisted of a train of biphasic pulses (−/+), with 0.1 ms pulse duration per phase, delivered at a frequency of 200 Hz, with an overall stimulus train duration of 200 or 300 ms. Currents tested ranged from 0.3 – 4.0 mA in sighted participants and up to 7.5 mA in blind participants, with maximum charge delivered per screening trial of 4 μC for sighted and 9 μC for blind. For each electrode, trials were initiated with a low current (0.3–1.0 mA) that gradually increased on successive trials until the participant reported a phosphene. If no phosphene was obtained at the maximum current levels of 4 mA in sighted and 7.5 mA in blind then the site was considered unresponsive.

Quantitative Phosphene Mapping Using Electrical Stimulation

To quantify phosphene locations, additional experiments were performed on each of the electrodes identified in the screening stage. The participant fixated visual crosshairs and electrical stimulation was delivered to a single electrode using the parameters that elicited a phosphene for that electrode in the screening stage. The participant drew the outline of the phosphene on a touchscreen; a cloth tape measure was used to measure the distance between the participant’s face and the touchscreen in order to accurately assess visual angle. The distance was adjusted so that the participant could draw on the touchscreen comfortably. Three to five trials per electrode were typically conducted. On the first trial, the participant was instructed to draw the shape as accurately as possible. On subsequent trials, the participant adjusted the size and location of the phosphene using a custom designed graphical user interface so that it matched the phosphene as precisely as possible. For participant YBH, phosphenes were drawn with a pen and paper instead of a touchscreen. The participant inspected the drawing following the trial. If it did not match the percept, an additional trial was performed, and a new drawing created. The paper drawings were digitized using a flatbed scanner. Phosphene drawings for each electrode (touchscreen or pen and paper) were fit with an ellipse for quantification and display. Blind participants were instructed to touch (and attend to) a small Velcro square placed on a touchscreen and trace the location or outline of the phosphene percept on the touchscreen.

Quantification and Statistical Analysis

Receptive Field Mapping Using Visual Stimuli

For sighted participants YBH and YBN the visual responses of each electrode were measured (Bosking et al., 2017b; Ozker et al., 2018; Yoshor et al., 2007); Supplementary Figure 1 illustrates receptive field mapping for two sample electrodes from participant YBN. The participant viewed an LCD screen at a distance of 57 cm. Square checkerboards were presented at screen locations that varied randomly from interval to interval (checkerboard duration 167 ms, blank interval of 167 ms between locations). Each checkerboard subtended 2° and contained a five-by-five grid of black and white checks, resulting in a spatial frequency of 2.5 cycles per degree. To ensure fixation, participants performed a letter detection task at the fixation point. A Cerebus amplifier (Blackrock Microsystems, Salt Lake City, UT) record electrode signals referenced to an inactive intracranial electrode implanted facing the skull. Signals were amplified, filtered (low-pass: 500 Hz, Butterworth filter with order 4; high-pass: 0.3 Hz, Butterworth filter with order 1) and digitized at 2 kHz. For each electrode, the average visual response evoked by the checkerboards presented at each location was measured. Outliers were discarded and the visual response was smoothed with a Savitzky-Golay polynomial filter (order 5). The evoked potential at each location was converted to a single value by calculating the root mean square deviation from baseline during the time window from 50 ms to 250 ms after stimulus onset. To estimate the spatial receptive field of the ensemble of neurons underlying the electrode, the amplitudes of the evoked responses were fit with a two-dimensional Gaussian function. To allow visualization of the receptive field as a single discrete shape, the half-maximum value of the fitted Gaussian was plotted.

Electrical stimulation details

All stimulation consisted of cathodic-first biphasic (charge-balanced) pulses with a frequency of 200 Hz and a pulse width per phase of 100 microseconds, unless noted otherwise. For monopolar stimulation, the stimulation ground was a conductive pad, typically attached to the participant’s leg. Bipolar stimulation (for current steering) occurred between adjacent stimulating electrodes.

The current amplitude for each electrode was held constant and was the same amplitude as that used for phosphene mapping of individual electrodes (equal to the minimum current that reliably produced a phosphene during the screening stage).

Dynamic current steering (monopolar)

Monopolar dynamic current steering was used to create virtual electrodes in between the physical electrodes on the array by using current steering. Current steering consisted of simultaneous stimulation of two adjacent electrodes in the sequence at a particular current ratio. The ratio is adjusted to change the location of the virtual electrode. The ratio can be held constant for an entire pulse train (as illustrated in Figure 2) or varied for each individual pulse within the pulse train (as illustrated in Figure 3). The rate of change of the current ratio determines how rapidly the pattern is “drawn” on the cortex and how dynamic the pattern is perceived to be.

A sample dynamic current steering sequence, using three virtual electrodes in between each real pair of electrodes, would consist of electrode 1 stimulation at 100% current, followed by electrode 1 stimulation at 80% current and electrode 2 stimulation at 50% current (creating a virtual electrode near electrode 1), then electrode 1 stimulation at 70% current and electrode 2 stimulation at 70% current (creating a virtual electrode midway between electrodes 1 and 2), then electrode 1 stimulation at 50% and electrode 2 stimulation at 80% (creating a virtual electrode near electrode 2), and then electrode 2 stimulation at 100% current, and so forth throughout the remainder of the sequence.

Dynamic current steering (bipolar)

In bipolar dynamic current steering sequences, virtual electrodes were generated by using bipolar stimulation between two electrodes in the sequence. A typical bipolar dynamic current steering sequence would begin with monopolar stimulation of electrode 1 alone, followed by bipolar stimulation between electrode 1 and 2, and then monopolar stimulation of electrode 2 alone, etc., throughout the remainder of the sequence.

Dynamic stimulation sequences used

Sighted participant YBN:

Dynamic stimulation sequences were used with a current range of 1.2–1.5 mA per electrode. Each electrode was stimulated for a duration of 50 ms with a gap interval between successive electrodes of 50 ms.

Sighted participant YAY:

Bipolar dynamic current steering sequences were used with a current range of 0.7–1.5 mA per electrode. The stimulation duration for each real or virtual electrode was 200 ms and the gap interval between electrodes was 125 ms.

Sighted participant YBH:

Dynamic sequences were used with a current range of 2.0–3.0 mA. The stimulation duration for each electrode was 50 ms and the gap interval between electrodes was 10 ms.

Blind participant BAA:

Dynamic stimulation sequences were used with all electrodes stimulated at 2.0 mA. The stimulation duration for each electrode was 200 ms and the gap interval between electrodes was 2000 ms. The long gap interval on this participant was due to limitations of the control system for the implanted device.

Blind participant 03–281:

At each testing session, current values were determined for the electrodes under investigation. For the first electrode (F10), a stimulation pulse of 120 Hz pulse frequency with 100 ms pulse train duration (biphasic, symmetric, cathodal first) was delivered at low current. The current was gradually increased until the participant reported a clearly visible and distinct phosphene. This current value was used as the baseline current for all testing for electrode F10. This procedure was repeated for all electrodes used in the pattern sequence. For each electrode being calibrated, the participant was asked if the phosphene was of approximately equivalent brightness as the previous electrode; if it was dimmer, the current was increased until the brightness was equated. At higher currents, the participant reported that the brightness saturated and the phosphene increased in size. If this phenomenon was observed, the current was reduced to a value that produced a phosphene of equivalent brightness without an increase in size. Currents ranged between 3.5 and 7.5 mA per electrode.

Behavioral Tests in Participants with Implanted Electrodes

To assess the participants’ ability to make perceptual discriminations between different electrical stimulation sequences we used a forced choice discrimination task. Before discrimination testing, the participants drew the perceived pattern on the touchscreen several times and they were instructed to associate a particular letter with each stimulation sequence. During each trial of the discrimination task, a single sequence was presented while the participant fixated, or attended to, a defined place on the touchscreen, and then the participants gave a verbal report to indicate which of the sequences they had perceived. Sequences were presented in pseudo-random order. The statistical significance of the accuracy values was obtained using the binom.test() function in “R”.

In the blind participants, we used multidimensional scaling (MDS) to assess the reliability of differences between the percepts elicited by different electrical stimulation sequences. For participant BAA, one of seven stimulation sequences was presented on each trial corresponding to one of 7 forms (G, N, R, U, V, W, Z; electrodes shown in Figure 5D). Each shape was repeated 4 times for a total of 28 trials. For each trial, the drawing made by the participant on the touchscreen was converted into an ordered set of hundreds of evenly spaced circles using Adobe Illustrator. The x and y location of the center of each circle, and hence each point in the original drawing, was then obtained using the regionprops() function in Matlab. The list of coordinates corresponding to each trial was resampled to obtain exactly 100 points and a correlation matrix 28 × 28 in size was created by obtaining the correlation between the ordered list of x, y points from each trial, and the ordered list of points from every other trial, using the corr2() function in Matlab. The correlation matrix was used as input to Matlab code that performed the MDS analysis. For participant 03–281, a similar analysis was performed, except that only four different stimulation sequences were tested (W, N, M, U; electrodes shown in Figure 6D).

In participant 03–281, two experiments examined the rate at which forms could be delivered using dynamic stimulation. In the first experiment, two stimulation sequences were tested in which four electrodes were successively stimulated for 50 ms each (no gap between successive electrodes), producing a total sequence time of 200 ms. In the first sequence, the electrodes were stimulated in an order from the highest to the lowest in the visual field, producing the percept of a line drawn in a downward direction (electrode order F10, E07, C06, B10; see Figure 6C for individual phosphene locations). In the second sequence, the electrodes were stimulated in the reverse order, producing the percept of a line drawn in an upward direction. All stimulation was delivered at 120 Hz and the current for each electrode was F10, 6.0 mA; E07, 6.5 mA; C06, 6.5 mA; B10, 6.5 mA. Following each 200 ms sequence, there was a 500 ms response window during which the participant verbally reported his percept (“down” or “up”).

In the second experiment, three sequences were tested in which four electrodes were successively stimulated for 100 ms with a 100 ms gap between successive electrodes for a total sequence time of 700 ms. In the first sequence, the electrodes were stimulated to produce the percept of a “C” shape (electrode order F01, F10, B10, C04; see Figure 6C for individual phosphene locations). In the second sequence, the electrodes were stimulated to produce the percept of a “U” shape (F10, B10, C04, F01). For the third sequence, the percept was a backwards “C” (B10, C04, F01, F10). All stimulation was delivered at 120 Hz and the current for each electrode was B10, 7.0 mA; C04, 7.0 mA; F01, 6.5 mA; F10, 6.5 mA. Following each 700 ms sequence, there was a 1300 ms response window during which the participant verbally reported the percept.

Additional Resources

Blind participant 03–281 was tested under the auspices of a clinical trial entitled “Early Feasibility Study of the Orion Visual Cortical Prosthesis System” (NCT03344848).

Supplementary Material

1

Supplementary Video 1. Related to Figure 4. Video of sighted participant YAY drawing a “Z” shape in response to a bipolar dynamic current steering stimulation sequence.

Download video file (34.4MB, mp4)
2

Supplementary Video 2. Related to Figure 5. Video of blind participant BAA first drawing the location of phosphenes perceived when electrodes were stimulated individually, and then drawing each of three letter forms obtained with dynamic stimulation.

Download video file (169.8MB, mp4)
3

Supplementary Video 3. Related to Figure 6. Video of blind participant 03–281 drawing letter forms obtained with dynamic stimulation.

Download video file (65.7MB, mp4)
4

Supplementary Video 4. Related to Figure 6. Video of blind participant 03–281 identifying rapid sequences of forms.

Download video file (200.6MB, mp4)
Supplementary figure

Supplementary Figure 1. Related to Figure 4. Methods for constructing electrode receptive fields. (A) Dark blue ellipse shows the location of the receptive field under examination (see Figure 4 for electrode array). (B) Square (2°) checkerboards were presented at 49 different visual field locations. Each small blue trace shows the mean visual evoked potential in the window from 0 ms to 500 ms after the onset of a checkerboard presented at that visual field location (range for each trace, −600 μV to +600 μV). (C) The amplitude of the evoked potential in the window from 50 ms to 250 ms after stimulus onset (root mean square voltage) was calculated. The color at each visual field location shows the amplitude of the evoked response at that location (for display purposes, one level of interpolation was used). (D) A two-dimensional Gaussian was fit to the (un-interpolated) response amplitude matrix. The color shows the amplitude of the Gaussian at each visual field location. The black line shows the contour of the half-maximum value of the fitted Gaussian and was used for plotting receptive field location, as in (A). (E) Dark blue ellipse shows the location of the receptive field under examination. (F) Blue traces show the mean visual evoked potential at each visual field location (range for each trace, −700 μV to +700 μV). (G) Amplitude of the evoked response at each visual field location. (H) Two-dimensional Gaussian fit to the evoked response data.

Highlights.

  • Visual cortical prosthetic stimulation paradigms were examined

  • Letters or other shapes were traced on the surface of cortex with electrical current

  • Static stimulation was unable to produce letter percept in participants

  • Dynamic current steering evoked percepts of letter forms by blind and sighted participants

Acknowledgments

The authors gratefully acknowledge the immense effort invested in the research by the participants and the assistance of Second Sight Medical Products. This work was supported by NIH R01EY023336, UH3NS103442, R01NS065395 and R00MH103479.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Declaration of interests

WB, NP and DY receive research funding from Second Sight Medical Products, Inc., a manufacturer of visual cortical prosthetics. NP is also a consultant to Second Sight. A provisional patent application describing dynamic current steering (serial no. 62/638,365) was filed with the U.S. Patent and Trademark Office on March 5, 2018, entitled “Systems and Computer-Implemented Methods of Conveying a Visual Image to a Blind Subject Fitted with a Visual Prosthesis.”

References

  1. Argall BD, Saad ZS, and Beauchamp MS (2006). Simplified intersubject averaging on the cortical surface using SUMA. Hum Brain Mapp 27, 14–27. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bak M, Girvin JP, Hambrecht FT, Kufta CV, Loeb GE, and Schmidt EM (1990). Visual sensations produced by intracortical microstimulation of the human occipital cortex. Med Biol Eng Comput 28, 257–259. [DOI] [PubMed] [Google Scholar]
  3. Beyeler M, Rokem A, Boynton GM, and Fine I (2017). Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies. Journal of Neural Engineering 14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bosking WH, Beauchamp MS, and Yoshor D (2017a). Electrical Stimulation of Visual Cortex: Relevance for the Development of Visual Cortical Prosthetics. Annu Rev Vis Sci 3, 141–166. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bosking WH, Foster B, Sun P, Beauchamp MS, and Yoshor D (2018a). Rules Governing Perception of Multiple Phosphenes by Human Observers. In bioRxiv. [Google Scholar]
  6. Bosking WH, Foster B, Sun P, Beauchamp MS, and Yoshor D (2018b). Rules Governing Perception of Multiple Phosphenes by Human Observers. bioRxiv, 302547. [Google Scholar]
  7. Bosking WH, Sun P, Ozker M, Pei X, Foster BL, Beauchamp MS, and Yoshor D (2017b). Saturation in Phosphene Size with Increasing Current Levels Delivered to Human Visual Cortex. J Neurosci 37, 7188–7197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Brindley GS (1982). Effects of electrical stimulation of the visual cortex. Hum Neurobiol 1, 281–283. [PubMed] [Google Scholar]
  9. Brindley GS, and Lewin WS (1968). The sensations produced by electrical stimulation of the visual cortex. J Physiol 196, 479–493. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Chen R, Romero G, Christiansen MG, Mohr A, and Anikeeva P (2015). Wireless magnetothermal deep brain stimulation. Science 347, 1477–1480. [DOI] [PubMed] [Google Scholar]
  11. Christie BP, Ashmont KR, House PA, and Greger B (2016). Approaches to a cortical vision prosthesis: implications of electrode size and placement. Journal of Neural Engineering 13. [DOI] [PubMed] [Google Scholar]
  12. Cox RW (1996). AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical research 29, 162–173. [DOI] [PubMed] [Google Scholar]
  13. Dale AM, Fischl B, and Sereno MI (1999). Cortical surface-based analysis. I. Segmentation and surface reconstruction. Neuroimage 9, 179–194. [DOI] [PubMed] [Google Scholar]
  14. Davis TS, Parker RA, House PA, Bagley E, Wendelken S, Normann RA, and Greger B (2012). Spatial and temporal characteristics of V1 microstimulation during chronic implantation of a microelectrode array in a behaving macaque. J Neural Eng 9, 065003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Deisseroth K (2015). Optogenetics: 10 years of microbial opsins in neuroscience. Nat Neurosci 18, 1213–1225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. DeYoe EA, Lewine J, and Doty RW (1989). Optimal stimuli for detection of intracortical currents applied to striate cortex of awake macaque monkeys. Paper presented at: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society. [Google Scholar]
  17. Dobelle WH, Mladejovsky MG, Evans JR, Roberts TS, and Girvin JP (1976). “Braille” reading by a blind volunteer by visual cortex stimulation. Nature 259, 111–112. [DOI] [PubMed] [Google Scholar]
  18. Dumm G, Fallon JB, Williams CE, and Shivdasani MN (2014). Virtual electrodes by current steering in retinal prostheses. Investigative ophthalmology & visual science 55, 8077–8085. [DOI] [PubMed] [Google Scholar]
  19. Firszt JB, Koch DB, Downing M, and Litvak L (2007). Current steering creates additional pitch percepts in adult cochlear implant recipients. Otol Neurotol 28, 629–636. [DOI] [PubMed] [Google Scholar]
  20. Fischl B, Sereno MI, and Dale AM (1999). Cortical surface-based analysis. II: Inflation, flattening, and a surface-based coordinate system. Neuroimage 9, 195–207. [DOI] [PubMed] [Google Scholar]
  21. Foroushani AN, Pack CC, and Sawan M (2018). Cortical visual prostheses: from microstimulation to functional percept. Journal of neural engineering 15, 021005. [DOI] [PubMed] [Google Scholar]
  22. Gross RE, and McDougal ME (2013). Technological advances in the surgical treatment of movement disorders. Curr Neurol Neurosci Rep 13, 371. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kalkman RK, Briaire JJ, and Frijns JH (2016). Stimulation strategies and electrode design in computational models of the electrically stimulated cochlea: An overview of existing literature. Network 27, 107–134. [DOI] [PubMed] [Google Scholar]
  24. Khodagholy D, Gelinas JN, Thesen T, Doyle W, Devinsky O, Malliaras GG, and Buzsaki G (2015). NeuroGrid: recording action potentials from the surface of the brain. Nat Neurosci 18, 310–315. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Klauke S, Goertz M, Rein S, Hoehl D, Thomas U, Eckhorn R, Bremmer F, and Wachtler T (2011). Stimulation with a wireless intraocular epiretinal implant elicits visual percepts in blind humans. Invest Ophthalmol Vis Sci 52, 449–455. [DOI] [PubMed] [Google Scholar]
  26. Lauritzen TZ, Harris J, Mohand-Said S, Sahel JA, Dorn JD, McClure K, and Greenberg RJ (2012). Reading visual braille with a retinal prosthesis. Front Neurosci 6, 168. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. LeCun Y, Bengio Y, and Hinton G (2015). Deep learning. Nature 521, 436. [DOI] [PubMed] [Google Scholar]
  28. Legon W, Sato TF, Opitz A, Mueller J, Barbour A, Williams A, and Tyler WJ (2014). Transcranial focused ultrasound modulates the activity of primary somatosensory cortex in humans. Nat Neurosci 17, 322–329. [DOI] [PubMed] [Google Scholar]
  29. Lewis PM, Ackland HM, Lowery AJ, and Rosenfeld JV (2015). Restoration of vision in blind individuals using bionic devices: a review with a focus on cortical visual prostheses. Brain Research 1595, 51–73. [DOI] [PubMed] [Google Scholar]
  30. Lewis PM, Ayton LN, Guymer RH, Lowery AJ, Blamey PJ, Allen PJ, Luu CD, and Rosenfeld JV (2016). Advances in implantable bionic devices for blindness: a review. ANZ J Surg. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Liu Y, Stiles NR, and Meister M (2018). Augmented reality powers a cognitive assistant for the blind. Elife 7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Lowery AJ (2013). Introducing the monash vision group’s cortical prosthesis. Paper presented at: 2013 IEEE International Conference on Image Processing (IEEE). [Google Scholar]
  33. Matteucci PB, Chen SC, Tsai D, Dodds CW, Dokos S, Morley JW, Lovell NH, and Suaning GJ (2013). Current steering in retinal stimulation via a quasimonopolar stimulation paradigm. Investigative ophthalmology & visual science 54, 4307–4320. [DOI] [PubMed] [Google Scholar]
  34. Mirochnik RM, and Pezaris JS (2019). Contemporary approaches to visual prostheses. Mil Med Res 6, 19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Murphey DK, Maunsell JH, Beauchamp MS, and Yoshor D (2009). Perceiving electrical stimulation of identified human visual areas. Proceedings of the National Academy of Sciences of the United States of America 106, 5389–5393. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Niketeghad S, Muralidharan A, Patel U, Dorn JD, Bonelli L, Greenberg RJ, and Pouratian N (2019). Phosphene perceptions and safety of chronic visual cortex stimulation in a blind subject. Journal of neurosurgery, 1–8. [DOI] [PubMed] [Google Scholar]
  37. Normann RA, Greger B, House P, Romero SF, Pelayo F, and Fernandez E (2009). Toward the development of a cortically based visual neuroprosthesis. J Neural Eng 6, 035001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Ozker M, Yoshor D, and Beauchamp MS (2018). Frontal cortex selects representations of the talker’s mouth to aid in speech perception. Elife 7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Pavan A, Marotti RB, and Mather G (2013). Motion-form interactions beyond the motion integration level: evidence for interactions between orientation and optic flow signals. J Vis 13, 16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Penfield W, and Rasmussen T (1950). The cerebral cortex of man A clinical study of localization of function (New York: The Macmillan Company; ). [Google Scholar]
  41. Rizzo JF 3rd, Wyatt J, Loewenstein J, Kelly S, and Shire D (2003a). Methods and perceptual thresholds for short-term electrical stimulation of human retina with microelectrode arrays. Invest Ophthalmol Vis Sci 44, 5355–5361. [DOI] [PubMed] [Google Scholar]
  42. Rizzo JF 3rd, Wyatt J, Loewenstein J, Kelly S, and Shire D (2003b). Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Invest Ophthalmol Vis Sci 44, 5362–5369. [DOI] [PubMed] [Google Scholar]
  43. Roelfsema PR, Denys D, and Klink PC (2018). Mind Reading and Writing: The Future of Neurotechnology. Trends Cogn Sci 22, 598–610. [DOI] [PubMed] [Google Scholar]
  44. Rosenke M, Weiner KS, Barnett MA, Zilles K, Amunts K, Goebel R, and Grill-Spector K (2018). A cross-validated cytoarchitectonic atlas of the human ventral visual stream. Neuroimage 170, 257–270. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Rousche PJ, and Normann RA (1998). Chronic recording capability of the Utah Intracortical Electrode Array in cat sensory cortex. J Neurosci Methods 82, 1–15. [DOI] [PubMed] [Google Scholar]
  46. Schiller PH, and Tehovnik EJ (2008). Visual prosthesis. Perception 37, 1529–1559. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Schmidt EM, Bak MJ, Hambrecht FT, Kufta CV, O’Rourke DK, and Vallabhanath P (1996). Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex. Brain 119 (Pt 2), 507–522. [DOI] [PubMed] [Google Scholar]
  48. Shivdasani MN, Sinclair NC, Gillespie LN, Petoe MA, Titchener SA, Fallon JB, Perera T, Pardinas-Diaz D, Barnes NM, Blamey PJ, et al. (2017). Identification of Characters and Localization of Images Using Direct Multiple-Electrode Stimulation With a Suprachoroidal Retinal Prosthesis. Invest Ophthalmol Vis Sci 58, 3962–3974. [DOI] [PubMed] [Google Scholar]
  49. Tehovnik EJ, and Slocum WM (2013). Electrical induction of vision. Neurosci Biobehav Rev 37, 803–818. [DOI] [PubMed] [Google Scholar]
  50. Torab K, Davis T, Warren D, House P, Normann R, and Greger B (2011). Multiple factors may influence the performance of a visual prosthesis based on intracortical microstimulation: nonhuman primate behavioural experimentation. Journal of neural engineering 8, 035001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Troyk PR (2017). The Intracortical Visual Prosthesis Project. In Artificial Vision (Springer), pp. 203–214. [Google Scholar]
  52. Yoshor D, Bosking WH, Ghose GM, and Maunsell JH (2007). Receptive fields in human visual cortex mapped with surface electrodes. Cerebral cortex 17, 2293–2302. [DOI] [PubMed] [Google Scholar]
  53. Zrenner E, Bartz-Schmidt KU, Benav H, Besch D, Bruckmann A, Gabel VP, Gekeler F, Greppmaier U, Harscher A, Kibbel S, et al. (2011). Subretinal electronic chips allow blind patients to read letters and combine them to words. Proc Biol Sci 278, 1489–1497. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1

Supplementary Video 1. Related to Figure 4. Video of sighted participant YAY drawing a “Z” shape in response to a bipolar dynamic current steering stimulation sequence.

Download video file (34.4MB, mp4)
2

Supplementary Video 2. Related to Figure 5. Video of blind participant BAA first drawing the location of phosphenes perceived when electrodes were stimulated individually, and then drawing each of three letter forms obtained with dynamic stimulation.

Download video file (169.8MB, mp4)
3

Supplementary Video 3. Related to Figure 6. Video of blind participant 03–281 drawing letter forms obtained with dynamic stimulation.

Download video file (65.7MB, mp4)
4

Supplementary Video 4. Related to Figure 6. Video of blind participant 03–281 identifying rapid sequences of forms.

Download video file (200.6MB, mp4)
Supplementary figure

Supplementary Figure 1. Related to Figure 4. Methods for constructing electrode receptive fields. (A) Dark blue ellipse shows the location of the receptive field under examination (see Figure 4 for electrode array). (B) Square (2°) checkerboards were presented at 49 different visual field locations. Each small blue trace shows the mean visual evoked potential in the window from 0 ms to 500 ms after the onset of a checkerboard presented at that visual field location (range for each trace, −600 μV to +600 μV). (C) The amplitude of the evoked potential in the window from 50 ms to 250 ms after stimulus onset (root mean square voltage) was calculated. The color at each visual field location shows the amplitude of the evoked response at that location (for display purposes, one level of interpolation was used). (D) A two-dimensional Gaussian was fit to the (un-interpolated) response amplitude matrix. The color shows the amplitude of the Gaussian at each visual field location. The black line shows the contour of the half-maximum value of the fitted Gaussian and was used for plotting receptive field location, as in (A). (E) Dark blue ellipse shows the location of the receptive field under examination. (F) Blue traces show the mean visual evoked potential at each visual field location (range for each trace, −700 μV to +700 μV). (G) Amplitude of the evoked response at each visual field location. (H) Two-dimensional Gaussian fit to the evoked response data.

NIHMS1586199-supplement-Supplementary_figure.jpg (760KB, jpg)

Data Availability Statement

The datasets and code generated during this study have been deposited in the DataDryad repository with identifier doi:10.5061/dryad.gtht76hhk.

RESOURCES