Electrical Stimulation of Visual Cortex: Relevance for the Development of Visual Cortical Prosthetics

Abstract

Electrical stimulation of the cerebral cortex is a powerful tool for exploring cortical function. Stimulation of early visual cortical areas is easily detected by subjects and produces simple visual percepts known as phosphenes. A device implanted in visual cortex that generates patterns of phosphenes could be used as a substitute for natural vision in blind patients. We review the possibilities and limitations of such a device, termed a visual cortical prosthetic. Currently, we can predict the location and size of phosphenes produced by stimulation of single electrodes. A functional prosthetic, however, must produce spatial temporal patterns of activity that will result in the perception of complex visual objects. Although stimulation of later visual cortical areas alone usually does not lead to a visual percept, it can alter visual perception and the performance of visual behaviors, and training subjects to use signals injected into these areas may be possible.

1. INTRODUCTION

Direct application of electrical current [direct current stimulation (DCS)] to brain tissue as a technique to evoke neural activity has both clinical and research uses. It has long been known that stimulation of the human occipital cortex results in the perception of a small, bright spot of light known as a phosphene. This observation over a century ago offered strong evidence that neural activity in the occipital cortex is causally related to vision and has provided an important tool for studying the neural basis of visual perception in human subjects.

Although experiments in human patients offer the advantage of having subjects who can explain their percepts with verbal or written descriptions, experiments involving DCS in trained behaving monkeys also provide useful information. Stimulation of visual cortex in nonhuman primates has been a key technique for understanding the functional role of different cortical areas in the performance of visually guided behaviors. Despite recent advances with other stimulation techniques, such as optogenetics, DCS remains the technique of choice for clinical mapping studies in human patients and experiments in humans and nonhuman primates that require direct activation of specific regions of the brain.

Electrical stimulation of visual cortex has long been recognized for its potential in the development of a visual cortical prosthetic device (VCP) for use in blind subjects [reviewed by Lewis & Rosenfeld (2016)]. Several prototype devices have been tested with varying degrees of success (Bak et al. 1990; Brindley & Lewin 1968a; Dobelle 2000; Dobelle & Mladejovsky 1974; Dobelle et al. 1974, 1976; Schmidt et al. 1996). Advances in wireless and implantable device technologies have renewed interest in the field, with several groups attempting to bring devices to clinical trials in the near future using either penetrating electrodes (Bradley et al. 2005, Lowery et al. 2015, Troyk et al. 2005) or surface electrodes (Luo & da Cruz 2016; Second Sight Med. Prod. 2015, 2016). These new efforts render it timely to review the current understanding of DCS of visual cortex and discuss further advances that could improve prosthetics. The reader is referred to other publications for descriptions of noninvasive stimulation techniques, such as transcranial magnetic stimulation (TMS) or transcranial direct current stimulation (tDCS); for comprehensive accounts of the use of electrical stimulation in nonhuman primates (Cicmil & Krug 2015, Cohen & Newsome 2004, Histed et al. 2013); for lists of all cortical areas stimulated (Histed et al. 2013, Selimbeyoglu & Parvizi 2010); and for the history and current state of VCPs (Fernandes et al. 2012, Lewis & Rosenfeld 2016, Lewis et al. 2015, Normann et al. 2009, Schiller & Tehovnik 2008, Tehovnik & Slocum 2013, Tehovnik et al. 2009).

We argue that most results from DCS of visual cortex can be understood by evaluating three factors: the spatial extent of the activated population of neurons, the functional organization of the cortical area stimulated, and the task or context in which the stimulation is delivered to the subject. In keeping with this framework, we first review some essential information about the functional organization of visual cortex. We then review the ways that DCS has been used in nonhuman primates, including experiments that have evaluated how DCS actually affects cortical circuits. Finally, we examine how electrical stimulation has been used in human subjects, including previous attempts to create VCPs, and current work in this area.

2. BRIEF REVIEW OF VISUAL CORTICAL ORGANIZATION

To fully understand the effects of electrical stimulation of visual cortex, it is important to first consider the complex organization of the visual pathways in the primate brain. Primates are highly visual animals, and a large portion of the cerebral cortex is devoted to visual processing and use of visual information to guide behavior. One of the defining organizational properties of the cerebral cortex is that neurons are highly organized at multiple scales, ranging from the clustering of cells with similar response properties, to functional mapping within single areas, to the grouping of multiple cortical areas relevant for particular behaviors. We first examine functional organization within a single visual area, the primary visual cortex (V1).

2.1. Functional Organization in the Primary Visual Cortex

A primary defining characteristic of neurons in visual cortex is that they respond at a high firing rate when stimuli are placed in a specific part of the visual field, known as the receptive field (RF) (Hubel & Wiesel 1959, 1968). The location of the RF of cortical neurons remains relatively constant within a vertical column that extends orthogonal to the cortical surface and progressively changes with movement across the cortical surface. Following this organization, the entire visual field is mapped across V1 in a very orderly fashion. The precision of this retinotopic mapping has been demonstrated beautifully by lesion studies in humans (Horton & Hoyt 1991) (Figure 1a,b), by metabolic labeling studies in monkeys (Tootell et al. 1988), and by functional magnetic resonance imaging (fMRI) in humans (Engel et al. 1997, Sereno et al. 1995, Wandell et al. 2007, Wandell & Winawer 2011) (Figure 1c,d ).

Figure 1.

Retinotopic maps in early human visual cortex. (a) Medial view of the left occipital cortex. Dashed lines indicate the location of the primary visual cortex (V1), which is partially buried within the calcarine sulcus. Numbers indicate eccentricity in degrees of visual space from the fovea. (b) Schematic showing V1 after it has been extracted from the cortex shown in panel a and flattened. Numbers indicate iso-eccentricity and iso-azimuth lines in the map of visual space. (c) Medial view of the occipital cortex showing locationof V1, visual area 2 (V2), and visual area 3 (V3), based on retinotopic mapping experiments with functional magnetic resonance imaging (fMRI). Dotted lines and numbers indicate the location of iso-eccentricity lines. Dashed lines indicate location of the calcarine sulcus and the V1/V2 border. (d ) Medial view of the occipital cortex indicating iso-eccentricity regions of V1, V2,and V3, as determined by fMRI. White dashed lines indicate the borders of V2 and V3. Abbreviation: HM, horizontal meridian. Panels a and b reproduced with permission from Horton & Hoyt (1991). Panels c and d reproduced with permission from Wandell & Winawer (2011).

Importantly, different parts of the visual field are not represented equally in V1. More cortical space is devoted to the representation of the fovea and the central visual field than to more peripheral regions of visual space. The amount of cortex (in millimeters) devoted to representation of 1° of visual space is called the linear cortical magnification factor (CMF), and this factor varies substantially across the map of visual space (Dougherty et al. 2003, Duncan & Boynton 2003, Harvey & Dumoulin 2011, Horton & Hoyt 1991).

Cells in visual cortex are also tuned to respond to particular visual features, or attributes of objects, in the visual world. For example, cells in V1 respond most strongly when an edge or contour placed at a particular orientation passes through the RF (Hubel & Wiesel 1959). As with RF location, cells with similar orientation preferences are organized in a columnar fashion, and overall, orientation preference is mapped smoothly across the cortical surface (Blasdel 1992; Blasdel & Salama 1986; Bonhoeffer & Grinvald 1991; Grinvald et al. 1986, 1991; Ikezoe et al. 2013; Nauhaus et al. 2012; Ohki et al. 2005, 2006; Ts’o et al. 1990). The orientation preference map is just one example of a feature map. There are other features, such as ocular dominance, that are also mapped in V1, and the overall set of maps is overlaid in a complex fashion.

2.2. Visual Hierarchy

Visual cortex contains many different visual areas beyond V1, each containing a map of the visual field (Wandell & Winawer 2011, Wandell et al. 2007). The exact structure of the visual field map varies from area to area, as does the set of stimulus features to which the neurons are most responsive. Progressing from V1 toward later stages of the cortical hierarchy, RFs become larger (Dumoulin & Wandell 2008, Harvey & Dumoulin 2011, Yoshor et al. 2007), the map of visual space becomes somewhat less organized, and cells become selective for complex objects rather than single features, including places, faces, body parts, or objects of different types.

In nonhuman primates, columnar organization for features’ selectivity exists within many of these later cortical areas. For example, in visual area 2 (V2), features such as binocular disparity, color, and contrast are mapped into particular domains (Chen et al. 2008; Lu & Roe 2007, 2008; Sincich & Horton 2005). In the middle temporal area (MT), columnar organization has been demonstrated for direction of motion and disparity (Albright et al. 1984, DeAngelis & Newsome 1999). Finally, in the inferotemporal cortex (IT), there are columns of cells that are tuned to respond to particular features of objects or faces (Tanaka 1996, Wang et al. 1996).

Recently, fMRI has revealed a similar organization in early visual cortex of human subjects (Nasr et al. 2016) (Figure 2a). Furthermore, a recent model has proposed organization of the human ventral temporal cortex (VTC) with spatial structure at three levels: clustering of cells with similar low-level response properties; modules for processing of faces, body parts, objects, and places; and large-scale separation between areas that respond to animate and inanimate objects (Figure 2b) (Grill-Spector & Weiner 2014).

Figure 2.

Functional domains in human visual cortex. (a) Mapping of functional domains in the primary visual cortex (V1), visual area 2 (V2), and visual area 3 (V3), using functional magnetic resonance imaging in one subject. Black, green, and blue lines indicate the borders of V1, V2, and V3. In the left panel, yellow indicates the location of color-selective regions. In the middle panel, yellow areas and black outlines indicate the location of disparity-selective regions. The right-hand panel shows the interdigitation of the color and disparity domains in V2 and V3. Reproduced with permission from Nasr et al. (2016). (b) Overlapping functional representations in the ventral temporal cortex, including mapping of animacy (top), domains selective for processing of faces or body parts (middle), and clustering of neurons with similar response properties (bottom). Reproduced with permission from Grill-Spector & Weiner (2014).

Although the mix of response properties and connections within each visual area is unique, it is important to emphasize that the performance of any visually guided behavior relies on the coordinated activity of large and distributed populations of cells, residing in multiple visual cortical areas. Understanding the effects of electrical stimulation requires knowledge about how the activity patterns generated by stimulation intersect with these complex and overlapping functional maps.

3. ELECTRICAL STIMULATION OF VISUAL CORTEX IN NONHUMAN PRIMATES

Macaque monkeys are highly visual primates whose visual abilities and visual pathways are similar to humans’. Much of our detailed knowledge about the structure and function of visual cortex has come from electrophysiological and electrical stimulation experiments in monkeys.

3.1. Electrical Stimulation Used to Alter Behavior

One of the primary ways DCS has been used in nonhuman primates is to alter performance of a visually guided behavior, demonstrating a causal role for a given area in the behavior. For example, Newsome and colleagues used DCS to examine the role of MT in the performance of a motion discrimination task (Cohen & Newsome 2004; Newsome & Salzman 1993; Newsome et al. 1990; Salzman & Newsome 1994; Salzman et al. 1990, 1992). When a small amount of current (10–20 μA) was introduced into a column of cells within MT that preferred a particular direction of visual motion, the animal’s choices were biased toward that direction, demonstrating a key role for MT in motion discrimination. An important follow-up experiment (Murasugi et al. 1993) found that usage of larger currents (40–80 μA) resulted in a nonspecific disruption of behavior instead. The authors inferred that low currents activated a single direction-preference column in MT (<200 μm), whereas higher currents activated multiple direction-preference columns, leading to task disruption.

DCS has been used in a similar fashion to test other hypotheses about the functional role of MT and other visual areas. For example, stimulation of MT has also been used to alter performance in depth discrimination tasks (DeAngelis et al. 1998, Uka & DeAngelis 2006) and form-from-motion perception (Krug et al. 2013). Stimulation in the middle superior temporal area has been used to alter direction discrimination (Celebrini & Newsome 1995) or heading discrimination (Britten & van Wezel 1998). Stimulation of visual area 4 (V4) has been used to alter performance in a fine-depth discrimination task (Shiozaki et al. 2012), and stimulation of the IT has been used to alter performance in a face-categorization task (Afraz et al. 2006). In addition to providing evidence for the overall role of particular areas in visual processing, electrical stimulation has also been used to probe how signals are actually integrated to guide behavior in particular tasks (Born et al. 2000, DeAngelis & Newsome 2004, Groh et al. 1997, Nichols & Newsome 2002, Salzman & Newsome 1994). Overall, these experiments have emphasized that the effects of electrical stimulation depend on the functional organization of the cortical area that is stimulated, the magnitude of the current used for stimulation, and the task that is performed by the animal.

3.2. Detection of Electrical Stimulation Without Presentation of Visual Stimuli

To examine whether DCS of all visual cortical areas is equally detectable, monkeys were trained in a temporal two-alternative forced choice (2AFC) task in which an electrical stimulus was delivered during one of two time intervals (Murphey & Maunsell 2007). The animals quickly learned to detect stimulation of early visual areas (V1, V2). However, detection of stimulation of later areas [visual area 3A (V3A), MT, IT] required more training, and slightly higher currents, to be effective. These results indicate that animals can be trained to detect DCS of multiple cortical areas and not just the primary sensory area (in this case V1). A later set of experiments found that although animals quickly learned to detect V1 stimulation at moderate current levels (~50 μA), training with thousands of trials was required to reach best performance at low current levels (~5–10 μA; Figure 3) (Ni & Maunsell 2010). The improved performance in detection of electrical stimuli obtained with training persisted for a long period of time (up to one year). Furthermore, there was a trade-off in the ability to detect electrical stimuli and the ability to detect visual stimuli placed in the RF of the stimulation site. Overall, training to detect DCS had many parallels with the training required to perform a difficult perceptual discrimination with a real visual stimulus (Histed et al. 2013, Ni & Maunsell 2010).

Figure 3.

Training produces sustained improvement in detection of microstimulation of the primary visual cortex (V1). Exponential decrease in macaque monkey behavioral threshold for detection of microstimulation at a single site in V1. Each point indicates a threshold determination based on 100 trials, and error bars reflect 67% confidence intervals. Detection threshold was reexamined after a one-year pause and remained at the trained level. Reproduced with permission from Ni & Maunsell (2010).

3.3. Electrical Stimulation Used to Measure Characteristics of Phosphenes

As just discussed, monkeys can easily be trained to detect DCS of visual cortex. However, the exact nature of the generated percept is unknown, making it difficult to assess its utility for a VCP. Depending on the area stimulated, the percept could have been a simple phosphene, a more complex visual percept (such as a disembodied face part or whole face when stimulating a face-selective region of cortex), or a completely nonvisual percept (such as a hunch that stimulation occurred during a certain time period). Whereas humans can easily provide a verbal report about the nature of the percept, monkeys are unable to do this. Two methods have been utilized to elicit behavioral reports from monkeys about the quality of stimulation-induced percepts.

3.3.1. Estimation of phosphene characteristics using saccade delay fields.

On the basis of reports of human patients, monkeys likely perceive phosphenes during V1 stimulation. To examine the properties of these putative phosphenes in more detail, researchers trained monkeys to make saccades to visual targets, and then DCS was delivered just prior to the execution of the saccade (Tehovnik et al. 2004). By manipulating stimulation parameters and other elements of this task, Tehovnik and colleagues were able to make some important inferences about how DCS affects the activity of neural populations in V1 and how this population activity is linked to behavior.

By moving the location of the visual target to different locations relative to the RF of the stimulation site, they mapped what was termed a delay field, which is the region of visual space for which stimulation induced a saccade delay (Tehovnik et al. 2005b). The size of the delay field was related to the magnitude of the current delivered (Tehovnik & Slocum 2007a) and to the eccentricity of the stimulation site within the map of visual space (Tehovnik et al. 2005b). Specifically, sites that were closer to the representation of the fovea had smaller delay fields, and the rate at which delay-field size increased with eccentricity was correlated with the rate of change in the CMF with eccentricity (Tehovnik & Slocum 2007c, Tehovnik et al. 2005b). These experiments strongly implied that phosphene size was related to the stimulation current and the CMF at the stimulation site (Tehovnik & Slocum 2007b,c; Tehovnik et al. 2005b).

3.3.2. Estimation of phosphene characteristics by comparison to visual targets.

Further demonstration that the animals perceived phosphenes, and more direct measurement of the characteristics of these phosphenes, was provided by a set of experiments in which monkeys were trained in 2AFC tasks to compare the size, color, or contrast of a perceived phosphene to that of an actual visual stimulus (Figure 4) (Schiller et al. 2011). The monkeys appear to have perceived a phosphene approximately 0.25° in size and equivalent to a visual stimulus that is set at 50% contrast with injection of 30–40 μA of current, based on these experiments. These experiments are generally in accord with the previous measures of delay-field size (Tehovnik & Slocum 2007a, Tehovnik et al. 2005a); however, in this case, the authors did not systematically evaluate changes in phosphene size with current or eccentricity.

Figure 4.

Contrast and size of percepts created by electrical stimulation of the primary visual cortex. Macaque monkeys learned a two-alternative forced choice task where they were rewarded for making a saccade to the target that was perceived as higher in contrast or larger in size. (a,b) Two-alternative forced choice task using only visual targets. Plots show the percentage of times the animal selected the lower target when two visual targets were displayed and the contrast (a) or size (b) of the upper target was varied. (c,d ) Two-alternative forced choice task using one visual stimulus and one electrical stimulation site. Percentage of times the animal chose the target that corresponded to the electrical stimulation site as the contrast (c) or size (d ) of the visual target was modulated. Reproduced with permission from Schiller et al. (2011). Abbreviation: RF, receptive field.

3.4. Measuring Spread of Activity and Cortical Integration with Electrical Stimulation

A number of different methods have been used to directly measure the distribution of cortical activity that results from DCS. The first experiments to examine this issue used electrical stimulation through one electrode combined with electrophysiological recording through a second, nearby electrode (Stoney et al. 1968). These experiments found that the direct activation of cortical neurons was local and could be described by a simple equation for passive current spread.

Experiments that combined electrical stimulation with fMRI found that much larger regions of V1 were activated by DCS than was predicted by passive current spread alone (Tehovnik et al. 2006, Tolias et al. 2005) (Figure 5). The amount of additional spread was several millimeters in each direction. This distance approximately matches the extent of horizontal connections within V1, suggesting these as a conduit for the lateral spread of activity. Similar spread has been found by combining DCS with optical imaging of intrinsic signals (Brock et al. 2013), optical imaging with voltage-sensitive dyes (Seidemann et al. 2002), and two-photon imaging of calcium signals in mice (Histed et al. 2009).

Figure 5.

Blood-oxygen-level-dependent activity induced by electrical microstimulation of the primary visual cortex (V1). (a) A statistically thresholded functional map superimposed over an anatomical image of one horizontal brain slice from one subject. Electrical stimulation at a site in V1 resulted in significant activation on the operculum of area V1, the posterior bank of the lunate sulcus [visual area 2 (V2)/visual area 3 (V3)], and the posterior bank of the superior temporal sulcus [middle temporal area (MT)]. (b) Radius of spread of activity in V1 as a function of current amplitude. There is a significant correlation between current amplitude and radius (R² = 0.06, F = 4.5, p < 0.05, linear regression). The radius of current spread is 1.8 mm (standard error of mean = 0.27 mm) beyond that predicted for direct activation of pyramidal neurons. Abbreviations: A, anterior; LS, lateral sulcus; STS, superior temporal sulcus; V5, visual area 5. Reproduced with permission from Tolias et al. (2005).

These recording studies suggest that DCS leads to neuronal activity outside of the area directly activated by the injected current, implying a role for synaptic spread (Tehovnik et al. 2006). However, the behavioral effects of stimulation (such as the size of saccade delay fields) appear to be more correlated with the directly activated region of cortex. This could be due to stronger or more synchronous activation of the directly activated region, leading to greater influence on perception (Logothetis et al. 2010).

DCS appears to result in spread of activity across cortex via transsynaptic excitation, and it seems clear that strong cortical inhibition is also evoked. When stimulation was delivered to the lateral geniculate nucleus, the superior colliculus, pulvinar, and V1 were activated, but V2 and MT were suppressed (Logothetis et al. 2010), likely because of the recruitment of inhibitory circuits. Understanding how DCS influences the balance of cortical excitation and inhibition, and hence the propagation of signals through the network, will be important for the development of VCPs.

DCS has also been used to probe the limits of cortical integration. In one series of behavioral experiments, it was found that concurrent stimulation of two electrodes in V1 separated by less than 1 mm resulted in summation of cortical signals, whereas greater separation distances resulted in independent processing of the two signals (Ghose & Maunsell 2012). Further investigation is needed to determine whether this result will hold in other cortical areas, at higher stimulus currents, and when other stimulation parameters are varied.

4. ELECTRICAL STIMULATION OF VISUAL CORTEX IN HUMANS

The first large-scale and systematic reports of electrical stimulation of human visual cortex were made by neurosurgeon Wilder Penfield (Penfield 1947, Penfield & Rasmussen 1950). Penfield reported on the nature of the visual percepts that subjects reported following DCS (phosphenes) and demonstrated that the sites in cortex that were most likely to produce a visual percept were those located near the occipital pole. Following these pioneering investigations by Penfield, there have been three basic ways in which testing of DCS of human visual cortex has been attempted: (a) testing of prototype VCPs in blind patients, (b) stimulation of surface or penetrating electrodes semichronically implanted in patients undergoing neurophysiological monitoring for clinical purposes, and (c) stimulation of patients in the operating room during a cranial surgery.

4.1. Early Visual Cortical Prosthetic Devices

The first attempts at using electrical stimulation to create a VCP for use in blind patients were made by Button (Button 1958, Button & Putnam 1962). The initial device was made from crude components, and had only four penetrating electrodes, but was successful in evoking limited visual percepts in the subject (Lewis & Rosenfeld 2016). This demonstrated that activation of the occipital cortex can result in visual percepts even in subjects with long-standing blindness.

The next attempt at VCPs used wireless transmission of signals to an array of surface electrodes implanted on the occipital pole and the medial face of the occipital lobe in two patients (Brindley & Lewin 1968a,b; Brindley et al. 1972; Rushton & Brindley 1978). Mapping the phosphenes produced by this device formed a half-hourglass pattern in the contralateral visual field. This distribution is to be expected when the medial wall of V1 is used for stimulation, because the visual field map representing the horizontal meridian is buried within the calcarine sulcus, inaccessible to surface electrodes (Horton & Hoyt 1991) (Figure 1a). Subjects reported seeing simple patterns when multiple electrodes were stimulated at once and that phosphenes appeared larger with stimulation of more peripheral locations of visual cortex.

Another attempt at a VCP used surface electrodes with wired connections from the implanted array implanted in dozens of sighted patients undergoing occipital lobe surgery and ultimately in seven blind patients (Dobelle 2000; Dobelle & Mladejovsky 1974; Dobelle et al. 1974, 1976). Dobelle and colleagues reported that subjects could discriminate simple patterns when multiple electrodes were stimulated, and one subject could reportedly read braille patterns presented at a rapid rate. However, these impressive claims were not well quantified or documented. Both the Brindley & Lewin research group and the Dobelle research group reported that currents in the milliampere range were required to produce phosphenes, and the creation of distinct phosphenes required a separation of ~2–4 mm between adjacent surface electrodes.

A VCP with penetrating microelectrodes was tested in three subjects with normal sight undergoing occipital craniotomies (Bak et al. 1990) and in one blind subject (Schmidt et al. 1996). Electrodes were inserted in pairs separated by 250, 500, or 750 μm. Most pairs of electrodes separated by 500 μm or more generated two distinct phosphenes, and the threshold current was low (mostly below 25 μA). Because of technical difficulties, however, the development of the device ceased.

4.2. Next-Generation Visual Cortical Prosthetic Devices

Largely because of advances in brain–computer interfaces, there is renewed interest in VCPs. The Monash Vision Group based in Melbourne, Australia, has described plans for a device illustrated in Figure 6a (Lowery et al. 2015). The device would consist of a pair of glasses with a camera, a processing unit carried in a pocket, and wireless transmission to arrays of penetrating electrodes placed on tiles designed to float on the cortical surface (Figure 6b). Multiple tiles would be placed on the lateral occipital surface and the medial wall of the occipital cortex for a total of hundreds of electrodes and presumably phosphenes. This tile strategy is advantageous over a single device because of the convoluted sulcal and gyral anatomy of the human brain; a single flat chip cannot mold to a large region of cortex. A similar strategy utilizing multiple independent tiles is under development by a group at the Illinois Institute of Technology directed by Philip Troyk (Bradley et al. 2005; Bredeson et al. 2015; Musallam et al. 2007; Troyk et al. 2003, 2005). A VCP that uses surface, rather than penetrating, electrodes is under development by the Second Sight corporation, which has plans to modify its Argus II retinal prosthetic device (Luo & da Cruz 2016) for use in visual cortex. The cortical device will have a similar configuration of 60 or more platinum electrodes embedded in silastic, with wireless communication to an external receiver.

Figure 6.

Monash Group visual cortical prosthetic device. (a) Overview of entire device. The subject wears glasses that contain a camera, electronics, and a coil for sending data to the implanted device. The glasses communicate with a pocket processor that performs some signal processing before sending signals back to the glasses and the transmitter coil. The coil provides a wireless link to the electrode tiles that are placed over visual cortex. (b) Floating electrode array with 43 penetrating electrodes. Multiple tiles would be placed over the occipital pole and medial wall of the occipital cortex. Panel a reproduced with permission from Jeffrey Rosenfeld; panel b reproduced with permission from Lowery et al. (2015).

4.3. Electrical Stimulation in Epilepsy Patients

Epilepsy patients implanted with surface or penetrating electrodes for clinical purposes provide an extremely valuable opportunity for recording and DCS in human visual cortex. In these patients, unlike with occipital lobe surgery patients, testing of multiple brain areas, with the patient performing multiple behavioral tasks, can occur over a period of days. And, unlike in patients who have been blind for many years, it is possible to rapidly switch between visual and electrical stimulation experiments.

4.3.1. Cortical areas that produce phosphenes in humans.

DCS has been used in epileptic patients to examine which areas of human visual cortex are capable of producing phosphenes, using both qualitative surveys (Lee et al. 2000) and quantitative behavioral experiments using a two-interval forced choice task (Murphey et al. 2009). In these human experiments, Murphey et al. used retinotopic mapping with fMRI to define the borders of visual cortical areas, and the locations of the implanted electrodes were determined relative to these borders. DCS of early visual cortical areas (V1, V2, and V3) was far more likely to produce perception of a phosphene than was stimulation of visual cortical areas that lie later in the visual cortical hierarchy (Figure 7a). The greater effectiveness of V1/V2/V3 may relate to the increasing complexity of the organization of later visual areas (Figure 7a,b). We expect that DCS at the currents used activated several millimeters of cortex. In early visual cortical areas, which have precisely organized maps of visual space and simple receptive field properties, this activity pattern is interpreted as a spot in visual space (Figure 7b). Later visual areas may be predominantly organized by responses to complex features such as objects, body parts, faces, or locations. The same region of active cortex might therefore overlap a set of functional domains (Figure 7c) that would never be activated simultaneously for a natural visual stimulus and hence would be uninterpretable by the subject.

Figure 7.

Visual areas that produce phosphenes. (a) Each symbol ( gray circles) indicates the threshold current for detecting a phosphene at one site in the indicated cortical area. Cortical area assignments were made by aligning electrodes to retinotopic mapping data from functional magnetic resonance imaging in the same subjects. If the subject did not detect a phosphene at any current tested at that site, then the highest current tested was used to plot the symbol (red x’s). Data from Murphey et al. (2009). (b) Schematic indicating the typical location and size of the primary visual cortex (V1). Dashed lines indicate the approximate boundary of V1 and show iso-eccentricity lines in the map of visual space. With an electrical stimulation current of 2 mA delivered to a site in V1, we would expect to see about 5.3 mm in diameter activated (black circle) and would expect the subject to easily see a phosphene. (c) Schematic indicating potential functional organization in the ventral temporal cortex (VTC). Domains representing faces, face parts, body parts, and places can be found close together, potentially within the same 5.3-mm diameter (black circle). Thus, stimulation of this overall region might lead to no coherent percept. Abbreviations: FFA, fusiform face area; LO, lateral occipital area; MT, middle temporal area; PPA, parahippocampal place area; UVR, unidentified visually responsive area; V2, visual area 2; V3, visual area 3; V3A, visual area 3A; V4, visual area 4; V8, visual area 8.

4.3.2. Additional cortical areas required for phosphene perception.

Just as activity in the retina must propagate to many brain areas to allow for visual behavior and perception, phosphenes are not perceived by activation of V1 alone: Conscious awareness requires coordinated activity across a whole network of brain areas. To determine critical nodes in this network, we investigated cortical activity during trials in which the exact same stimulation current was applied to the same electrode in visual cortex. On some trials, subjects reported perceiving a phosphene whereas on other trials, they did not (Beauchamp et al. 2012). One cortical area, the temporal parietal junction (TPJ; Figure 8a), was much more active on trials when subjects did perceive a phosphene (Figure 8b,c). Furthermore, as the electrical stimulation current in early visual areas increased, the amount of activity in TPJ (but not other areas) also increased. Stimulation of sites in visual cortex that did not produce a phosphene also never induced TPJ activity. These results suggest that TPJ is a critical gatekeeper for determining whether activity in visual cortex reaches conscious perception. A recent study used fMRI to identify a region of TPJ that is responsive to visual stimulation (Horiguchi et al. 2016), and future studies could examine whether this area correlates with the region identified in our DCS experiments.

Figure 8.

Importance of the temporal parietal junction (TPJ) for perception of phosphenes. (a) Map showing the difference in gamma power between percept electrode stimulation and nonpercept electrode stimulation in one subject. One percept electrode and the nearest nonpercept electrode in the implanted hemisphere were repeatedly stimulated, and the significance of poststimulation difference in gamma power at each electrode (except for the stimulation electrodes) was calculated and mapped to the cortical surface. Black spheres show electrode locations. (b,c) TPJ response during stimulation with near-threshold currents, averaged across subjects. Data averaged from trials in which subjects correctly (b) or incorrectly (c) discriminated which of two intervals contained electrical stimulation. Stimulation current was the same for correct and incorrect trials. Reproduced with permission from Beauchamp et al. (2012).

4.3.4. Predicting phosphene location and size.

A fundamental tenet of systems neuroscience is that the receptive field properties of neurons in a given brain area corresponds to the contribution of that area to conscious vision. Electrical recording and stimulation of visual cortex provides an opportunity to test this tenet (Bosking et al. 2017) (Figure 9). By presenting visual stimuli and recording from an electrode over visual cortex, we can map the receptive field properties of small groups of neurons (Figure 9b). By stimulating the same electrode and interrogating the subject, we can then determine the conscious percept produced by the same group of neurons (Figure 9c). We found that RF and phosphene locations were highly correlated (Figure 9d,e).

Figure 9.

Location and size of phosphenes produced by stimulation of the primary visual cortex (V1). (a) A posterior-medial view of the occipital portion of the left hemisphere of one brain showing the typical placement of a custom electrode strip used for recording and stimulating in epilepsy patients. The strip wraps around the occipital pole and extends into the interhemispheric fissure. (b) Method for mapping receptive fields. Subjects performed a letter detection task at a central fixation point while checkerboard stimuli were flashed at various locations on the screen. (c) Method for mapping phosphenes. Subjects fixated on a cross on a touchscreen monitor while electrical stimulation was delivered and then drew the outline of the phosphene they perceived using a stylus. (d ) Receptive field (RF) eccentricity versus phosphene eccentricity tested in five subjects; each symbol represents testing from one electrode. (e) RF polar angle versus phosphene polar angle for the same subjects. ( f ) Phosphene size tested for six different electrical currents using one electrode located near the occipital pole in one subject (red circle inset). Square symbols show the actual data, the blue line shows a sigmoidal function fit to the data (Pearson correlation: r = 0.98; p < 0.001). ( g) Phosphene size versus eccentricity plotted for six electrodes in one subject (current = 2 mA). Blue line indicates linear regression (Spearman correlation: r = 0.83; p =0.058; 95% confidence interval = 0.0−1.0). Inset shows the location of the electrodes (red circles). Abbreviations: A, anterior, L, lateral, S, superior. Modified with permission from the Society for Neuroscience (Bosking et al. 2017).

To construct a VCP, control over phosphene size is also desirable. At least two parameters are likely to be important for phosphene size: magnitude of the current delivered and the eccentricity of the electrode within the map of visual space. It is intuitive to predict that higher currents should activate more cortex and result in larger phosphenes. Contrary to this simple prediction, however, we found that phosphenes increased in size as current was increased above threshold but rapidly reached a plateau above which size did not increase with current (Figure 9f ). On average, phosphene size plateaued at a size about three times larger than the size measured near threshold current. This could occur if the size of the population of active neurons also plateaued because of changes in the balance between excitatory and inhibitory circuits. Just as DCS of the thalamus evokes strong inhibition in visual cortex that limits the propagation of signals (Logothetis et al. 2010), stimulation of visual cortex at high currents may evoke inhibition, limiting the propagation of signals within V1 and from V1 to other areas. Phosphene size also varied with eccentricity (Figure 9g). Just as predicted by changes in the CMF, stimulation with the same current amplitude, presumably activating the same amount of cortex, results in increasingly larger phosphenes as eccentricity is increased.

To integrate these two results, we developed a simple model that accurately predicts phosphene size on the basis of current magnitude and eccentricity (Bosking et al. 2017) (Figure 10a,b). Our model is similar to the method used by Tehovnik and colleagues to predict saccade delay field (and presumed phosphene) size in monkeys (Figure 10c,d), with the key difference of a saturation term that is used to explain the observation of the plateau in phosphene size above a particular stimulation current. Similar results demonstrating the linkage between the size and location of phosphenes, and the stimulation parameters of electrode eccentricity and the magnitude of the electrical charge delivered, have also recently been obtained in studies that combined DCS, fMRI, and psychophysical measurements (Winawer & Parvizi 2016).

Figure 10.

Relationship between predicted cortical activity and behavior. (a) Schematic showing the map of visual space on a flattened human primary visual cortex (V1) (Horton & Hoyt 1991). The circles indicate the area of cortex that we predict would be active in our experiments when subjects are stimulated at three different points in the map of visual space (red circle =1° eccentricity, blue circle=7.5°, green circle = 20°) with a near-threshold current (0.8 mA: 2-mm diameter activation) and at two higher currents (1 mA: 3.5-mm =diameter activation; 2 mA: 5.3-mm diameter activation; open circles). Note that the predicted diameter of activation in V1 is the same at the three eccentricities shown, but the magnification factor changes substantially. Also note that the predicted diameter of activated cortex is the same for any current above 2 mA. (b) The phosphene location and size predicted for electrical stimulation at each of the sites shown in panel a. Phosphenes increase in size with both eccentricity and current. (c) Predicted cortical activation during nonhuman primate execution of saccades following electrical stimulation (Tehovnik & Slocum 2007b, Tehovnik et al. 2005a). The schematic indicates the map of visual space on a flattened macaque V1. For simplicity, the macaque map is a scaled replica of the map of visual space shown for human V1. The colored circles indicate the ~750-μm diameter activation area that is predicted using 100 μA. Again, the predicted activation diameter in V1 is the same at the two eccentricities shown, but the magnification factor changes. The inset to the right of the full map of V1 shows an expanded view of the region between 2.5° and 5° eccentricities for one sector of the map. (d ) The size of saccade delay fields based on the cortical activation at the two eccentricities shown in panel a. The delay field predicted for stimulation of the site at 4° eccentricity is larger because of the larger inverse magnification factor at this site. Reproduced with permission from the Society for Neuroscience (Bosking et al. 2017).

4.3.5. Stimulation in later visual cortical areas.

As originally observed by Penfield, DCS of areas outside of primary sensory areas usually does not produce a percept when delivered in isolation. Instead, stimulation of extraprimary areas typically disrupts normal function. This is used during clinical mapping, in which key cognitive functions, such as speech perception and production, are tested during stimulation. If the function is disrupted, then the cortex under the electrode is not resected, preventing postsurgical deficits.

As already discussed, DCS of visual areas beyond V3 in humans typically does not produce a phosphene (Murphey et al. 2009). However, stimulation of these areas often produces impairment in performance of visually guided behaviors or disruption in visual perception when a visual stimulus is present. For example, disruption in the performance of motion discrimination has been demonstrated with stimulation of motion selective areas (Becker et al. 2013, Blanke et al. 2002), and impairment in facial recognition, without disruption of basic visual processing capabilities or recognition of other objects, has been observed with stimulation of face-selective areas (Allison et al. 1994; Jonas et al. 2012, 2014b, 2015; Mundel et al. 2003; Puce et al. 1999). In these cases, stimulation led to impairment in performance of the task, although the patient still reported clearly seeing the stimulus. In other cases, however, DCS was found to actually alter the perception of the currently presented visual stimulus. For example, stimulation of face-selective areas led to large distortions in the perception of faces that were present in the patient’s room (Parvizi et al. 2012, Rangarajan & Parvizi 2016, Rangarajan et al. 2014).

Although much more rare, stimulation of later visual areas can sometimes evoke a visual percept even when delivered in isolation. These percepts can be simple phosphenes (Lee et al. 2000; Murphey et al. 2008, 2009; Puce et al. 1999), movement of the entire visual field or moving objects (Rauschecker et al. 2011, Richer et al. 1991), complex objects such as faces or face parts ( Jonas et al. 2014a, Lee et al. 2000, Puce et al. 1999), or hallucinations of visual scenes that the patient is familiar with (Figure 11b) (Megevand et al. 2014).

Figure 11.

Stimulation of face- and place-selective regions of the ventral temporal cortex. (a) Comparison of electrical brain stimulation (EBS) and high-frequency broadband (HFB) electrocorticography (ECoG) responses for electrodes in the fusiform gyrus (FG) in ten subjects. Electrode locations from all subjects are shown in Montreal Neurological Institute (MNI) standard brain space. Because electrodes are normalized from native space to MNI space, some electrodes appear outside of the FG on the standard brain; these electrodes were included on the basis of their location within the FG in native space. White halos around the electrodes indicate a preferential HFB response to faces ( p < 0.01, false discovery rate corrected for all electrodes). Color indicates the result of the EBS at each site. There are face-selective sites in both hemispheres, but sites with face-related effects of EBS (red ) are found only in the right hemisphere. Reproduced with permission from Rangarajan et al. (2014). (b) Regions that displayed greater blood-oxygen-level-dependent responses to houses than other visual stimuli are overlaid on the patient’s isometric axial T1 magnetic resonance imaging. The location of two penetrating basal temporal electrodes DHb and DTIp are also displayed. The inset shows a coronal slice in the plane of DTIp. The effects of direct electrical stimulation of pairs of neighboring electrodes are depicted as colored circles, or stars if there were afterdischarges as a result of the stimulation. The axial slice shown was in the plane of contacts DTIp5 through DTIp2 and of contacts DHb4 and DHb3. DTIp7 and DTIp1 as well as DHb6 and DHb5 were on the slice immediately (1-mm) dorsal and were projected down; DTIp8 was 2-mm dorsal and was also projected down; DHb2 and DHb1 were 1-mm ventral and were projected up. Abbreviations: L, left; R, right. Reproduced with permission from Megevand et al. (2014).

There are several factors that may help to explain the variety of effects that have been observed with DCS of later visual areas. As already discussed, failure to produce a percept with stimulation of these areas could be because of the intersection between the population activity evoked by the stimulation and the underlying functional organization for visual space and other visual features (Figure 7b,c). In the few cases where visual percepts are generated by stimulation of these areas, this may be due to the specific location of the stimulation electrode within the local functional maps. For example, in a case where a colored phosphene is observed with stimulation of a later area, this can be due to the centering of the electrode over a particular color domain (Murphey et al. 2008).

However, there are several other points to consider. With movement through the cortical hierarchy, the nature of the information represented in each area changes from being mostly sensory toward being a mix of sensory and motor information and toward being more dependent on top-down modulation. Thus it may not be surprising that stimulation of one of these areas does not evoke a clear visual percept. As discussed earlier, when monkeys detect electrical stimulation, we do not know what they perceive. They could have learned to report something other than a visual percept, such as the desire to make an eye movement toward a particular part of the screen. As discussed earlier, another factor is that electrical stimulation in one cortical area can lead to inhibition in later downstream areas (Logothetis et al. 2010). It is important to remember that any effect, or lack of effect, from electrical stimulation can be due to this disruption in the normal flow of activity and not just to the pattern of population activity generated in the area that is directly stimulated. Finally, the effects of electrical stimulation, as with the effects of visual stimulation, are likely to be task dependent. For example, in some cases, subjects were instructed to simply attend to particular people, faces, or objects within the room, whereas in other cases, they performed forced-choice tasks with stimuli presented on a monitor.

Because of these complexities, it is critical that we continue to perform electrical stimulation in these areas in the rare human subjects where this is possible. First, human subjects can report the subtle perceptual effects that are induced by stimulation and can quickly switch between different tasks without extensive training. Second, stimulation can be used to test ideas or hypotheses that were developed on the basis of neuroimaging or recording data. For example, although selectivity for faces can be observed in both the left and right VTC, patients who develop prosopagnosia almost always have lesions in the right VTC. Recent experiments have revealed that face-related percepts are far more likely to be generated by electrical stimulation of modules in the right VTC, further validating the selective importance of the right side in the perception of faces (Figure 11a) ( Jonas et al. 2014a, Rangarajan & Parvizi 2016, Rangarajan et al. 2014).

Finally, continued experiments that perform electrocorticography (ECoG) recordings and electrical stimulation in later visual areas in humans are also critical for the development of VCPs. ECoG recordings from these areas can help resolve how electrical stimulation changes propagation through cortical networks and how we can modify stimulation to ensure activation of later areas. Furthermore, we will need to learn whether stimulation of these areas can enhance the subtlety and complexity of percepts that are generated by stimulation of early areas and also whether subjects can learn to perceive the stimulation of these areas provided in isolation. Understanding the local functional organization may be critical for deployment of a VCP. It may be possible to train a subject to use any pattern of stimulation, or it may be the case that the underlying functional domains set limits on the amount of plasticity and learning that is possible in each area.

5. CONCLUSIONS AND REMAINING QUESTIONS

The percepts produced by DCS of visual cortex can be understood given an accurate prediction for the spread of cortical activity combined with an understanding of the underlying functional maps in the stimulated area. For some visual areas where stimulation has been demonstrated to produce only interruption of function, more specific perceptual or behavioral effects might occur if electrodes were centered over functional domains and smaller electrodes or currents were used to stimulate only a population of neurons sharing a common response property, better mirroring the responses observed during natural vision.

Our review highlights several critical questions concerning the development of a useful visual prosthetic. Can the signals produced with direct activation of visual cortex via electrical stimulation be made to successfully propagate to higher visual areas and ultimately to influence perception and behavior in a useful manner? How can groups of electrodes be stimulated to generate spatial patterns of input that will propagate through the visual hierarchy and be interpreted as coherent visual shapes? Note that while there are many new techniques for activation of cortical neurons, including optogenetics, these techniques are likely to face many of the same issues as DCS related to integration and propagation of artificially created neural patterns of activity into a useful visual percept.

Simply adding more and more electrodes (like adding pixels in a computer display) may not allow for the perception of more complex forms if these electrodes do not evoke spatiotemporal patterns of activity similar to those that result from retinal stimulation. New stimulation paradigms may be required that find ways to safely and effectively generate spatial patterns of cortical activity without leading to massive inhibition at later cortical stages.

To create an effective cortical visual prosthetic, it will likely be necessary to stimulate multiple early visual areas. Because much of V1 is buried in the calcarine sulcus, creating a set of phosphenes that span the entire visual field will require stimulation of V2 and V3 in which more of the area is accessible to grids of implanted electrodes (Srivastava et al. 2007). However, it is unknown whether simultaneous stimulation in multiple areas (or even within a single area) will produce phosphenes that can be integrated into a single coherent pattern.

Finally, training is likely to be crucial in allowing patients to make full use of any implanted cortical prosthetic. We know that learning can improve detection of DCS of V1 in nonhuman primates (Ni & Maunsell 2010) and also that nonhuman primates can learn to detect stimulation of many later visual areas, but whether this information would be useful for a visual cortical prosthetic is open to debate. It may be possible to partially test the role of learning in the detection and usage of signals from later visual areas in epilepsy patients, but full testing will not be possible until the next VCPs are implanted in blind subjects.

SUMMARY POINTS.

1. Stimulation of early visual cortical areas produces simple percepts known as phosphenes.

2. Threshold currents for detection of phosphenes become lower with training.

3. Stimulation with penetrating microelectrodes and small currents likely activates a region of cortex several hundred microns in diameter.

4. Stimulation with cortical surface electrodes and moderate currents likely results in several millimeters of cortical activation.

5. Electrical stimulation can alter the flow of information through the cerebral cortex.

6. Stimulation of later visual areas, with no concurrent visual stimulation, typically does not result in a visual percept.

7. Stimulation of later visual areas can alter performance in a visual task or distort perception of visual objects.

8. The perceptual and behavioral results of electrical stimulation can be best understood by evaluating the intersection between the spatial pattern of cortical activation and the underlying functional maps.

FUTURE ISSUES.

1. It will be necessary to determine whether subjects can perceive coherent shapes or coherent motion when multiple electrodes in early visual areas are stimulated.

2. In cases where subjects do perceive coherent shapes or motion as a result of electrical stimulation, it will be useful to determine whether later visual cortical areas are activated.

3. It seems likely that we will need to develop new stimulation protocols that enhance the ability of subjects to perceive coherent shapes and maximize the transfer of information to the subject.

4. Another important step will be to determine whether human subjects can learn to perceive electrical stimulation of later visual cortical areas.

5. Finally, we must translate the knowledge obtained from studies in sighted subjects for use in next-generation visual cortical prosthetic devices in blind patients.

Glossary

DCS

direct current stimulation

VCP

visual cortical prosthetic device

V1

primary visual cortex

CMF

cortical magnification factor

ECoG

electrocorticography

VTC

ventral temporal cortex