Injecting Information into the Mammalian Cortex: Progress, Challenges, and Promise

Abstract

For 150 years artificial stimulation has been used to study the function of the nervous system. Such stimulation—whether electrical or optogenetic—eventually may be used in neuroprosthetic devices to replace lost sensory inputs and to otherwise introduce information into the nervous system. Efforts toward this goal can be classified broadly as either biomimetic or arbitrary. Biomimetic stimulation aims to mimic patterns of natural neural activity, so that the subject immediately experiences the artificial stimulation as if it were natural sensation. Arbitrary stimulation, in contrast, makes no attempt to mimic natural patterns of neural activity. Instead, different stimuli—at different locations and/or in different patterns—are assigned different meanings randomly. The subject’s time and effort then are required to learn to interpret different stimuli, a process that engages the brain’s inherent plasticity. Here we will examine progress in using artificial stimulation to inject information into the cerebral cortex and discuss the challenges for and the promise of future development.

Utilization of the neural elements of the cerebral cortex directly as receptors for conditional stimuli greatly extends the possible application of neurological techniques in the study of behavior. With this procedure the intricate and largely unknown elaboration of sensory data by the afferent system can be by-passed …

—Doty and others (1956)

Since the mid-19th century, when Fritsch and Hitzig evoked body movements somatotopically using electrical stimulation of a limited region of the canine cortex (Fritsch and Hitzig 1870), artificial stimulation has been used to study the function of the nervous system. In the 20th century, electrical stimulation was employed extensively to evoke and modify behavior—from investigating the cortical fields that control movement (Andersen and others 1975; Asanuma and Rosen 1972), to providing positive reinforcement (Olds and Milner 1954). Now in the 21st century, as brain-computer interfaces are being developed to restore lost function to individuals affected by injury, disease, or developmental disorder (Bockbrader and others 2018; Loeb 2018; Tsu and others 2015), artificial excitation of the nervous system can provide a means of introducing into the nervous system information that cannot enter through normal pathways.

In the present review we will examine current knowledge regarding artificial injection of specific, discriminable information into the nervous system, point out some of the challenges in making this an effective tool for studying nervous system function, and speculate on future developments. To focus on injecting information, we will minimize discussion of how stimulation has been used to investigate the normal functions of the nervous system. And although stimulation of subcortical and peripheral parts of the nervous system has great potential for delivering information artificially into the nervous system—as exemplified by the cochlear implant (Sun and others 2019)—here we will focus instead on injecting information into the mammalian cerebral cortex. Compared to stimulating the peripheral nerves or the deeper, more compact nuclei of the brain, the cerebral cortex historically has offered the advantages of lying largely just beneath the calvarium, making it relatively accessible, and of providing a relatively large target area.

Electrical stimulation is effective, of course, because almost all neurons naturally generate propagating electrical action potentials to transmit information, and such action potentials can be evoked by delivering electrical current artificially. Electrical stimulation therefore can be applied anywhere in the nervous system simply by putting an electrode in the desired location. Electrical stimulation, however, excites not only local neuronal somata but also axons passing through the vicinity even more readily (see Box 1).

Box 1. The Spatial Extent of Intracortical Microstimulation.

A microelectrode tip carefully positioned in the cerebral cortex can record action potentials from a single neuron or just a few neurons. Those unfamiliar with intracortical microstimulation (ICMS) often imagine that pulses of near-threshold current delivered through that microelectrode will evoke action potentials only in that(those) neuron(s). Such is not the case, however, for three basic reasons.

First, even if a single neuron is recorded by the microelectrode, numerous neuron somata in the vicinity will receive suprathreshold stimulation with even single, low-amplitude ICMS pulses. In the initial studies examining the spatial extent of ICMS, one microelectrode was positioned to record a pyramidal tract neuron (PTN) in cat M1 and a second microelectrode was lowered at 25 μm steps past the PTN, identifying the single-pulse (200 μs) cathodal current threshold needed to excite the PTN (Stoney and others 1968). Whereas minimum thresholds of ~1 to 17 μA could excite a PTN at the presumably closest position of the stimulating electrode, ICMS pulses of ~5 to 30 μA at 100 μm above or below this position could still excite the same PTN. Modeling current as falling off with the square of distance from the stimulating electrode, a single 20 μA × 200 μs (4 nC) cathodal pulse was estimated to excite PTNs within an 88 μm radius, which in cat M1 would include ~28 PTNs. PTNs are typically the largest neurons in the cortex, however. Subsequent studies have estimated that as many as 2175 additional low-threshold neurons may be excited directly by 10 μA × 200 μs (2 nC) pulses (Tehovnik 1996). In comparison to these near-threshold currents, substantially larger currents—up to 100 μA or even 500 μA—often have been employed in studies examining behavioral effects.

Second, besides the local neuron somata, their dendrites, and their axon initial segments, axons passing through the local neuropil will be stimulated (Nowak and Bullier 1998a, 1998b; Ranck 1975). In rat S1, single 8 μA × 100–600 μs (0.8–4.8 nC) pulses antidromically excite somata at distances up to 1350 μm (Butovas and Schwarz 2003). In cat visual cortex where lateral axons are known to extend several millimeters in layers 2/3, ICMS with trains of 10 μA × 200 μs (2 nC) pulses can activate somata up to 4 mm away (Histed and others 2009). Note that in addition to axons originating from local somata, axons arriving from other cortical areas and subcortical regions will be activated. Stimulating axons can both antidromically excite their somata of origin and orthodromically deliver postsynaptic potentials to their target neurons.

And third, much if not most of the effects of ICMS are trans-synaptic. In monkey M1 and F5 (ventral premotor cortex), single pulses of ≤400 μA × 200 μs (≤80 nC) evoke action potentials in corticospinal neurons most often only at latencies consistent with indirect (i.e., trans-synaptic) rather than direct activation (Maier and others 2013). Repetitive trains of ICMS pulses, by producing temporal summation of successive EPSPs, provide the opportunity for trans-synaptic excitation of additional neurons, even through oligo-synaptic linkages. In rodent agranular sensorimotor cortex, excitation of most intracellularly recorded layer 5 pyramidal neurons by ICMS trains (13 pulses of up to 60 μA × 200 μs (12 nC) at 333 Hz) delivered 100–150 μm away is abolished during blockade of synaptic transmission, demonstrating that most of the local activation normally produced by trains of ICMS is trans-synaptic (Hussin and others 2015). One of the earliest applications of ICMS provides perhaps the most overt example (Asanuma and Ward 1971). Trains of 4 to 20 μA × 200 μs (0.8–4.0 nC) ICMS pulses delivered at 300 Hz in the motor cortex of cats elicited overt contractions of forearm muscles. Cats have no monosynaptic connections from cortical neurons to spinal motoneurons. So such trains of ICMS pulses delivered in the cortex excited pools of spinal motoneurons through at least di-synaptic linkages from the cortex.

In recent years artificial excitation also has been produced optogenetically (Deisseroth 2015; Deng and others 2018; Galvan and others 2017). Optogenetic stimulation requires genetic manipulation to transduce neurons such that they express photoreceptor-linked membrane channels (channelrhodopsins). Illuminating transduced neurons with light of the appropriate wavelength then produces depolarization sufficient to trigger action potentials. While requiring genetic manipulation, optogenetics can achieve selective stimulation of specific types of neurons, which electrical stimulation cannot. But optogenetic stimulation has yet to be used extensively to inject specific, discriminable information into the nervous system, and the present review therefore will emphasize electrical stimulation.

We will distinguish two alternative approaches to injecting information using artificial excitation—biomimetic and arbitrary. The biomimetic approach endeavors to mimic the natural activity of neurons insofar as possible. Biomimetic stimulation is thought to have the advantage of seeming as natural as possible to the intact portions of the nervous system, thereby minimizing the cognitive burden of learning to interpret an artificial input. In the near future, however, biomimetic stimulation—whether electrical or optogenetic—is unlikely to be capable of reproducing the asynchronous action potentials in large numbers of cortical neurons that characterize the normal spatiotemporal patterns of neural activity naturally involved in any complex percept.

In contrast, the arbitrary approach makes no attempt to mimic the normal, natural patterns of neural activity. Instead, different artificial stimuli are associated randomly with different pieces of information, and the inherent ability of the nervous system to learn through adaptive, plastic processes is called upon to make the different, arbitrarily assigned patterns of stimulation interpretable. Arbitrary stimulation removes the need to emulate the natural neural activity underlying specific behaviors, and instead puts the burden of learning to interpret the incoming information on the subject’s own nervous system. While arbitrary stimulation lacks the long-term promise of accurately recreating percepts, in the short-term it may be as effective as biomimetic stimulation for injecting specific, discriminable information.

Biomimetic Stimulation

Somatosensory Cortex

Since the early report of two cases studied by Cushing, electrical stimulation of the surface of the postcentral gyrus in awake humans has been known to evoke somatic sensations (Cushing 1909). Subsequent studies have found that, although the evoked perceptions may include motion of a specific body part or stroking of the skin, subjects most commonly describe the evoked sensation as “numbness” and/or “tingling” (Penfield and Boldrey 1937; Penfield and Rasmussen 1950; Roux and others 2018). These artificially evoked sensations thus are similar to the abnormal sensations we all experience when a nerve “goes to sleep” as the result of temporary ischemia from external pressure or local anesthesia, and then subsequently recovers. But in contrast to the numbness and tingling evoked by stimulation through electrodes placed on the surface of the postcentral gyrus, electrical stimulation through microelectrodes inserted into the cortical gray matter (intracortical microstimulation or ICMS) in the primary somatosensory cortex (S1) has been found to produce cutaneous sensations comparable in certain respects to natural stimulation of the skin.

A number of distinct features of natural tactile stimulation delivered to the skin surface have been produced with ICMS in S1. In perhaps the earliest demonstration of biomimetic stimulation, monkeys initially were trained to indicate whether the second of two “fluttering” mechanical vibrations (in the range of 10–36 Hz) delivered to the skin of a fingertip was higher or lower in frequency than the first (Romo and others 1998). The monkeys were trained to perform this task as accurately as possible, typically distinguishing frequency differences as small as 8 Hz (e.g., 28 Hz vs. 36 Hz) correctly more than 75% of the time (Fig. 1). ICMS in S1 then was delivered (through microelectrodes positioned near quickly adapting neurons in area 3b) in place of the second mechanical vibration. The frequency of doublet pulses of electrical stimulation was in the same range as the frequency of the mechanical flutter/vibration applied to the skin. The monkeys could accurately indicate whether the second stimulus was higher or lower in frequency than the first stimulus when the second stimulus was either ICMS or a mechanical vibration of the fingertip. The monkeys thus discriminated the frequency of the ICMS as if it were the mechanical stimulus. Interestingly, when the ICMS was made aperiodic—delivering doublets at random intervals but with the same number of doublets during the 500 ms stimulus—the animals continued to discriminate the average frequency accurately.

Figure 1.

Biomimetic ICMS perceived as vibration. (A) Experimental paradigm. Within 1 second of when the mechanical probe indented a fingertip (probe down—PD), the monkey placed its other hand on a home key (key down—KD). After a delay, the mechanical probe oscillated at the base frequency, and after an interstimulus interval either a second mechanical stimulus (blue) or a train of ICMS (orange) was delivered at the comparison frequency. The monkey then released the home key (key up—KU) and pressed one of two push-buttons (PB) indicating that the comparison frequency was higher or lower than the base frequency. (B) Percent correct using mechanical (blue bars) versus ICMS (orange bars) comparison stimuli with 8 Hz frequency differences. Base frequencies (violet) are shown in the upper row of numbers, comparison frequencies (green) in the lower row. (C) Performance with a base frequency of 20 Hz as the comparison frequency was varied (blue circles—mechanical; orange circles—ICMS). Reproduced with permission from Romo and others (1998).

Additional recent studies extending such work to area 1 as well as area 3b have shown that the frequency of ICMS pulses in either area can be discriminated up to 200 Hz, and that increasing ICMS current (up to 100 μA) is perceived as being comparable to increasing the pressure and depth (up to 2000 μm) to which the skin is indented mechanically (Callier and others 2020; Tabot and others 2013). At some electrodes, however, high pulse frequencies can change the perceived magnitude of the stimulation. Furthermore, when ICMS is delivered in the somatotopic representation of different fingertips (identified by the local neural responses to natural tactile stimulation) monkeys can discriminate the somatic location of mechanical versus ICMS stimuli (Tabot and others 2013).

Importantly, in these studies of biomimetic S1 ICMS in nonhuman primates, the monkeys performed the discriminations as soon as ICMS trains were substituted for mechanical stimulation. The monkeys had no opportunity to learn to interpret the artificial stimuli. ICMS in S1 thus can be biomimetic for vibration frequency, pressure magnitude, and skin location. Beyond these passively perceived features, monkeys also have discriminated grating textures mimicked by delivering an ICMS pulse in S1 each time the fingertip of an actively exploring avatar hand (controlled by the monkey with either a joystick or a BCI) encountered a virtual ridge at spatial frequencies of 0.5 to 4.0/cm, which requires integrating the frequency of ICMS pulses with knowledge of the speed of hand motion (O’Doherty and others 2019). In this texture discrimination task the monkeys’ performance was above chance from the outset, but improved further with experience. Whether other features of tactile stimulation—such as temperature, or motion across the skin surface—can be produced biomimetically with artificial stimulation has yet to be determined.

In humans as well, ICMS in S1 can produce naturalistic sensations. A human subject was able to discriminate which finger region in S1 was being stimulated (Flesher and others 2016). The subject most often described the sensation produced by S1-ICMS in areas 1 and 2 as “pressure.” Increasing ICMS current from 10 up to 80 μA produced a linear increase in the perceived intensity of the pressure, consistent with the earlier finding in macaques (Tabot and others 2013). Yet when asked to rate the naturalness of the stimulus, the subject gave a rating only midway between totally natural and totally unnatural 95% of the time.

This “partially natural” quality may reflect a limitation of biomimetic stimulation. Single ICMS pulses synchronously excite a large number of neural elements distributed over a substantial local area, and trains of pulses can excite neural elements at a distance (see Box 1). Although local and long-range synchrony occur naturally among inhibitory interneurons (Buzsaki and Wang 2012; Swadlow and others 1998), widely distributed synchronous activation of large populations of excitatory neural elements, though pathophysiologic in seizure disorders (Chauvette and others 2016), rarely if ever occurs in the natural physiology of normal, healthy cortex. So percepts generated by ICMS, even biomimetic ICMS, may not be experienced as entirely natural.

Visual Cortex

In the visual cortex, electrical stimulation evokes the percept of a small spot of light, termed a “phosphene.” Phosphenes are evoked with stimulation at the cortical surface through electrocorticographic electrodes or with ICMS (Bosking and others 2017; Dobelle 2000; Tehovnik and Slocum 2013; Winawer and Parvizi 2016). In normally sighted human subjects, ICMS within 2 cm of the occipital pole (most likely in primary visual cortex, V1) produces phosphenes which subjects describe as small (1–2°), brightly colored (blue, yellow, or red) spots of light at locations in the visual field that depend on the location of the electrode in the retinotopic map (Bak and others 1990; Schmidt and others 1996). Theoretically, very low current stimulation could evoke phosphenes with additional features of cortical columns, such as line orientation (Tehovnik and others 2009; Troyk and others 2003). Phosphenes thus can have the biomimetic properties of color and retinotopic location, and potentially other features as well.

Phosphenes reveal a second challenge for biomimetic stimulation, however, having to do with what we might term “interaction” with normal neural activity. In normal vision, a complex network integrates shifting retinal images as the eyes move to foveate various stationary or moving objects, thereby providing a stable visual scene (Wurtz 2008). Human subjects report, however, that phosphenes appear to jump around in the earth-based reference frame when the subject makes saccadic eye movements, that is, phosphenes maintain their retinotopic location when the eyes move. Phosphenes induced by ICMS thus fail to engage the normal scene-stabilizing network. To provide a stable visual scene, therefore, biomimetic stimulation in the primary visual cortex may require additional stimulation that somehow engages the scene-stabilizing network. Furthermore, although one might think that simultaneous stimulation at multiple points selected to approximate a shape (e.g., a circle) would evoke the visual percept of the appropriate shape immediately, instead subjects initially perceive multiple separate phosphenes (Najarpour Foroushani and others 2018). Interestingly, interleaved ICMS pulses on nearby electrodes can produce the percept of a single fused shaped (Bak and others 1990), so it may be the well-known phenomenon of surround inhibition that creates separate phosphenes when pulses are delivered simultaneously (Tanaka and others 2019). Only after considerable experience have human subjects learned to recognize relatively simple shapes such as printed letters evoked by simultaneous artificial stimulation at multiple cortical points (Dobelle 2000).

Hippocampus

In the somatosensory and visual examples described above, stimulation was delivered through a given electrode to evoke a particular percept. But artificial stimulation also can be delivered at multiple locations in complex spatiotemporal patterns. Perhaps the most sophisticated application of biomimetic spatiotemoporal electrical stimulation has been applied in the hippocampus to enhance short-term memory in rodents and nonhuman primates (Deadwyler and others 2017). Multiple electrodes (tetrodes) were placed in both the CA3 input field of the hippocampus and in the corresponding region of the CA1 output field, which receives strong input from CA3 via the Shaffer collaterals. A nonlinear, dynamic, multiple-input/multiple-output (MIMO) model was trained to record the responses of neurons in CA3 and deliver ICMS pulses that mimicked the corresponding spatiotemporal pattern of spikes discharged by neurons at the CA1 electrodes. The monkeys had been trained to perform a delayed-match-to-sample task that incorporated varying numbers of distractors and required the monkeys to remember the sample for various time delays. Using the MIMO model to determine the spatiotemporal pattern of the pulses, ICMS then was delivered in CA1 based on the activity recorded in CA3 during presentation of the sample. Exactly how this biomimetic ICMS interacts with the ongoing natural activity of the hippocampus remains unclear, but delayed matching performance improved substantially: with multiple distractors and at any given delay interval, the monkeys were more likely to match the sample correctly. For example, with MIMO stimulation monkeys could remember at delays of 30 to 40 seconds as well as they could at delays of 10 to 20 seconds without stimulation. MIMO-model hippocampal stimulation recently has been shown to improve delayed recall in human subjects as well (Hampson and others 2018).

Interestingly, in the monkey studies the investigators noticed that within a given session the enhanced memory performance persisted for a considerable number of trials after delivery of MIMO-ICMS was discontinued (Deadwyler and others 2017). This persistent effect of MIMO-ICMS indicates a third limitation on biomimetic stimulation: plastic changes produced in the hippocampus or elsewhere in the nervous system outlasted the duration of the artificial stimulation. ICMS is known to produce long-term potentiation and long-term depression in the hippocampus and in the sensorimotor cortex (Castro-Alamancos and Connors 1996; Hess and Donoghue 1999; Iriki and others 1989; Kitagawa and others 1997; Nicoll 2017; Pinar and others 2017; Teskey and others 2007). Furthermore, when triggered to occur at short intervals near spontaneous spikes in motor cortex for 1 to 4 days, ICMS has been shown to be capable of inducing spike-timing dependent synaptic plasticity that can last for several days after the artificial stimulation is stopped (Bi and Poo 2001; Guggenmos and others 2013; Jackson and others 2006; Nishimura and others 2013; Rebesco and Miller 2011; Rebesco and others 2010). Additionally, training monkeys to detect ICMS in the primary visual cortex raises the threshold for detection of visual stimuli at the corresponding retinotopic location (Ni and Maunsell 2010). Through such mechanisms, biomimetic stimulation in any cortical area—including sensory areas that have previously reorganized after prolonged deafferentation (Flesher and others 2016; Pons and others 1991)—inadvertently may induce some degree of plastic reorganization in the nervous system, altering the perception evoked by the otherwise biomimetic stimulus.

Another Challenge for Biomimetic Stimulation in Any Cortical Area

Although we have brought out different challenges for biomimetic stimulation—achieving naturalness, integrating with normal function, and avoiding plastic changes—in relation to different cortical areas, all these issues are likely to arise with artificial stimulation anywhere in the cortex. Still another challenge for biomimetic stimulation regardless of the target cortical area will be to deliver spatiotemporally patterned stimulation at multiple sites in such a way that the subject experiences a complex percept (Overstreet and others 2016). When you use multiple fingertips to pick up an object, for example, you can instantly judge its size and shape, as well as whether it is hard or soft, rough or smooth, warm or cold, which allows you instantly to know whether you are holding a cup of hot tea or an ice cube. Such complex perceptions require integration of input from different types of cutaneous receptors in each fingertip, as well as proprioceptive inputs from the joints and muscles of the fingers. Although biomimetic ICMS may be able to create percepts of flutter frequency and pressure at different points on the skin, whether appropriate spatiotemporal patterns of ICMS at multiple electrodes can be integrated into a more complex percept of an object with specific size, shape, and texture has yet to be determined. To what extent will appreciation of such complex percepts be intuitive and immediate versus learned through practice over time?

Likewise, can stimulation in visual cortex be delivered such that subjects fuse multiple phosphenes into the image of a numeral or a face without considerable learning? The concept of delivering a pixelated image using biomimetic stimulation through an array of electrodes implanted in visual cortex might seem straightforward. But efforts to date have been primarily arbitrary, in that an electrode grid placed over both primary and higher order visual cortex typically does not stimulate the cortex in a well-ordered retinotopic mapping. Nevertheless, implanted human subjects have learned to use such visual neuroprostheses to discriminate a variety of visual shapes (Dobelle 2000). Of note, when the image is acquired with a head-mounted camera, the subject scans the shape using active head movements, that is, the subject does not recognize the shape using a single, stationary pattern of phosphenes. Rather the subject artificially “looks around,” creating a dynamically changing pattern of phosphenes, to recognize a visual shape.

Yet to be explored is the possibility of delivering biomimetic stimulation in complex spatiotemporal patterns in higher order sensory areas such as the secondary somatosensory area or the fusiform face area. Conceivably more complex percepts might be elicited by biomimetic stimulation in these areas. But we have yet to understand the patterns of normal neural activity that provide such complex percepts. Beyond processing peripheral input through sequentially hierarchical cortical areas, these percepts may involve multiple stages of cortico-thalamo-cortical processing (Sherman 2016) and may be modulated by both bottom-up and top-down attention (Katsuki and Constantinidis 2014). Whether mimicking activity in a primary sensory receiving area with ICMS will suffice to evoke a naturalistic complex percept, or whether mimicry of additional processing stages and modulatory influences will be needed as well, remains unknown.

Advances with Optogenetic Biomimetic Stimulation

To date, optogenetic approaches have provided behavioral examples primarily of detection of a single stimulus rather than discrimination of different stimuli. For example, optogenetic stimulation in S1 has been shown to be detected both in mice (Huber and others 2008) and in monkeys (May and others 2014). Recently, however, mice have discriminated optogenetic stimulation of visual cortex that biomimetically emulated the natural spatiotemporal patterns of neural activity occurring as they viewed vertical versus horizontal moving gratings (Marshel and others 2019). Achieving this result employed two major technical advances. First, a fast channelrhodopsin, ChRmine, was identified, enabling effective optogenetic stimulation in the millisecond time frame. Second, holographic laser stimulation was developed to stimulate multiple targeted neurons in different layers throughout the cortical depth asynchronously. Mice initially were trained in a Go/No-Go paradigm to lick a spout for a water reward when a vertical grating appeared, but not to lick and thereby avoid an air puff when a horizontal grating appeared. Population neuronal responses to vertical versus horizontal gratings then were recorded. Without the visual gratings to view, using only optogenetic playback of the recorded activity patterns, the mice then could perform the Go/No-Go task as if they were still seeing the gratings. In the future, such optogenetic approaches may go far in extending studies that currently employ biomimetic electrical stimulation.

Arbitrary Stimulation

Limitations on the extent to which stimulation can emulate the natural activity of the nervous system raise the question of whether stimulation used to inject information into the cortex must necessarily be biomimetic at all. Could arbitrary stimulation that makes no attempt to reproduce natural neural activity be useful?

To a certain extent, this question has been approached noninvasively by using one sensory modality to replace another, often referred to as “sensory substitution.” When a blind person reads Braille, for example, he or she uses natural tactile sensation to substitute for the vision used by sighted individuals. The various configurations of raised dots that form the Braille alphabet are designed to be recognized easily with touch, not to emulate the shape of visually perceived letters. Braille thus uses an arbitrary tactile representation of the “normal” visual alphabet. And in a deeper sense, all alphabets substitute another sensory modality for the way in which the human cortex naturally receives language input: listening to someone else speak.

The idea that information normally arriving in the nervous system through vision can be delivered instead through touch has been extended by stimulating the skin either mechanically through an array of vibrotactile devices or else electrically through an array of surface electrodes (Bach-y-Rita 2004). Using such arrays, a pixelated representation of the image from a video camera can be delivered to the skin. With such a tactile vision substitution system, human subjects can learn to identify the direction in which an object moves, and even to perceive faces, although perception at this level is probably far from our normal ability to recognize complex visual images immediately.

Arbitrary Stimulation in S1 to Substitute for Vision

Combining the idea that tactile sensation can substitute for vision with the concept that ICMS in S1 can evoke tactile percepts has led to studies in which S1 ICMS has been used to substitute for cues that might otherwise be delivered visually. Rats have been trained to turn either rightward or leftward when ICMS was delivered in the right or left S1 whisker cortex (Talwar and others 2002). Owl monkeys have learned to interpret different spatial or temporal patterns of ICMS in the S1 hand representation as cues to find food behind a right versus a left door (Fitzsimmons and others 2007), or to move a cursor to a right or left target using either hand control of a joystick or brain control of a cursor (O’Doherty and others 2009). And Rhesus monkeys actively exploring a workspace learned to hover a cursor over one of three visually identical targets when a particular rewarded pattern of S1 ICMS was delivered, rather than an alternate unrewarded pattern or no ICMS at all (O’Doherty and others 2011). In all of these studies, rather than delivering ICMS in biomimetic patterns that the subjects appreciated as natural touch, the subjects learned an arbitrary relationship between different sites or patterns of ICMS in S1 and the correct target.

A more complex scheme of arbitrary S1 ICMS has been used to provide Rhesus monkeys with continuous information on the direction and distance to unseen targets (Dadarlat and others 2015). Monkeys first learned to perform the task cued by a visual stimulus of moving dots. The direction and speed of the dots informed the monkey of the instantaneous direction to the target and the distance to the target, respectively. The monkeys then also received ICMS concurrently at 8 electrodes in Brodmann’s areas 1 and 2. ICMS pulses at each electrode were delivered faster to indicate greater distances to the target, and each electrode was assigned a different preferred direction without regard to the natural receptive field properties of local neurons, that is, arbitrarily. After considerable practice with both the visual and ICMS stimulation delivered concurrently, the monkeys were able to reach to the unseen targets using the S1-ICMS alone. The S1-ICMS was delivered continuously throughout each trial, and after extensive training both the initial movements and subsequent corrective submovements were made in the appropriate direction to the target. The monkeys thus learned to integrate the information delivered concurrently through eight electrodes and to use that information repeatedly (perhaps continuously) within single reaching trials. Furthermore, when the moving dots were not turned off, but their coherence was reduced (making the information they provided noisier), the monkeys could perform more accurately with both the visual and ICMS cues than with either one alone, indicating that they were able to integrate the “multisensory” natural visual and artificial, arbitrary S1-ICMS information.

Monkeys may be able to form such learned associations when ICMS is delivered in any of Brodmann’s subdivisions of S1. Discrimination of ICMS at different frequencies as instructions for different targets has been achieved in proprioceptive area 3a of a Rhesus monkey (London and others 2008). Rhesus monkeys can discriminate different ICMS frequencies accurately in area 3b (Romo and others 1998), though arbitrary learned associations of ICMS specifically in area 3b has not been reported. An owl monkey learned to detect ICMS in area 1 (O’Doherty and others 2009). Hence, targeting specific Brodmann subdivisions may not be important for purposes of arbitrary stimulation in S1.

Could ICMS in S1 ever come to be perceived as visual? This seems unlikely. Although a fraction of normal humans experience synesthesias—experiencing visual colors upon hearing particular musical passages, for example—most of us know clearly whether we are perceiving our coffee cup through touch, through vision, or through both at the same time. Those parts of the brain that receive output from S1 interpret that output as having arisen from somatosensory input, while those parts that receive output from V1 interpret that information as visual. Hence, it seems most likely that arbitrary ICMS in S1 will continue to evoke percepts that, while not entirely normal, are somatosensory in quality, while arbitrary ICMS in V1 will evoke visual percepts. Because of the difficulties in asking an animal whether a perception is somatosensory versus visual in quality, answering this question may await relevant studies in humans.

Beyond Primary Sensory Cortical Areas

Up to this point we have dealt with biomimetic or arbitrary stimulation delivered in sensory receiving areas of the cerebral cortex, a reasonable starting point given that all the information that enters the central nervous system (CNS) normally arrives through the senses. But all regions of the CNS are excitable. We therefore can ask whether information can be injected in cortical association areas beyond the sensory areas. Using surface stimulation of the human cerebral cortex exposed during surgical procedures, Penfield and his colleagues observed that stimulation in the human temporal lobe might evoke complex visual or auditory hallucinations or alterations of perceived size or distance (Penfield and Rasmussen 1950). But in large sectors of the frontal, parietal, and temporal association cortex, the patient seemed “oblivious” to stimulation unless the patient was engaged in a specific behavior involving that particular cortical region. If delivered while the patient is speaking, for example, electrical stimulation in language regions may evoke linguistic errors or frank “speech arrest” of which the patient is aware; but if not speaking, the patient may be unaware of electrical stimulation at the same location. Can artificial stimulation in such cortical association areas be used to inject information?

In an early study addressing this question, cats were trained to lift the right forepaw to avoid a shock if they detected electrical stimulation delivered between bipolar electrodes (2 to 4 mm apart) implanted on the surface of the cerebral cortex (Doty and others 1956). Cats could be trained successfully at most sites tested, which were largely in visual association areas (green dots in Fig. 2A). In a subsequent study, monkeys were trained initially that when a green light came on, they could press one lever to receive a food reward, whereas when a red light came on pressing a different lever prevented an aversive shock to the tail or leg (Doty 1965). The green light thus became a food-reward conditional stimulus (FRCS), whereas the red light became a shock-avoidance conditional stimulus (SACS). Electrical stimulation at one site in the cortex then was delivered simultaneously with the green light, and stimulation at another site was delivered simultaneously with the red light. After a number of such trials the green and red lights could remain off, leaving only the electrical stimulation at one site or the other as the FRCS or SACS. Monkeys still pressed the correct lever either to receive the food reward or to avoid the shock. Whereas the cats simply had detected the occurrence of electrical stimulation, the monkeys distinguished which of two cortical sites had been stimulated and responded differentially. Monkeys were trained successfully to respond to electrical stimulation at a wide variety of locations throughout the cerebral cortex, illustrated in Figure 2B. Many of these sites lay in primary or secondary visual, somatosensory, or auditory cortical areas, and the ability of the monkeys to detect and respond to stimulation at those sites might have been expected. But monkeys also were able to respond differentially to electrical stimulation at sites including premotor (sites 5, 6, 19, and 33) and prefrontal (sites 4, 20, 28) cortex, areas not involved directly in processing sensory input.

Figure 2.

Cortical locations used for arbitrary conditioning with electrical stimulation. (A) Cat. Overhead view of the feline cerebral cortex summarizing sites from 59 cats at which electrical stimulation was used as the conditional stimulus. Filled green circles indicate sites at which conditioning stimulation was successful in <400 trials; open orange circles, successful after >400 trials; and open red triangles, unsuccessful. (B) Macaque. Lateral view of the left hemisphere summarizing sites in either hemisphere from eight monkeys at which electrical stimulation at two separate sites was used for differential conditioning. Filled green circles indicate that the electrode was at or near the cortical surface; violet xs indicate that the electrode was >3 mm below the surface. Orientation crosshairs indicate: a—anterior, p—posterior, r—right, l—left, d—dorsal, v—ventral. Reproduced with permission from Doty and others (1956) (A) and Doty (1965) (B).

The locations used for the FRCS and SACS most often were separated widely (e.g., premotor vs. parietal), but in some cases monkeys distinguished stimulation between different pairs of four electrodes in a 3 mm square array in primary visual cortex (Doty 1965). Can stimulation at different electrodes in the same association area be distinguished? In a recent study, two Rhesus monkeys were trained successfully to use ICMS at different electrodes in the premotor cortex as instructions about which of four objects to reach to, grasp, and manipulate (Fig. 3; Mazurek and Schieber 2017). Prior to the introduction of ICMS, these monkeys were trained to operate the four different objects cued by four different rings of blue LEDs. Then ICMS at different electrodes was paired with each of the blue LED rings for several sessions, after which the intensity of the LEDs was gradually reduced causing the monkey’s success rates to decline and reaction times to increase. But with further reductions in LED brightness, success rates increased and reaction times decreased back to their baselines. Ultimately the LEDs could be turned off entirely and the monkeys still performed the reach-grasp-manipulate task close to 100% correct. The monkeys then could also perform well when the blue LED instructions and ICMS instructions were interleaved on different trials.

Figure 3.

Differentiating arbitrary ICMS instructions in premotor cortex. (A) Monkeys initially were trained to perform a reach, grasp, manipulate task in which blue LEDs instructed the target object. (B) ICMS in either the primary somatosensory (S1) or premotor cortex (PM) then was used to deliver instructions for performing the same task. (C) Monkey L first was trained to use S1-ICMS and subsequently was trained to use PM-ICMS. (D) Monkey X first was trained to use PM- and subsequently S1-ICMS. In each case, the monkey first performed the task using LED instructions. Beginning at session 0 (solid vertical line), ICMS was delivered through different electrodes concurrently with illumination of each of the four blue LED rings. Then, starting in the session indicated by the dotted line, the blue LEDs were dimmed progressively. Eventually the LEDs were turned off completely (dashed line), after which success rates and reaction times returned gradually to their prior baseline levels seen initially when the monkey used only LED instructions. Both monkeys thus successfully learned to use ICMS—in either S1 or PM—as instructions for performing the task. Reproduced with permission from Mazurek and Schieber (2017).

One might assume that ICMS at different sites in premotor cortex predisposed the monkeys to make different movements. But the currents used were subthreshold for evoking any muscle activity, as demonstrated by stimulus-triggered averaging of electromyographic activity. Moreover, when the assignment of the four different electrodes to the four different movements was shuffled randomly, both monkeys relearned the new assignments. These observations indicate that the monkeys differentiated four different qualia—qualities experienced, distinct from their physical source—upon stimulation at the four different electrodes, and learned to associate each of the four qualia with a different movement.

Exactly what these monkeys experienced when ICMS was delivered at different electrodes in premotor cortex remains unknown. Premotor cortex receives both processed somatosensory and visual inputs (Graziano and others 1997; Rizzolatti and others 1981a, 1981b) and contains many mirror neurons that are active when the subject observes another individual making movements of the arm and hand (di Pellegrino and others 1992; Gallese and others 1996). ICMS in the premotor cortex therefore might produce some somatosensory and/or visual qualia, or some internal representation of particular movements.

These considerations reopen the question of the extent to which subjects experience stimulation in otherwise “silent” areas of association cortex (Mazurek and Schieber 2019). Given that neurons anywhere in the CNS can be stimulated, do subjects experience some qualia no matter where stimulation is delivered? Monkeys can detect ICMS in the frontal eye field at currents subthreshold for evoking eye movement (Murphey and Maunsell 2008), but surface stimulation of the human fusiform face area often goes undetected (Murphey and others 2009). Neither an owl monkey nor a Rhesus monkey was able to learn to use ICMS in the posterior parietal cortex as an instruction about where to reach (Fitzsimmons and others 2007; O’Doherty and others 2009). Defining exactly which cortical areas can be used to inject information and which cannot will require further studies.

For such investigations, the nonverbal motor responses of trained nonhuman primates may prove as useful as the verbal responses of humans. Even in humans, not all experiences are necessarily available to the systems providing expression through language. Well known examples include the phenomenon of “blindsight,” in which a cortically blind subject can walk down a hallway avoiding obstacles while being unable to describe verbally the presence, location, or nature of the objects (de Gelder and others 2008). Similarly, subjects with visual agnosia may scale the size of their grasp aperture accurately as they reach for objects of different size without being able to verbally describe the different sizes of the objects (Goodale and Milner 1992; Goodale and others 1994). Future studies thus may show that humans can learn to use artificial stimulation at different sites without any experience they can verbalize. The ability of ICMS to induce spike-timing dependent plasticity (Guggenmos and others 2013; Jackson and others 2006; Rebesco and others 2010) may contribute to learning arbitrary associations without qualia that can be described verbally by the subject (Lebedev and Ossadtchi 2018).

Challenges for Arbitrary Stimulation

To a certain extent, arbitrary stimulation will face the same challenges as biomimetic stimulation: (1) unnaturally synchronous discharge is evoked in large numbers of neurons; (2) evoked qualia need to be integrated with normal neural activity; (3) the stimulation per se is likely to result in plastic changes in the stimulated circuits; and (4) complex spatiotemporal patterns of stimulation may be needed to provide complex information. The chief difference from biomimetic stimulation is that arbitrary stimulation outsources the solutions of these challenges to the native plasticity and learning capability of the CNS. Any plastic changes induced by artificial stimulation are unlikely to involve substantial reorganization of existing cortical representations, not only because the scale of cortical reorganization may be limited (Makin and Bensmaia 2017), but also because by definition arbitrary stimulation imposes no systematic spatial organization—for exampld, somatotopic, retinotopic, or tonotopic—on the information being delivered.

Coming up with internal solutions to these four challenges puts a larger burden (sometimes referred to as “cognitive load”) on each individual subject, requiring substantial time and effort before the artificial stimulation can be incorporated proficiently into the desired behaviors. The internal solutions may vary from subject to subject and therefore show varying degrees of success. And the rate at which information can be input arbitrarily—in terms of number of channels and bits/second in each channel—is likely to be substantially below the standard of naturally evolved sensory perception or the capacity of fully developed biomimetic stimulation. Nevertheless, in the near term, arbitrary stimulation may be more useful than biomimetic stimulation for circumstances in which our understanding of exactly what needs to be mimicked is limited.

Ethical Considerations Going Forward

Consideration of ethical issues will become increasingly important as the technology for artificially injecting various kinds of information into the nervous system improves and enters the mainstream. Injecting information raises ethical concerns somewhat different from those raised by neuroprosthetic communication devices, vision neuroprostheses, or neuroprosthetic limbs with somatosensory feedback (Bockbrader and others 2018; Loeb 2011). Like such neuroprostheses, artificial stimulation has the potential to help numerous individuals by restoring or even enhancing function. More directly than such neuroprostheses, however, injecting information—particularly arbitrary information—could provide the potential for manipulating the nervous system (Pugh and others 2018), intensifying in particular questions of the individual’s autonomy and personhood (Sample and others 2019; Yuste and others 2017).

On a daily basis, the normal nervous system takes in natural sensory and cognitive information that not only affects the individual’s choices and actions, but also is stored to extend knowledge and develop skill. This includes not only information the individual wants, needs, and intends to get but also information with which others seek to influence the individual’s actions, thoughts, and choices (e.g., targeted advertisements, social media campaigns). Artificial stimulation eventually might provide similar capabilities both to learn from and to be influenced by the incoming information stream. Major ethical issues then will have to do with maintaining the individual’s ability to differentiate information the individual intends to receive from information with which others seek to influence the individual, as well as the ability to distinguish one’s own internal thoughts from ideas introduced from outside.

Promise

Still, the ability to inject information into the cerebral cortex, and into other parts of the CNS as well, promises many opportunities for advancing both science and technology. As suggested by Doty and colleagues in the opening quotation, the ability to bypass sensory systems and inject information directly into the cerebral cortex provides new opportunities to investigate the functional capabilities of the brain. One might ask, for example, whether activity in the primary motor cortex (M1) involved in generating a particular movement differs depending on whether the instruction arrives through natural visual pathways versus through arbitrary stimulation in the premotor cortex. No difference would indicate that M1 activity generates the same movement in either case, but substantial differences in M1 activity for the same movement depending on the source of the instruction would indicate a more cognitive, context-dependent role for M1 in interpreting the inputs that lead to voluntary movement.

Injection of information into the cerebral cortex also promises to be a key functional component of neuroprosthetic devices. Visual neuroprostheses already have been developed for blind individuals, although such devices are not yet practical (Dobelle 2000; Lewis and others 2015). The recent development of robotic upper extremities controlled by neural recordings from the cortex of human participants has brought out the need for somatosensory feedback from the hand (Bensmaia and Miller 2014). Although the prosthetic arm can be moved under visual guidance, only crude grasping and manipulation of an object can be accomplished without somatosensory feedback (Wodlinger and others 2015). Current efforts to provide the requisite feedback directly into the CNS therefore focus on biomimetic stimulation in S1 (Flesher and others 2016; Hiremath and others 2017; Hughes and others 2020). Ultimately, a “bidirectional” neuroprosthetic upper extremity would both be moved under output decoded from neural activity and return feedback into the nervous system through artificial stimulation.

And beyond restoring sensory information, where biomimetic stimulation has advantages, we can envision cortico-cortical neuroprosthetic devices employing arbitrary stimulation (Averna and others 2020; Jackson and others 2006; Lebedev and Nicolelis 2017). Such a device would record neural activity from an electrode array in one cortical region, decode relevant information from the recorded activity with an on-board computer chip, and then deliver that information to a different cortical region via artificial stimulation (see Box 2). Cortico-cortical neuroprostheses could bypass injured or disordered regions of cortex and thereby contribute to restoring a wide variety of higher cortical functions.

Box 2. Cortico-cortical Neuroprostheses to Restore Higher Cortical Functions.

Work employing artificial stimulation to inject information into the nervous system has focused primarily on restoring sensory inputs, including somatosensory feedback to help control a prosthetic limb, visual prostheses, and the cochlear implant. However, technological advances in neural recording and stimulating arrays, progressive miniaturization of computer chips, and increasingly sophisticated computational neural models have provided the building blocks for realizing a rehabilitative system that could restore information along CNS pathways damaged by a focal neurologic injury. Cortical areas communicate information with one another to accomplish particular functions. For example (A), neural information (curved arrows) from visual cortex (V) about object shape and location normally is processed in the posterior parietal cortex (PPC) and communicated to the premotor cortex (PM) to plan reaching and grasping controlled by the primary motor cortex (M1).

A focal neurologic injury along this pathway (B, red X) would disrupt communication between V and PM, and impair reaching and grasping. Neural recordings made in visual cortex (violet electrode grid) then could be decoded by an implanted computer chip (square), and the necessary information could be injected directly into PM (yellow lightning bolt). Such a cortico-cortical prosthesis (lower right-angled arrow) would effectively bypass the lesion and restore neural function that the patient could learn to use. Similar prostheses could be applied to any number of CNS pathways where connectivity might be disrupted by different injuries, restoring function for a large number of patients in whom such a rehabilitative solution might improve quality of life.