Comparing temporal aspects of visual, tactile, and microstimulation feedback for motor control

Abstract

Objectives

Current brain-computer interfaces (BCIs) rely on visual feedback, requiring sustained visual attention to use the device. Improvements to BCIs may stem from the development of an effective way to provide quick feedback independent of vision. Tactile stimuli, either delivered on the skin surface, or directly to the brain via microstimulation in somatosensory cortex, could serve that purpose. We examined the effectiveness of vibrotactile stimuli and microstimulation as a means of non-visual feedback by using a fundamental element of feedback: the ability to react to a stimulus while already in motion.

Approach

Human and monkey subjects performed a center-out reach task which was, on occasion, interrupted with a stimulus cue that instructed a change in reach target.

Main results

Subjects generally responded faster to tactile cues than to visual cues. However, when we delivered cues via microstimuation in a monkey, its average response was slower than that for both tactile and visual cues.

Significance

Tactile and microstimulation feedback can be used to rapidly adjust movements mid-flight. The relatively slow speed of microstimulation raises some interesting questions and warrants further investigation. Overall, these results highlight the importance of considering temporal aspects of feedback when designing alternative forms of feedback for brain-computer interfaces.

1. Introduction

Cortically controlled brain-computer interfaces (BCIs) were first demonstrated over ten years ago in humans and monkeys [1, 2, 3, 4]. BCI control has been demonstrated in sophisticated behaviors such as point-and-click cursor control [5, 6, 7, 8, 9, 10, 11, 12, 13], use of a robotic arm [14, 15, 16, 17, 18], and the voluntary control of paralyzed muscles [19, 20, 21]. Recently, striking improvements have been made in the performance of visual-feedback BCIs based on engineering advances [8, 22], and such improvements will surely continue. However, as we strive to achieve BCI performance that is comparable to natural movement, it may also prove profitable to incorporate nonvisual feedback [23].

Human subjects who have lost proprioception and touch can still move, they but must rely solely on vision for feedback. Their movements are slow and inaccurate, requiring a great deal of concentration to perform [24, 25]. In able-bodied individuals, certain actions, like grip force when picking up an object, are controlled largely under non-visual feedback modalities. Decoding ability can be improved by accounting for sole reliance on visual feedback [22], however, tactile and proprioceptive feedback will most likely be essential in next-generation BCIs that will perform dexterous grasping and object manipulation tasks [26, 27].

Numerous studies have shown that tactile stimuli can improve performance in a wide range of tasks [28, 29], including driving [30], stimulus detection reaction time paradigms [31, 32], and target indication in BCI paradigms [33]. We extend the use of tactile stimuli to a new context: as a feedback modality to alter a reach in progress. We quantified the effectiveness of a tactile stimulus as a feedback modality by comparing it to vision.

Microstimulation in primary somatosensory cortex () can be behaviorally indistinguishable from response to tactile stimuli [34, 35]. While cortical microstimulation is only beginning to be applied in a BCI context [36], microstimulation in sensory cortex has a long history as a means of providing sensory information [26, 37, 38, 36, 39]. Microstimulation serves to evoke sensory effects that mimic the functional contribution of the stimulated area [40, 34, 35, 41, 42, 43, 44, 45, 46, 47]. Microstimulation in humans has been used to induce visual [48, 49], and tactile perceptions [50, 51, 52]. These demonstrations of the efficacy of microstimulation for sensory substitution led us to predict that the reaction time to microstimulation would be similar to that of tactile stimuli, and perhaps even faster because microstimulation allows us to bypass afferent pathways.

Previous studies utilizing microstimulation during hand control or BCI control have applied it when the subject was at rest, either prior to movement onset [39, 34, 35, 53], or while the subject lingered over a target area [36, 38]. The emphasis in those studies was on elucidating the discrimination thresholds between different stimuli and in demonstrating the robustness of microstimulation as an information channel. Notably, subjects were not required to utilize the stimulus during a movement to change the action, which is the very essence of feedback. In order for microstimulation to be used successfully as a sensory feedback channel, it must be delivered and interpreted quickly enough to be able to effect an ongoing movement.

Here we compare the temporal aspects of three different sensory modalities as sources of feedback about movement. We defined the elemental form of feedback as the ability of a subject to change an ongoing reach when confronted with new sensory information. To this end, we used a modified center-out reach task in which the subject’s ongoing reach was occasionally interrupted by a stimulus cue that instructed an immediate change in reach goal. In both humans and monkeys, the response to tactile stimuli was faster than to visual stimuli. Surprisingly we found that the behavioral response to microstimulation in was 25 percent slower than for visual stimuli. These results set a realistic basis of expectation for these three feedback modalities in behavioral and BCI applications.

2. Materials and Methods

2.1. Basic Experimental Procedures

Human subjects sat ~1 m in front of a computer monitor where they could make unrestricted arm movements. Although the reaching hand was visible, it controlled a cursor on the computer monitor, and only the monitor provided feedback on the hand position in relation to the targets. Monkeys were seated in a primate chair with the head fixed using a modified halo system [54] and the non-reaching arm restrained. Monkeys sat in front of a mirror and performed the task in a virtual reality environment with the hand moving freely unseen in the space behind the mirror. Hand position was represented by an onscreen cursor. Both humans and monkeys wore active motion capture markers (Phasespace, San Leandro, CA) on the index finger to record movements at 120Hz. Unrestricted hand movements occurred in a 1 cubic meter empty workspace with the virtual workspace centered on the reaching hand’s resting position and reach targets adjusted to span the subject’s side-to-side reaching range. Human reach distances from the center to the peripheral target were 25cm; monkeys 1 and 2 reached 12.5cm and 15cm respectively.

2.1.1. Human Subjects

Ten subjects (6 women, 4 men, ages 20–32 years) participated in the experiment. Participants gave informed consent before the start of the experiment and all studies were approved by the University of Pittsburgh Institutional Review Board. Subjects trained until proficient (>70% success rate for redirect trials) over 1–2 short training sessions. Customized timing parameters were calculated for the subject to efficiently sample the entire reach span during the redirect task. The data used for analysis were collected during a final session of 1000 trials. To ensure that there were not ongoing learning effects that improved response times, two subjects participated in additional long sessions. Their response times improved between the training session and first long session then remained consistent in subsequent long sessions. Thus we only analyzed only the first long session for all subjects.

2.1.2. Non-Human Primate Subjects

Two adult male Rhesus monkeys, 1 and 2 (Macaca mulatta, 8.4 and 10.4kg) participated in this study. All animal procedures were approved by the University of Pittsburgh's Institutional Animal Care and Use Committee, in accordance with the guidelines of the US Department of Agriculture, International Association for the Assessment and Accreditation of Laboratory Animal Care, and the National Institutes of Health.

2.2. Behavioral Tasks

Our key behavioral task was a redirect task in which a stimulus presented during an ongoing movement signaled a change in the goal location. The redirect task was randomly interleaved with a classic center-out reaching task, here called the direct reach task (supplemental video 1). Our main quantification was the response time to the redirect stimulus, which indicates how rapidly a subject can respond to a stimulus as a source of feedback about an ongoing movement. We also employed a pressured reaction time task in one monkey to assess how quickly he could respond to a particular stimulus.

2.2.1. Direct Reach Task

Seventy percent of the trials were of a control task, performing a prompt reach to one of 2 possible opposing targets (figure 1(a)). Each trial began when the subject moved their hand to position a computer cursor into a central 4cm square target (green square). Two peripheral grey squares then appeared indicating potential reach targets (figure 1(b)). One of the targets flashed yellow for 500ms to indicate the intended reach target. After a delay of 100–500ms, the start target disappeared as the signal to reach. We required movement to begin within a window of 80–400ms after the go cue to penalize false starts or lingering on the start target. The subject was required to reach promptly and directly to the intended target within 800ms, or else the trial failed. Human subjects were instructed to ”reach normally as if to pick up a glass full of water”; unusually fast reaches lack the control needed to adjust trajectory mid-movement. After acquiring and holding still at the target for 300ms, the subjects were rewarded with an audible tone (humans) and a juice reward (monkeys). Failed trials were punished with a 2s intertrial interval (humans and monkeys) and occasional verbal feedback from the experimenter (humans only).

Figure 1.

Experimental paradigm. (A) Task configuration showing center start position (green), potential reach target (gray), and instructed reach target for that trial (yellow). The subject moved the hand to position a cursor on the target. Mid-reach redirect cues were given via vibrating motor, a blue circle, or microstimulation of. (B) Task Timeline. Sample horizontal hand trajectory across time during a direct reach task (black dashed line) and a redirect task (solid blue line) with the go cue represented at time 0. Horizontal bands represent targets displayed during the redirect task. Target presentation during the direct reach task was identical up to onset of the redirect cue (blue). Potential reach targets (gray) appear after the start target is acquired and remain visible. Position of the hand is marked at the redirect cue onset (white circle) and moment of reach inflection (red circle). (C) Horizontal hand velocity of the direct reach and redirect tasks. Response time to the redirect cue was measured as the time elapsed between redirect cue onset (white circle) and moment of reach inflection (red circle).

2.2.2. Redirect Task

The redirect task was randomly interleaved with the direct task, occurring in 30% or less of the trials. Redirect tasks were identical to the direct reach task except that at a random time during the movement, a cue indicated that the other target had become the reach goal. The cue was visual, tactile, or direct microstimulation of the somatosensory cortex. The visual redirect cue was a large circle appearing in the center of the screen (represented in blue in figure 1(a)). The cue subtended 7–10° of visual angle. Tactile redirect cues were delivered via a vibrating motor placed on the non-reaching hand (human), forearm (monkey 2), or back (monkey 1). The microstimulation redirect cue was a pulse train delivered through an electrode in S1. The duration of all redirect cues was 400ms (figure 1(b)). The cue modality was randomly interleaved during each session, with 1, 2, or 3 modalities used in a given session. In sessions using multiple modalities, the cue modalities were randomly interleaved, each with the same chance of appearance, resulting in about the same number of occurrences within the session. After onset of the redirect cue, the subject was required to reverse reach direction and touch the opposite target to earn a reward (figure 1(b)). If he or she touched the original reach target before changing direction, the trial would fail immediately.

The response time was quantified as the time between the onset of the redirect cue and the time when the horizontal component of the hand velocity crossed zero (figure 1(c)). We deemed the moment the hand reverses trajectory to be the definitive point in the action where it was clear that the redirect stimulus had been effective in allowing the subject to successfully avoid the original target. Note that the response time is not a classicallydefined “reaction time”, which would have been harder to measure in this task, but would be strictly shorter. Another advantage of this quantification is that it avoids any confusion between deceleration due to the redirect stimulus and due to normal deceleration of the reach to the original target.

We measured the onset time of the redirect cues with a photo-transistor circuit placed on the monitors, an accelerometer fixed to the vibrating motor, and by recording the output of the microstimulator. Lags between the time when the computer commanded the redirect stimulus and when it actually occurred were 77ms ±14ms (visual stimuli), 11ms ±3ms (tactile), and <1ms (microstimulation).

2.2.3. Pressured Reaction Time Task

In monkeys, to elicit the fastest reaction time to a stimulus modality without interference from conflicting motor plans or cognitive demands, we used a delayed center-out task with the stimulus as the go cue. In this “pressured reaction time task”, one of eight possible targets appeared, and then after a delay period 300–2500ms (uniformly distributed), a go cue appeared. The go cue could be microstimulation, vibrotactile stimulation, or a visual stimulus. In visually cued trials, the start target disappeared. In tactilely cued trials, a small vibrating motor placed on the upper arm turned on until movement began. In microstimulation trials, a 400ms pulse train was delivered through an electrode in S1 to indicate when to reach. Ten percent of the trials were “catch trials” in which no go cue appeared, and subjects were rewarded for continuing to hold the start target. Like the direct reach task, the subject’s behavior was tightly constrained to penalize early starts and late responses. Early starts were defined as a reach that began prior to 80ms after the go cue was delivered. At the start of the experiment, the subjects were allowed up to 1500ms within which they could initiate the reach. We decreased the duration of this temporal window over the course of the session. Reaction times became faster and less variable as the temporal window decreased. Eventually within the session, the window became so short that the subject could not respond fast enough to succeed in the task. Reaction time was measured as the time from stimulus onset to movement onset (i.e. the first detectable acceleration in hand velocity toward the target). To remove lucky guesses and leisurely responses, the top and bottom 5% of reaction times were removed when calculating the mean.

Finally, for various training purposes, we employed the pressured reaction time task without tight time constraints. This is referred to as the stimulus detection task.

2.3. Behavioral Timing and Controls

We used a combination of approaches to ensure that our human and monkey subjects made brisk, smooth, and goal-directed reaches. We adjusted task parameters to reduce the prospect of strategies like anticipatory “lucky guessing”, hesitations and lingering while awaiting a redirect cue, and early termination of trials. To this end, we distributed the timing of redirect cues so they occurred throughout the reach span. We titrated the trial type distribution, the punish time-out, and the reward volume (for monkeys) to promote smooth performance of the direct reach task. Figure 1(c) illustrates example velocity profiles. For the redirect task, we noted that the response time was not dependent on the position of the hand along the trajectory.

2.4. Tactile Stimulation

Tactile stimuli were delivered via a Jameco Reliapro 1.3V 8500RPM vibrating motor. This device consists of an off-axis asymmetric mass fixed in a plastic capsule. Adjusting the speed of the motor rotation controlled stimulus intensity. For human subjects, the motor was positioned on the back of the non-reaching hand. For monkey subjects the motor was placed on the back (monkey 1) or the forearm of the non-reaching hand (monkey 2), to coincide approximately with the receptive field location for the area recorded in. The stimulus was set to an intensity sufficient to be readily detected (defined as >95% correct in the stimulus detection task section 2.2.3), but without inducing an overt startle response.

2.5. Intracortical Microstimulation

Monkey 2 was implanted with an acrylic-free titanium recording chamber [55] placed over intact skull above the somatosensory cortex. Burr holes (~1mm) were drilled over the arm area of S1, guided by MRI images, until we found one where the tactile receptive fields for multi-neuronal activity were on the arm (from the bicep to the elbow of the non-reaching arm), and remained stable during the course of each session. Neural activity was amplified and played through a loudspeaker; multiunit responses were driven best by brisk brushing of the hair, rather than by constant pressure applied to the receptive field. During most sessions, the depth where neurons were first encountered was noted, then the electrode was advanced an additional 1 to 2 mm during a stimulus detection task until microstimulation was consistently detected. We infer that the 70 penetrations done over the course of our experiments were made into the gyral portion of (area 1) and not the sulcus wall of (area 3b), as confirmed by the fact that audible neural activity tapered off when the electrode was advanced by 5 mm or less from where neurons were first encountered, and an absence of noticeable change in type or location of receptive field. Neurophysiological recording and intracortical microstimulation (ICMS) was accomplished with 1MΩ tungsten electrodes (FHC Inc., Bowdoin, Maine) positioned and manipulated with a Narishige single electrode drive (Narishige Co., LTD, East Meadow, New York).

A computer-controlled pulse generator (A&M Systems, Carlsborg, WA) produced symmetric, biphasic, chargebalanced pulse trains with the cathode phase leading [56, 57]. Early in experiments, we designed microstimulation parameters using a range of amplitudes (30–90µA) with a fixed duration (500ms) and a range of durations (50–500ms) with fixed amplitude (90µA) (black points in figure 2) [58]. We used a stimulus detection task (section 2.2.3) in which the go cue was microstimulation. Each amplitude and duration combination was tested 20–100 times. A psychometric curve was fitted to the success rates as a function of amplitudes with duration fixed, then averaged across the fitted curve for varying durations and fixed amplitude (figure 2A). The same method was done for mean unpressured reaction times in figure 2B. This method illustrates the location of peak performance plateaus for amplitude and duration, allowing for stimulation parameters that are easy to detect.

Figure 2.

Microstimulation parameters. (A) Detection probability as a function of stimulus amplitude and duration. (B) Reaction time as a function of stimulus amplitude and duration. Black dots represent tested stimulus parameters. The grayscale gradient represents an averaging of the fitted curves across varying amplitudes and durations. White dots represent the parameters of the stimulus chosen for the microstimulation redirect cue. Stimulus parameters were chosen to be clearly within the behavioral performance plateau (past the sharp change from black to white of the left edge) but without using an excessively large amplitude or duration. In both plots, whiter shading corresponds to better performance.

We ascertained that 80µA was well above the threshold for detection (white point in figure 2(a)), and thus we used that current for all following experiments. Stimulus duration was evaluated in a similar fashion while the amplitude was set at 90µA (figure 2). For all subsequent experiments, duration was set at 400ms, well within the range of fast responses with high levels of detection (white point in figure 2(b)). Stimulus frequency was set at 200 Hz [36]. Pulse widths were 200 µs per phase with 25 µs between phases [36].

We defined threshold as the point at which the stimulation was detected on 90% of trials (figure 2(a)). This detection paradigm was briefly recapitulated at the beginning of each session. Before and after each training session, stimulus detection was evaluated to ensure peak performance was consistent with that of previous days. Microstimulation was paired with tactile stimulation for several sessions when it was first introduced to the animal and briefly during stimulus detection trials at the beginning of subsequent sessions as a training aid.

We opted to keep the microstimulation site as consistent as possible from one day to the next. Our experiment goal was to measure the fastest reaction times possible under different stimulus modalities. Since monkeys might not generalize as well as humans do, if the stimuli differed from one session to the next, we would anticipate a learning effect as animals adjusted to the new stimulus. We wanted to match stimulus conditions for all modalities over days. Thus we kept visual stimulus conditions the same, placed the vibrotactile motors in the same location, and inserted the electrode into the same burr hole each day.

3. Results

3.1. Human Study

Our main experimental manipulation was a redirect task, in which a cue instructed the subject to alter a reach in mid-flight. The redirect cue could be tactile or visual. We compared the response times (see methods) to stimuli of different modalities. The response time for tactile stimuli was significantly faster than for visual stimuli when pooled across the cohort of ten subjects (Paired t-test p<0.01) (figure 3). The average difference between tactile and visual response times across all subjects was 12ms, with the greatest difference being 23ms, and the smallest being −5ms (that is, vision faster than tactile). When tested individually, half the subjects’ response times to tactile cues were significantly shorter than for visual cues (Controlling for multiple comparisons using the Benjamini–Hochberg procedure with a false discovery rate of 0.05). For the other half, there was a trend toward a faster response to tactile stimuli that did not reach significance, or there was no difference between the two stimulus types (figure 3). We found no subjects for whom the response to a visual cue was significantly faster than the response to a tactile cue.

Figure 3.

Response time depends on stimulus modality. Human and monkey response times (mean +/− standard error) to visual cues (solid blue circle), tactile cues (hollow red square), and for monkey 2, microstimulation cues (solid black triangle). *: p<0.05; **: p<0.01 pairwise comparison for all modalities.

3.2. Monkey Study

3.2.1. Visual and Tactile Redirect Task

The redirect task with visual and tactile cues was conducted with two monkey subjects. Both animals were highly trained on the tasks. They produced movements with a bell-shaped velocity profile [59] (dashed trace in figure 1(c)). Pooling data across multiple sessions (Monkey 1=24, Monkey 2=31), the mean response time for tactile cues was significantly faster than for visual cues (188ms vs 217ms for monkey 1; 190ms vs 211ms for monkey 2, p < 0.001, t-test) (figure 4). Tactile response times were also significantly faster in every individual session (supplementary figure 1). This difference in response times for the two modalities (21 and 29ms) is approximately as large as the largest difference in response times observed among the human subjects (23ms).

Figure 4.

Monkey response time distribution. (A) Monkey 1 response time histogram for visual (blue) and tactile (red) redirect cues. (B) Monkey 2 response time histogram for visual (blue), tactile (red), and microstimulation (gray) redirect cues. Dot and line indicate mean +/− standard deviation.

The vibrating motors were audible. The animals could have just used the sound of the motors as the redirect cue. To control for this, for one monkey (2), we placed the motors nearby but not touching him. After a brief training block, the monkey responded to the sound of the motor, but the mean response time was 230 +/− 20ms, which was slower than the response to both the tactile and the visual redirect cues. We conclude that it was the touch, not the sound, of the motors to which the animals responded.

3.2.2. Microstimulation Redirect Task

We found that tactile cues can be used to influence behavior more rapidly than visual cues can. Could response times be even faster if a signal were provided directly to the brain, thus bypassing afferent pathways?

We tested this hypothesis by using microstimulation in S1 as the redirect cue. After becoming proficient at the redirect task, monkey 2 was implanted with a recording chamber and microstimulation was delivered via a single electrode. We stimulated within area 1 of the somatosensory cortex () in the vicinity of quickly adapting neurons. When microstimulation was first introduced, it was trained in combination with the tactile cue using the stimulus detection task and then the redirect task. The animal began to make the association within the first day and became proficient at using only microstimulation after two weeks of daily training. Across 23 sessions, response times for microstimulation cues were significantly slower than for tactile redirect cues (263 vs 190ms, p <0.001) and even visual redirect cues (263 vs 211ms, p <0.001) (figure 4(b)).

We considered whether extended experience with microstimulation might improve its response time. There was a decrease in response time over the first two weeks while the animal learned the task, but after becoming proficient (>80% correct for early redirect cues) no further improvements were noted. The differences in response time between modalities showed the same ordering and similar magnitudes across sessions. They were significantly different from each other every day (supplemental figure 1).

3.2.3. Critical feedback point

If feedback is too slow, it cannot influence behavior. Another way to express the utility of a feedback modality is to identify a spatial critical feedback point beyond which feedback would arrive too late to alter a movement. For better (i.e. faster) feedback modalities, this critical feedback point occurs later in the movement. Success at responding to the redirect cue decreases to zero the closer the subject’s hand gets to the target. Figure 5 shows performance as a function of the hand position at the time the redirect cue is given for monkey 2. We define the critical feedback point as the position along the reach trajectory beyond which the animal has a greater than 50% chance of successfully responding to the redirect cue. The critical feedback point for tactile and visual cues is at 37 and 38.5mm from the target, while for microstimulation, the critical feedback point is at 60mm from the target. Thus tactile and visual cues provide actionable information until much later in the movement than could microstimulation cues.

Figure 5. Likelihood of success based on hand position at the time the redirect cue is given.

Performance (percent correct) in the redirect task (3 mm bins). Curves are logistic fits for visual (R²=0.98), tactile (R²=0.99), and microstimulation (R²=0.95) cued trials. The left bound of the × axis is the behavioral tolerance window around the start target window.

Figure 5 also makes evident that even when delivered sufficiently early in the reach, microstimulation cues were considerably less reliable than visual or tactile cues in eliciting a redirected reach. (The monkey was ~80% correct in responding to a microstimulation cue presented just after the reach begins, versus >90% correct for visual and tactile cues.)

Note that this difference in ideal performance does not account for the differences in the critical feedback point. Even after rescaling the microstimulation curve so its peak performance matched the other modalities, the critical feedback point remained furthest from the target (55mm).

3.2.4. Pressured Reaction Time Task

The response to microstimulation during an ongoing movement was slower than we predicted. Cognitive demands of the redirect task might impair the animal’s ability to respond to microstimulation, and in contrast, attention and other cognitive factors may accelerate the animal’s response to the visual and tactile cues. We simplified the task in order to measure very directly how rapidly microstimulation could be used to influence behavior. In the pressured reaction time task, the monkey was required to react as quickly as possible to a cue. A visual, tactile, or microstimulation stimulus served as the go cue in a delayed center-out reaching paradigm. This allowed us to measure reaction times to the stimulus without any potential interference from ongoing movements, sensory feedback, or heightened cognitive demands. Even under these straightforward conditions, the animal’s reaction time to microstimulation was significantly slower than the response to tactile and visual cues (p <0.001, t-test) (table 1).

Table 1.

Reaction times in the pressured reaction time task and response times

Modality	Response Time (ms)	Reaction Time (ms)
Visual	211 +/− 34 (sd)	222 +/−48
Tactile	190 +/−27	184 +/−41
Microstimulation	263 +/−60	237 +/−34

4. Discussion

A crucial direction in the improvement of brain-computer interface performance is to provide fast, salient, informative feedback signals to BCI users [26]. In this study we compared the feasibility of three sensory modalities - vision, touch, and microstimulation in somatosensory cortex - as potential feedback signals to influence an ongoing movement. We defined a fundamental element of feedback as the ability of subjects to quickly adjust an ongoing movement after being provided with new sensory information about their movement goal. While human and monkey subjects reached toward visible targets, cues were presented that indicated a change in the goal location. Our first main finding is that tactile cues could be utilized to change the ongoing reach more rapidly than could visual cues. This suggests that a tactile feedback-based BCI could enhance BCI performance, at least in temporal aspects, compared to devices that provide only visual feedback. Of course, the tactile feedback would need to be provided on a body part not affected by the impairment. Our results indicate that such sensory transfer will not be a problem – our tactile stimulator was placed on the limb opposite to the reaching arm, and was behaviorally efficacious.

We did not attempt to mimic the natural sensory experience that would arise during a reach. Instead we focused on the question of whether an abstract sensory signal can provide meaningful feedback. Several considerations motivated this design choice. First, in a BCI context, for a patient with sensory loss as well as motor impairment, restoring natural sensory experience is a prospect that will face many challenges [26]. Here we wished to explore the feasibility of strategies that might be closer at hand. The visual stimulus was a large central flash of light – which does not resemble naturally occurring feedback – and we opted to compare tactile and visual stimuli as directly as we could.

Response times to tactile and visual cues have been compared in other studies, for example [31, 32], but in prior work the stimuli were used to indicate movement initiation, not to alter an ongoing movement. To our knowledge this is the first demonstration that tactile information can be used more rapidly than visual information to alter an ongoing movement. Considering that an ongoing movement can alter afferent sensory responses [60], this was important to verify. It is also important to have this information in a substantial cohort of subjects (n=10). This result might be unsurprising, but it is reassuring.

Our second main finding is that microstimulation can also serve as an informative stimulus modality to alter an ongoing movement. Microstimulation in sensory cortical areas has a long and impressive history as a means of providing sensory information to the brain [61, 34, 62]. Microstimulation in motor areas rapidly causes a movement [63, 64, 65, 66, 67, 68]. We reasoned that microstimulation in sensory areas could be used to adjust an ongoing movement more rapidly than tactile stimulation, since microstimulation can bypass sensory transduction delays. We were surprised to discover that microstimulation affected behavior considerably more slowly than did tactile and visual stimuli. To verify this observation we delivered sensory cues in a pared-down paradigm in which the animal simply had to react rapidly to the cue. Behavioral reaction times to microstimulation cues were still 53ms slower than to tactile stimulation.

4.1 Why is microstimulation feedback so slow?

Our results indicate that the neural activity induced by electrical stimulation must undergo further neural processing that takes time before it can influence behavior. What might happen to the stimulation-induced activity before it can influence behavior? We consider three candidate explanations. First, it may be that amplification by downstream processing is needed. Microstimulation presumably affects directly neurons in a small volume near the electrode tip, and indirectly affects neurons synaptically connected to the direct neurons [69, 70]. Through axonal branching, perhaps including positive feedback, neural activity induced by stimulation at a single point may gradually accrue and spread to the point that it achieves some threshold past which it is sufficient to influence subsequent processing. Natural sensory stimuli may be amplified in such a manner as they move from the sensory periphery through the thalamus to the cortex. If microstimulation were applied at an earlier stage in the sensory processing hierarchy, it may have the opportunity to exploit branching neural pathways that perform amplification, and actually lead to a faster response time than direct cortical stimulation can provide. The somatosensory portion of the thalamus (the core of the ventral posterior lateral region) projects to areas 1, 2, 3a, and 3b, with the strongest projection going to 3b. Had we stimulated in the thalamus or in an “earlier” cortical recipient zone, the response to microstimulation might have been faster.

Second, it might be that further processing is needed to refine microstimulation into a usable signal. Microstimulation presumably activates excitatory and inhibitory cell bodies, as well as fibers of passage with minimal discrimination [46, 71]. Such patterns may not be ‘natural’ in the sense that that particular combination of neural activities would never occur through endogenous processes, and thus, it is not directly effective in driving subsequent processes. As the perturbed network settles back into a more natural configuration of activity that is capable of influencing subsequent processes, eventually resulting in behavior. These concerns are exacerbated by the consideration that microstimulation must interact with ongoing neural activity, perhaps over-riding it [68], or combining with it in complex ways. Also in support of the notion that further cortical processing may be necessary to amplify and refine artificial signals, Horwitz and colleagues [72] elicited a behavioral response with optogenetic stimulation in primary visual cortex. They speculated that these effects might not have occurred if they had stimulated downstream, and thereby bypassed the amplification provided by subsequent cortical processing.

These explanations warrant further elucidation, and even if they are correct, they may not be sufficient to explain the slow response to microstimulation we observed. Our task was a simple detection paradigm - the animals only had to use the presence of a stimulus to react; they were not required to interpret it in any naturalistic manner. Stimulus detection is known to be a faster process than stimulus discrimination [73]. Our findings are reminiscent of the observations that, when peripheral nerves are stimulated in humans, they report that the sensation feels “artificial” [74]. An important avenue for future research will be observing how the brain responds to microstimulation [75]. It cannot be assumed those responses will resemble natural patterns as evinced by real sensation.

4.2 Perceptual comparisons

Many authors have shown behavioral effects of stimulating in sensory areas, including [34, 35, 41, 47, 52, 57, 76]. A question that underlies all these studies, not directly addressable using non-human subjects, is what perceptions are evoked by microstimulation in sensory areas? If the evoked perception from microstimulating S1 were similar to those evoked by stimulating the skin with tactile stimuli, then after a task is learned for tactile cues, replacing those cues with microstimulation should produce only marginal degradation in performance and engender a rapid understanding of the new stimulus. In studies stimulating area 3b, monkeys were able to compare the frequency of a microstimulation stimulus to a tactile stimulus [35]. Additionally, this comparison task was trained in only a few days, presumably because somatosensory microstimulation “felt like” touch to the animals. In contrast, other authors stimulating in area 1 have reported extensive training was required for animals to affiliate microstimulation with touch [39].

When stimulating in area 1, we verified that extensive training is required. When the microstimulation was first introduced, the animal ignored it until it was paired with a natural stimulus despite having ample experience performing the task with tactile stimuli. We first paired microstimulation and tactile stimuli using our detection task. Once the animal could react to microstimulation alone reliably in that simple task, we introduced microstimulation alone in the redirect task. Again, the animal would not react to the microstimulation until it was paired with tactile stimuli in the redirect task. After this new role for microstimulation had been learned, it was remembered in subsequent days of training without pairing. It seems to us that the perceptions for the two stimuli, microstimulation and tactile, were different enough that no perception-based association was made in either task. Only when the cues occurred simultaneously could the association then be made.

4.3 Applications to BCI

Our goal is to inform the design of feedback for BCIs. To provide feedback usable to guide an ongoing movement, a sensory modality should be informative and fast. Most current BCIs function solely under visual feedback. While vision is highly informative, it is slow. Our data suggest that a tactile-based feedback scheme might complement visual feedback by providing faster information. To our surprise, microstimulation-based feedback, at least in area 1, affected behavior more slowly than did tactile and visual feedback. We targeted area 1 for this study, since it is on the cortical surface, and thus is the most accessible target for a flat multielectrode array (Blackrock Systems Inc.). It may be that carefully designed patterns of microstimulation across a multielectrode implant [38] or optogenetic techniques might provide richer, faster feedback than we could achieve with a single electrode. Microstimulation patterning might be designed to mimic the responses that are recorded during actual touch [26].

Our study provides a strategy for the quantification of the effectiveness of temporal aspects of sensory feedback. It also highlights the need of careful quantifications like these; it cannot simply be assumed that microstimulation will yield a superior form of feedback in comparison to simpler approaches such as tactile and visual stimulation.

Supplementary Material

JNE497159suppfig1.pdf

Mean reaction times and standard error for monkey 2 plotted across days with regression line for visual, tactile, and microstimulation redirect cues. Day 1 took place two weeks after initial microstimulation training began. Success rate exceeded 80% for redirect trials with early microstimulation cues for every session thereafter. Regression line slopes are not significantly different from zero (p>0.05).

JNE497159suppvideo1.mp4

Sample direct reach trials and visual redirect trials. The video contains 3 successful direct reach trials followed by 2 redirect trials, the first successful, the second unsuccessful. Video of the trials were spliced together to show the different task types. Very rarely do two redirect trials occur back to back. Task configuration showing center start position (purple), potential reach target (gray), instructed reach target for that trial (yellow), hand position (small red sphere), and visual redirect cue (large central red sphere).