Sensing with the Motor Cortex

Abstract

The primary motor cortex is a critical node in the network of brain regions responsible for voluntary motor behavior. It has been less appreciated, however, that the motor cortex exhibits sensory responses in a variety of modalities including vision and somatosensation. We review current work that emphasizes the heterogeneity in sensori-motor responses in the motor cortex and focus on its implications for cortical control of movement as well as for brain-machine interface development.

Introduction

A curious observer outside of the field of motor neurophysiology might think that everything there is to know about the primary motor cortex (MI) has been learned. After all, MI is one of the earliest cortical structures to be functionally studied beginning with the electrical stimulation experiments of Fritz and Hitzig in the late 1800s (Fritsch and Hitzig, 1960). Motor cortical neurons have been designated historically as “upper motor neurons” implying that they are perhaps one synapse away from the motor neurons in the spinal cord that activate muscles (Kandel et al., 2000). So, according to this viewpoint, MI can’t be any more complex or interesting than muscle drivers sitting in the brain instead of in the spinal cord. As it turns out, however, the motor cortex is not so simple and its function remains elusive. First of all, despite extensive experimental and theoretical efforts for over fifty years, the exact computational and representational role played by the motor cortex remains unclear. Moreover, a number of recent studies have documented interesting sensory or sensory-triggered responses in the motor cortex which may require us to revise our understanding of the functional role of the motor cortex.

The use of the term “primary motor cortex” to define Brodmann area 4 is a designation that comes from the fact that movement can be most easily elicited through electrical stimulation of this area (Penfield and Boldrey, 1937). Moreover, it is known that approximately 30 to 50 percent of corticospinal projections originate from MI (Porter and Lemon, 1993). In addition, MI neurons typically begin modulating their firing rate up to several hundred milliseconds before a movement of the limb is initiated (Georgopoulos et al., 1982). Therefore, it is reasonable to consider this cortical structure as a primary motor area. However, this designation can obscure the fact that MI exhibits sensory responses and is part of a set of complex circuits that not only controls movement but also receives sensory inputs from the periphery. In the same vein, the designation of somatosensory cortex conceals the fact that this cortical structure also contribute to motor control as is evident in recent findings that the mouse barrel field controls retraction of the whiskers (Matyas et al., 2010). In this review, we present recent experimental data demonstrating a rich heterogeneity in MI response properties including strong visual and somatosensory effects and describe implications of these effects on encoding models of MI as well as their utility for augmenting brain-machine interface control.

Encoding models of motor cortex

Research over the past half century has attempted to understand what features of movement are encoded by individual MI neurons. Typically, these studies have developed models that capture the relationship between the firing rate of a neuron and the value of some kinematic, kinetic, or muscle variable. Although relationships have been documented with nearly every possible variable including force and torque (Cabel et al., 2001; Cheney and Fetz, 1980; Evarts, 1968; Hepp-Reymond et al., 1978; Kalaska et al., 1989; Smith et al., 1975; Taira et al., 1996), position (Georgopoulos et al., 1984; Paninski et al., 2004), velocity (Moran and Schwartz, 1999), acceleration (Stark et al., 2007), and distance (Fu et al., 1993), the most robust variables include movement direction and speed (Georgopoulos et al., 1982; Moran and Schwartz, 1999). A canonical model has emerged in the literature that linearly relates neuronal firing rate with velocity which includes speed and direction (Moran and Schwartz, 1999):

$$\mu ("\text{t})=\text{a}+"\stackrel{\rightharpoonup }{\text{B}}"·"\stackrel{\rightharpoonup }{V}"(\text{t}+\text{T})"$$ (1)

where μ(t) is the average firing rate, a is the baseline firing rate, B⃑ captures the preferred direction (i.e. the direction at which the cell’s firing rate is maximum) of the cell, V⃑(t) is the instantaneous velocity of the hand at time t, and τ is the delay between MI modulation and the kinematics. Typically, this delay parameter is estimated to be approximately 100 to 150 ms as described above (Ashe and Georgopoulos, 1994; Moran and Schwartz, 1999; Paninski et al., 2004; Suminski et al., 2009).

A number of recent studies, however, have shown that the preferred direction (PD) of a cell is highly context dependent, varying in orientation depending on the posture of the arm (Scott and Kalaska, 1995) and the position of the hand (Caminiti et al., 1990; Wu and Hatsopoulos, 2006a). More strikingly, PDs can even vary in time over the course of a simple reaching movement (Churchland and Shenoy, 2007; Mason et al., 1998; Sergio et al., 2005; Sergio and Kalaska, 1998). Sergio and Kalaska (Sergio et al., 2005) compared the tuning properties of MI neurons while monkeys performed nearly identical tasks under either isometric or movement conditions. In the isometric condition, monkeys were trained to exert forces on a transducer to move a cursor from a center target to one of eight peripherally positioned targets, while, in the movement condition, the monkeys moved the end of a manipulandum to guide the cursor to each of the eight targets. Although PDs remained temporally stable under the isometric condition, the authors observed dramatic shifts in PD orientation in time under the movement condition.

Churchland and Shenoy (Churchland and Shenoy, 2007) found similar temporal shifts in preferred direction when monkeys performed reaches to targets in the vertical plane in different directions and distances and under two different instructed speeds. Moreover, they observed a rich heterogeneity and complexity in temporal response properties among the population of recorded neurons which could not be accounted for with just the canonical model (equation 1). In fact, there were many cases where neurons exhibited unique temporal firing profiles that were not shared by any other neuron in their population. They authors put forth the possibility that this heterogeneity and complexity may serve as a rich basis set to represent a variety of different movement parameters, However, they favored an alternate and intriguing idea that the motor cortex may actually not be specifically encoding any particular feature of movement (Wu and Hatsopoulos, 2006b). Instead, the heterogeneity and temporal complexity of observed responses is simply the consequence of a recurrent network that is attempting to provide signals to the spinal cord to control movement. Output neurons that form the corticospinal tract represent a subset of a much higher dimensional, dynamical system of neurons which may not clearly represent anything but rather serve to shape the appropriate temporal responses of the output neurons.

We have recently put forth a model that attempts to capture the heterogeneity of motor cortical responses (Hatsopoulos et al., 2007). This model suggests that MI represents a rich set of movement fragments which is more in line with the basis set idea described by Churchland and Shenoy (Churchland and Shenoy, 2007). The model begins with the observation that the PDs vary not only in absolute time (i.e. over the course of a movement) but also in relative time (i.e. relative to the observed neural modulation). Instead of postulating that the motor cortex encodes a parameter of motion such as direction and speed at a fixed time lag as in equation 1, we have suggested that MI neurons are tuned to direction at multiple time leads and lags relative to the time of the measured firing rate and that these preferred directions can vary sometimes substantially at these different time delays. More relevant to this review, we have found that MI neurons have preferred directions at negative time lags suggestive of “sensory” as well as “motor” tuning (Figure 1A). By vectorally adding these preferred directions, we argued that individual neurons are tuned to complex movement fragments or trajectories (Figure 1B). This led us to build a generalized linear encoding model where MI neurons are tuned to velocity trajectories measured at multiple time lags including negative, sensory and positive motor influences on MI activity (Hatsopoulos et al., 2007):

$$log\mu (t)=\text{a}+\sum _i\stackrel{\rightharpoonup }{B_i}·\stackrel{\rightharpoonup }{V}(\text{t}+{\tau }_i)$$ (2)

Figure 1.

Sensory and motor encoding in motor cortex. A. Temporal evolution of preferred directional tuning for one motor cortical neuron estimated at a 50 ms time resolution. Preferred directions (PDs) were estimated by comparing neural modulation with instantaneous movement direction at multiple relative lead/lag times. Blue arrows represent “sensory” PDs measured when neural modulation lagged movement direction. Red arrows represent “motor” PDs measured when neural modulation led movement direction. The green arrow represents the PD measured with a lead/lag time of zero. B. By adding vectorally the PDs at different lead/lag times, a “preferred” trajectory is generated. C. The preferred trajectory or “pathlet” generated for the same neuron using our generalized linear encoding model (Hatsopoulos et al., 2007). The red line represents the 300 ms preferred trajectory segment following the neural response while the blue line represents the 100 ms preferred trajectory segment preceding the neural response. D. A map of heterogeneous pathlet shapes from a population of MI neurons simultaneously recorded across a 4mm × 4mm region of the precentral gyrus (Figure panel adapted from (Hatsopoulos et al., 2007)).

Notice the logarithm transform on the mean rate of the neuron which ensures the rate cannot be negative. Using constrained movements in the horizontal plane, we found that more of the variability of MI activity could be captured if we assumed neurons were tuned to velocity trajectories that extended 400 ms in duration, 100 milliseconds prior to neural activity and 300 ms following neural activity. By temporally integrating these preferred velocity trajectories, a preferred movement fragment or “pathlet” can be constructed which possesses both a sensory and a motor component (Figure 1C). Over a population of simultaneously recorded MI neurons, we observed a heterogeneous set of pathlets with complex and unique shapes (Figure 1D).

More recently, we have provided further support for fragment encoding in MI during natural grasping behavior (Saleh et al., 2010). In particular, we demonstrated that MI neural modulation can be more accurately predicted if we assume that individual neurons encode joint angle and angular velocity trajectories involving joints of the wrist, and fingers. These temporally extensive trajectories express both “sensory” aspects of movement preceding the neuron’s response by up to 164 ms in the past as well as “motoric” features of the movement following neural activity extending up to 200 ms into the future. Similar sensory and motor properties have been documented even at the level of muscles (Pruszynski et al., 2010).

Sensori-motor information in MI

Instead of resorting to an explicit encoding model, one can quantify the sensorimotor relationships between motor cortical modulation and movement using information theory. In particular, mutual information can capture non-linear as well as linear relationships between these two variables (Cover and Thomas, 1991). By shifting the relative timing between the spike train and the movement, the strength of the peak mutual information as well as the relative time at which the peak occurs can provide clues as to whether the coded information is “motoric” or “sensory” in nature. In simple terms, mutual information quantifies the reduction of uncertainty in one variable given the value of a second variable. For example, if a monkey can move in one of eight possible directions (i.e. 3 bits of uncertainty), and the measured firing rate of a neuron reduces the uncertainty to only two directions (i.e. 1 bit of uncertainty), the mutual information between direction and the firing rate of the neuron is 2 bits (i.e. 3−1=2). The mutual information between the instantaneous direction of limb movement and the firing rate of an MI neuron measured at multiple relative time lags can capture the degree of directional tuning as well as the relative timing at which these two variables are most related. It is typically observed that MI firing is most strongly correlated with movement direction of the arm when neural activity is leading movement by approximately 100 to 150 ms as is evident in the peak in the information profile at a positive time lag (Figure 2, top panel) (Ashe and Georgopoulos, 1994; Moran and Schwartz, 1999; Paninski et al., 2004; Suminski et al., 2009). This lag time is perhaps not surprising given the axonal, synaptic, inertial, and muscle recruitment delays between the cortex and the limb. However, what is perhaps less expected is the existence of a peak at a negative time lag in some MI neurons indicating that movement direction is also providing information about neural activity in the future suggestive of a sensory as well as motor response in MI (Figure 2, bottom panel). This begs the question as to what role these sensory-like responses are playing. Are these sensory responses assisting in the sensory guidance of movement? Or perhaps, MI plays a fundamental role in kinesthetic perception together with the somatosensory cortex. In fact, surface electrical stimulation of the precentral cortex can evoke sensory percepts in human patients undergoing neurosurgical procedures (Nii et al., 1996; Penfield and Boldrey, 1937; Woolsey et al., 1979). In addition, lesions to the precentral cortex can effect kinesthetic perception (Naito et al., 2011).

Figure 2.

Mutual information profiles between neural modulation and instantaneous movement direction as a function of the lead/lag time between the measured firing rate and direction. Top panel. A motor cortical neuron with a peak mutual information value at a positive lead/lag time indicating that neural modulation correlates with instantaneous movement direction in the future. Bottom panel. A motor cortical neuron with a peak mutual information value at a negative lead/lag time indicating that neural modulation correlates with instantaneous movement direction in the past.

Visually-evoked motor responses in MI

Evidence over the past twenty years has demonstrated that many neurons in premotor cortices discharge similarly in response to overt motor performance and the observation of the same motor action. Rizzolatti and colleagues first documented the existence of these so-called ‘mirror’ neurons in the ventral premotor cortex of non-human primates (di Pellegrino et al., 1992; Gallese et al., 1996; Rizzolatti et al., 1996). Cisek and Kalaska observed a similar phenomenon in dorsal premotor cortex (PMd) when monkeys moved a visual cursor to one of eight peripherally positioned targets displayed on a computer monitor in front of them with their unseen arm or observed cursor movements made by an unseen experimenter (Cisek and Kalaska, 2004). Two colored targets appeared briefly to cue possible movements and the correct movement was subsequently identified by a color cue. Monkeys were trained to reach to the peripheral target indicated by the color cue at the presentation of a go signal. Behavioral evidence demonstrated that the animals engaged in mental rehearsal during the observation of action as the experimenters found that the monkey usually made saccades to the correct target before reaching or observing cursor motion. Furthermore, the spiking activity recorded from neurons PMd exhibited the same pattern of modulation during active performance and observation even during a delay period before movement had begun.

Based on the dense cortico-cortico connections between MI and these premotor areas (Dum and Strick, 1991; Dum and Strick, 2005) and indirect evidence from psychophysical (Flanagan et al., 1993; Mattar and Gribble, 2005), functional imaging (Cheng et al., 2007), magnetoencephelography (Jarvelainen et al., 2004), electroencephelography (Muthukumaraswamy and Johnson, 2004; Nishitani and Hari, 2000), metabolic labeling (Raos et al., 2007), and stimulation (Fadiga et al., 1995; Maeda et al., 2002; Stefan et al., 2005) studies, it would be expected that MI neurons discharge in response to action observation.

Recent experiments from our lab have demonstrated that many neurons in MI discharge similarly during action and action observation much the same way as neurons in dorsal and ventral premotor cortices (Suminski et al., 2009; Tkach et al., 2007). Monkeys were trained to perform planar movements in order to move a cursor to randomly positioned targets presented visually on a horizontal screen. During active performance the cursor’s position was dictated by the endpoint of a two-link exoskeletal robot moved by the monkey’s arm. The cursor movements and accompanying targets were recorded and subsequently replayed to the animal during the observation phase providing the same visual stimulation as active performance. We required the monkeys to voluntarily maintain the position of their arm in a fixed posture during observation to eliminate the effects of arm movements on the neural activity measured in MI. We found a tremendous amount of heterogeneity when examining the responses of individual neurons during active performance and observation. The firing rate profiles of some neurons exhibited noticeable differences under the two conditions (Figure 3A) while others modulated similarly in the two tasks (Figure 3B). Most surprising was a group of cells whose firing rate was strongly modulated during action observation but did not discharge in response to active movement of the limb (Figure 3C).

Figure 3.

Heterogeneous neural responses measured during action and visual observation. A–C. This diversity is well illustrated by peri-event average firing rates (Top panels) and rasters (Bottom panels) during active movement (blue) and visual observation (gold) for three exemplar neurons. D–F. Directional mutual information profiles for the same three neurons. Despite the modulations observed in the firing rates of individual neurons, analysis of the mutual information between spiking activity and cursor movement often revealed different patterns. For example, the neuron represented by panels B and E demonstrated similar firing rate modulations during both movement and observation, but the information analysis revealed that this neuron responds most strongly to the observation task.

While some neurons exhibited significant mutual information between spiking activity and cursor direction during active performance but not during observation (Figure 3D), a number of neurons showed significant mutual information during observation (Figure 3E), and, in some cases, there was significant mutual information only during observation (Figure 3F). This analysis provided evidence that MI may be a member of this putative ‘mirror’ neuron system.

Mutual information estimates indicated that neural modulation led cursor movement for both voluntarily executed movement and passively observed cursor movement. This prospective activity during observation in MI is consistent with a role in the mental rehearsal of action and is similar to other reports in both PMd (Cisek and Kalaska, 2004) and MI (Dushanova and Donoghue, 2010). However, the time lag between neural activity and movement tended to be shorter for observed cursor movements (approximately 50ms, e.g. Figure 2E,F) compared to overt arm movements (approximately 125ms, e.g. Figure 3D) meaning that the neural modulation occurred closer in time to the movement itself. This decrease in lag time may reflect the change in the dynamic properties of the task between active performance and observation. That is, during active performance the motor commands issued by MI are filtered by the motor plant (e.g. transmission, muscle recruitment and inertial delays) causing the typical delay between activity in cortex and subsequent behavior. In contrast, during observation these delays are not present leading to a shorter latency between cortical firing and observed movement. It is currently unknown whether the neural activity in MI elicited during action observation/mental rehearsal contains a representation of the kinetics of movement (i.e. hand force or joint torque) as has been well documented during active performance (Cabel et al., 2001; Evarts, 1968; Sergio et al., 2005) in addition to information about movement kinematics.

Somatosensory responses in MI

Despite the importance of somatosensation in movement control (Ghez and Sainburg, 1995; Sainburg et al., 1995; Sainburg et al., 1993), the functional significance of cutaneous and proprioceptive responses in motor cortex have been largely ignored over the past twenty five years (see (Herter et al., 2009; Pruszynski et al., 2011a), however, for recent work). A number of older electrophysiological studies have documented somatosensory responses in MI neurons using tactile stimulation, perturbation, and passive movement paradigms (Albe-Fessard and Liebeskind, 1966; Evarts and Tanji, 1976; Fetz et al., 1980; Flament and Hore, 1988; Fromm et al., 1984; Goldring and Ratcheson, 1972; Lemon et al., 1976; Lucier et al., 1975; Wise and Tanji, 1981; Wong et al., 1978). Many of these studies conceptualized these results within the framework of a long-loop “reflex” mediated by the motor cortex (Phillips, 1969; Wiesendanger et al., 1975). Early theories of the long-loop “reflex” suggested that it functioned much like the short-latency spinal reflexes receiving local spindle information from muscles about the joint that was perturbed and activating homonymous or synergistic muscles to generate corrective movements.

A more refined view argued that the long-loop “reflex” could generate a more intelligent coordinated response by activating multiple muscles in response to a local perturbation in order to compensate for undesired components of the corrective movement (Gielen et al., 1988). For example, a perturbation in the pronation direction would stretch both supinator and biceps muscles. However, the biceps also acts to flex the arm which would be undesired, and so the long-latency responses (presumably mediated by the motor cortex) was evident not only in the stretched muscles but also the triceps muscle to compensate for the undesirable flexion motion that would be generated by the biceps (Gielen et al., 1988). Very recently, “intelligent” feedback responses have been observed at the level of the motor cortex due to perturbations about the shoulder and elbow (Pruszynski et al., 2011b). These authors observed differential responses in shoulder-tuned MI neurons as early as 50 ms following two different perturbations (i.e. a perturbation at the shoulder and a perturbation at the elbow) even though the two perturbations resulted in the same shoulder motion. In other words, despite ambiguous sensory information from the shoulder muscles, MI integrated sensory information from both the shoulder and elbow to generate an appropriate corrective response. These “intelligent” forms of feedback control involving the motor cortex are consistent with current theories of optimal feedback control which go beyond older servomechanistic accounts of the role of sensory feedback in motor control (Scott, 2004; Todorov and Jordan, 2002).

We have recently examined the effects of somatosensory feedback on the directional tuning of MI neurons by comparing responses during active and passive movements in the awake monkey. As previous studies have found (Fetz et al., 1980; Lemon et al., 1976), we observed two distinct populations of MI neurons: one population which fired in an incongruent fashion for passive and active movements of the arm involving coordinated flexion and extension of the shoulder and elbow joints whereas a second population fired in a congruent manner (Suminski et al., 2009). The first “incongruent” neural population had preferred directions that were 180 degrees apart when measured during active and passive conditions (Figure 4A, green bars). During active movement, this subpopulation exhibited a median information lag time of +100 ms (Figure 4B, dark green curve) which suggested that this population was “driving” movement during voluntary movement. However, during passive movement, this population showed a median directional information peak lag time of −50 ms indicating that neural modulation lagged movement (Figure 4B, light green curve). This response latency is consistent with long-loop sensory effects on MI reported by others (Fetz et al., 1980; Lemon et al., 1976; Pruszynski et al., 2011b). If we assume that this population is providing “driving” signals to contract certain muscles during active movement but also receiving spindle afferent information from the same or synergistic muscles, then it would be expected that this cell subpopulation would show increased firing when the muscles were being stretched during passive movement.

Figure 4.

Tuning properties during active and passive movement. A. The distribution of neuronal preferred direction differences between active and passive movement conditions. The green bars represent neurons whose preferred directions were oriented 180 apart between active and passive movement conditions (i.e. incongruent population). The purple bars represent neurons whose preferred directions were similar during active and passive movement conditions (i.e. congruent population). B. Mean (+/−1 standard error) directional mutual information profiles for the incongruent population during active (dark green) and passive (light green) movement conditions. C. Left panel. Mean (+/−1 standard error) directional mutual information profiles for the congruent population during active (dark purple) and passive (light purple) movement conditions. Right panel. Hypothesized function of the congruent population. During active movement, neural information (blue dashed line) predicts sensory consequences of active movement (solid black line) by 50 ms. During passive movement (solid black line), a covert motor command is triggered by sensory feedback (solid red line) which lags movement by 50 ms. This covert motor command leads to neural modulation which predicts the measured sensory consequences by 50 ms. Therefore, neural information tracks movement with no lead or lag.

The second “congruent” neural population exhibited preferred directions that were similar during active and passive movements (see Figure 4A, purple bars). This population led movement by a median value of +50 ms during active movement (Figure 4C, left panel, dark purple curve). However, in contrast to the first population, the median information peak lag time was 0 ms during passive movement indicating neural modulation tracked movement direction with no motor lead or sensory lag (Figure 4C, left panel, light purple curve). How do we explain real-time tracking of movement without a sensory lag?

One intriguing albeit speculative hypothesis is that this population may be serving to predict the future sensory consequences of motor commands. Evidence from psychophysical and modeling studies suggests that the nervous system can predict the sensory consequences of motor actions (Desmurget and Grafton, 2000; Nelson, 1996). This function has been traditionally localized in the parietal cortex or cerebellum (Desmurget et al., 1999; Kawato, 1999; Mulliken et al., 2008b), although there is some evidence for predictive signaling even in the motor cortex (Flament and Hore, 1988). The argument for this hypothesis is as follows. During active movement, this population may not be causally “driving” movement because it leads movement by only 50 ms instead of 100–150 ms which is the typical “driving” delay seen in motor cortex during reaching movements (Ashe and Georgopoulos, 1994; Moran and Schwartz, 1999; Paninski et al., 2004). Instead, it could be predicting future movement direction 50 ms in advance of the actual movement (Figure 4C, right top panel, blue dashed line). The actual source of this predictive signaling could originate in some other cortical or subcortical area. During passive manipulation, one needs to assume that somatosensory feedback (i.e. tactile and proprioceptive input) can trigger covert motor commands much like the neural population described in the previous section that generated visually-evoked covert motor commands. Somatosensory feedback would reach motor cortex with a delay of ~50 ms (Figure 4C, right bottom panel, red curve). This input would trigger a covert motor command leading the sensory feedback by ~100 ms. If this population of neurons predicts the future sensory consequences of the covert motor command by 50 ms, then it would provide information preceding the sensory feedback by 50 ms (Figure 4C, right bottom panel, blue dashed line). Therefore, the predictive sensory lead in this population would offset the sensory delay in the periphery resulting in real-time tracking of movement. This hypothesis is further supported by the fact that the congruent subpopulation exhibited a 50% increase in peak directional information during passive movement as compared to the incongruent subpopulation indicating that the congruent subpopulation is more faithfully capturing the detailed dynamics of movement.

Sensation in training and control of brain machine interfaces

In the previous sections of this review, we have discussed literature demonstrating the richness and diversity in MI neural responses measured during the visual observation of familiar actions, passive movement of the limb, and voluntarily generated movements. This diversity is readily apparent in Figure 5 which shows the normalized binned firing rate as a function of time for each of the 87 neurons recorded during an experiment where monkeys generated active arm movements (blue region), observed playback of recorded movements with only visual (gold regions), proprioceptive (gray), or both types of feedback (red regions). Changes in the experimental condition were precisely correlated with substantial changes in the firing rate of individual neurons appearing as vertical striations in Figure 5. These heterogeneous responses are particularly interesting and potentially advantageous when placed in the context of a neuroprosthetic device or brain-machine interface (BMI).

Figure 5.

The heterogeneity of neural responses in MI is easily visible in the time series of binned firing rates (50ms) for all units recorded during a single session. The color bar at the top of the figure denotes task conditions where monkeys either generated active arm movements (blue region), observed playback of recorded movements with only visual (gold regions), proprioceptive (gray) or both types of feedback (red regions). Bins shown in white represent the highest firing rates for each cell, while black areas correspond to times when the firing rate was near its minimum. Notice the substantial changes in the firing rates of some cells at the transitions between experimental conditions (especially those cells denoted by the black brackets). Firing rates from each individual neuron were binned and normalized to their maximum firing rate. The resulting time series were then smoothed using a zero-phase, 4^th order, butterworth, lowpass filter with a cutoff frequency of 0.1Hz for display purposes. Figure reproduced with permission from (Suminski et al., 2009).

The paramount goal of all BMI research is to provide individuals with severe motor disabilities a device that can adequately restore lost functionality. The majority of BMIs consist of four constituent components: a neural interface, a neural decoder, some type of end-effector (i.e. a computer cursor or robotic device) and sensory feedback (Figure 6A). First, a neural interface monitors the activity of many neurons simultaneously. This interface is often an intracortical microelectrode array inserted directly into MI that records single and multi unit spiking activity. However, others have successfully implemented BMIs by recording the activity of neurons in parietal cortex using microelectrodes (Carmena et al., 2003b; Mulliken et al., 2008a; Musallam et al., 2004) or neural activity from multiple brain regions using electrocortography (Leuthardt et al., 2009; Moran, 2010) and electroencephalography (Wolpaw and McFarland, 2004). The activity recorded by the neural interface is presumed to encode task or goal specific information that can be translated into behavior by a neural decoder. The physical manifestation of the neural decoder’s output is realized through the motion of some end-effector which is most often the movement of a visual cursor or robotic arm in two or three dimensions. Finally, sensory feedback provides for a closed loop system allowing users to observe movements of the end-effector and correct errors when necessary.

Figure 6.

A schematic representation of the constituent components for using and training an intracortical brain machine interface. A. Similar to all movement tasks, the presentation of a goal is translated into a motor plan that is encoded in the activity of neurons in various movement related brain regions (primary motor cortex in our implementation). A neural interface is used to monitor and extract the activity of these neurons that encode movement. A neural decoder transforms the measured neural activity into behaviorally relevant variables (i.e. position, velocity, force or torque) that can be used to control the movement of an end-effector (i.e. a visual cursor or robotic arm). Various types of sensory information are provided to the user via intact visual and/or kinesthetic feedback pathways or through artificial electrical stimulation (intracortical microstimulation, ICMS) of primary somatosensory cortex. B. To create a neural decoder, neural population activity must be mapped to overt behavior with which it is related using some fitting or optimization methodology. This behavior can take on many forms including overt movements, passive observation of familiar actions and/or imagination of various motor tasks.

A critical procedure in the development of any BMI is the creation of the neural decoder (Figure 6B). In its simplest form, the decoder is created by finding a linear relationship between neural activity and some feature of the simultaneously recorded behavior (i.e. position, velocity or torque) that allows subjects to control the movement of an end-effector by modulating their neural activity. In pre-clinical studies using intact non-human primates, decoders have typically been constructed using neural activity measured while the subject performed overt arm movements (e.g. (Carmena et al., 2003a; Serruya et al., 2002; Taylor et al., 2002). Unfortunately, the majority of individuals who would benefit from a BMI are unable to produce overt movements requiring different procedures to train the neural decoder.

The visually evoked motoric responses observed during mental rehearsal/action observation represent an alternative methodology for training decoders. In fact, multiple groups have recently demonstrated the ability of both monkeys (Suminski et al., 2010; Velliste et al., 2008; Wahnoun et al., 2006) and human subjects (Hochberg et al., 2006; Truccolo et al., 2008) to successfully use BMIs with neural decoders that were trained using the neural responses evoked during mental rehearsal/action observation or motor imagery. Wahnoun and colleagues (Wahnoun et al., 2006) were the first to address the problem of establishing a neural decoder in the absence overt arm movements. They trained non-human primates to passively observe computer generated 3D cursor movements in order to derive an initial estimate of the tuning parameters for each neuron used in BMI control. After this short calibration period, monkeys were able to successfully move the visual cursor to hit targets by modulating the activity of the population of recorded neurons.

In a recent set of clinically relevant human experiments of an intracortical brain-machine interface, a more practical two-state (point and click) neural decoder was trained using neural activity measured in the absence of overt movement (Kim et al., 2011; Simeral et al., 2011). In order to train the trajectory generation component of the decoder, human subjects with tetraplegia were instructed to observe computer generated movements of a visual cursor while imagining that they were controlling the cursor. The patients were instructed to imagine squeezing or opening their hand in response to a discrete visual cue in order to train the click functionality. Despite the lack of overt movement during training, the patients were able to achieve successful control of the BMI with one participant reaching a 97% success rate. These studies clearly demonstrate the utility of the neural responses measured during observation and imagination of action for the creation of neural decoders.

Ultimately, the goal of all BMI research is to provide individuals with severe motor disabilities a device that can adequately replace lost afferent as well as efferent functionality. The potential utility of incorporating additional forms of sensory feedback, including tactile and proprioceptive feedback, to BMIs which typically incorporate feedback only from vision has been widely suggested (Abbott, 2006; Gilja et al., 2011; Hatsopoulos and Donoghue, 2009). In fact, some have begun to explore methodologies to integrate different forms of sensory feedback in BMI systems. Direct electrical stimulation of the somatosensory cortex via microelectrodes has been shown to elicit discernable sensory percepts in primates for the purpose of frequency discrimination (Romo et al., 1998) or cuing of upcoming reach targets (Fitzsimmons et al., 2007). Similarly, Dihillon and Horch reported that amputees were able to discern either the grip force or joint position of a prosthetic arm based on the frequency of electrical stimulation in residual peripheral nerves (Dhillon and Horch, 2005).

More recently, O’Doherty et al. have effectively combined an efferent intracortical brain-machine interface with somatosensory feedback provided by direct intracortical microstimuation (ICMS) of primary somatosensory cortex (O’Doherty et al., 2009; O’Doherty et al., 2011). Monkeys were trained to move a visual cursor from a central target to one of two peripheral targets based on the presence of a vibrotactile cue. After a training period of 15 sessions, the vibrotactile cue was replaced by ICMS. After a period of relearning (20 sessions), the monkeys achieved a task success rate (90%) in the ICMS condition that was equal to the performance level achieved with the vibrotactile stimulus (O’Doherty et al., 2009). In a later study (O’Doherty et al., 2011), monkeys were trained to actively explore an environment containing three targets using an intracortical brain-machine interface and select the appropriate target based on the type of vibrotactile or electrical stimulation. During the ICMS conditions, monkeys were required to discriminate between three different artificial textures (a rewarded texture, an unrewarded texture and no texture) and select the appropriate target based on the frequency of stimulation. Monkeys were able to achieve a success rate higher than chance demonstrating their ability to discriminate the textures communicated via ICMS (O’Doherty et al., 2011). Taken together these results demonstrate that ICMS is a valid methodology for providing artificial somatosensory feedback in order to cue the location of rewarded targets during BMI control,

Despite these efforts to augment BMIs with additional forms of feedback, their actual impact on real-time sensory guidance of a cortically controlled BMI has been largely unexplored. We recently applied an alternate approach to address this gap in BMI research and performed an experiment in which the presence of naturalistic proprioceptive feedback during BMI control was systematically varied (Suminski et al., 2010). First, monkeys observed a visual replay of active movements they made earlier in the same session while voluntarily maintaining a fixed arm posture in a robotic exoskeleton. During observation, we used the visually-evoked motoric responses present in MI (see Visually-evoked motor responses in MI) to build the neural decoders used in this study. Later in the experiment, the monkeys used the decoders to control the position of a visual cursor in a 2D environment. We found that each monkey moved the visual cursor faster and straighter when using a BMI that provided congruent visual and proprioceptive feedback (Vision + Proprioception BMI) by passively moving the arm to follow the visual cursor compared to a BMI with visual feedback alone (Vision BMI). These results support the generally assumed notion that incorporating additional feedback modalities (i.e. proprioceptive or somatosensation) in a BMI will lead to performance increases.

Unlike the active movement and Vision BMI conditions (Figure 7A & B), we found a bimodal distribution of peak mutual information lags during the Vision +Proprioception BMI condition indicating that two distinct populations of neurons in MI were active when both feedback modalities were congruent (Figure 7C). Three pieces of evidence led us to conclude that the first population of cells processes information related to either congruent sensory feedback or to proprioceptive feedback alone (Figure 7D). First, the time lags of peak mutual information for this population were negative indicating that neurons discharged 60ms after cursor movements. Second, we saw a very weak response in this population during the Vision BMI condition demonstrating the dependence of this population on arm movement. Finally, there was no significant difference in peak information magnitude or time lag in the active movement and Vision +Proprioception BMI condition suggesting that cells were performing similar computations in these different conditions. The second population had a response that was consistent with the typical driving behavior seen in MI (Figure 4E). That is, the spiking activity of MI preceded behavior and was strongly modulated by movement direction leading us to believe that this population is primarily responsible for movement of the arm or visual cursor in the active movement and BMI conditions, respectively.

Figure 7.

Variations in directional mutual information under active movement and brain-machine interface conditions. A–C. The distribution of lead/lag times at which directional mutual information peaked for Active Movement (blue), Vision BMI (gold), and Vision+Proprioception BMI (red) conditions, respectively. The dotted vertical line intersecting the histograms represents a time lag of zero. Only neurons having a significant peak information magnitude were included in each condition. D,E. We found separable neural populations during the Vision + Proprioception BMI condition. Mean (± 1 standard error) mutual information profiles of cells belonging to the first and second mode of the bimodal distribution of peak mutual information lags seen in upper inset. Figured adapted from (Suminski et al., 2010).

Conclusion

There is no dispute that the primary motor cortex is an important cortical site in voluntary motor control. However, the term “motor” cortex conceals the fact that MI can exhibit strong sensory responses as well. These sensory responses are not surprising if one considers that MI is a node in a set of complex sensori-motor loops. Moreover, sensory stimulation appears to be able to trigger covert motor commands in motor cortex even without overt movement execution. In particular, visually represented actions can trigger mirror-like responses in MI that mimic neural modulation that occurs during voluntary movement. Moreover, somatosensory inputs may also be able to trigger covert movement commands during passive movement paradigms.

What is perhaps the most striking conclusion from our recent studies as well as those of others is the heterogeneity of response properties in motor cortex (Churchland and Shenoy, 2007). Some neurons fire predominantly during voluntary movement but not during visual playback or passive movement. Other neurons fire predominantly during visual playback or during passive movement but not during voluntary movement. And still others respond to different combinations of voluntary movement, visual playback, and passive movement. This heterogeneity may explain in part the lack of a unified theory of motor cortical functioning. Moreover, this diversity in sensori-motor responses may have important implications for a cortically controlled brain-machine interface.