Useful signals from motor cortex

Abstract

Historically, the motor cortical function has been explained as a funnel to muscle activation. This invokes the idea that motor cortical neurons, or ‘upper motoneurons’, directly cause muscle contraction just like spinal motoneurons. Thus, the motor cortex and muscle activity are inextricably entwined like a puppet master and his marionette. Recently, this concept has been challenged by current experimentation showing that many behavioural aspects of action are represented in motor cortical activity. Although this activity may still be related to muscle activation, the relation between the two is likely to be indirect and complex, whereas the relation between cortical activity and kinematic parameters is simple and robust. These findings show how to extract useful signals that help explain the underlying process that generates behaviour and to harness these signals for potentially therapeutic applications.

I am a systems neurophysiologist who studies the volitional control of arm movement. Specifically, I am interested in the organization of patterned activity in frontal cortex thought to underlie volitional movement. This paper is based on a lecture I gave at a symposium entitled ‘Physiology of brain–computer interfaces’ that preceded the Society for Neuroscience meeting in October, 2006. The lecture is my attempt to place a body of work in the context of the historical development of motor cortical investigation, systems neurophysiology in general and where I believe the field is going. I have been lucky to work in a period when the traditional view of motor cortical function has been changing rapidly. As with any change of this type, there has been a fair amount of controversy and this has forced me to look at how and why the traditional view developed and why this shifted so rapidly. It is important to understand why scientific opinion formed the way it did and to separate this from the basic observations drawn from current work. Once this separation is distinct, the new findings can be used to gain fundamental insight into movement generation and how we might be able to use this knowledge therapeutically. The paper is laid out to provide a historical background leading to classical ideas of motor cortical function, an exposition of how common opinions formed to produce a number of key arguments and an avenue that may be useful in bypassing these arguments to generate advances in the understanding of cortical operations leading to behavioural output.

Historical development

The modern age of the study of cerebral motor control began with the work of Fritsch & Hitzig in 1870 (Fritsch & Hitzig, 1870). The experiment proposed by these two, who were then young investigators in the physiology department at the University of Berlin, used electric shocks applied to the cerebral cortex. The prevailing view was that the cortex, thought to house the soul, was electrically inexcitable and the proposed experiment was considered so dubious that it had to be performed on their own parlour table. Using ether to anaesthetize a dog, they then exposed its cerebral cortex and applied galvanic shocks through a pair of wires held in a piece of cork. When activating a region of cortex around the pericruciate sulcus, they found they could elicit movement consistently on the contralateral side of the animal. These experiments were instrumental in showing both that the cerebral cortex was electrically excitable and that movement could be elicited from a circumscribed cortical region. Over the ensuing century, this work was repeated and extended with more precise techniques (Ferrier, 1875; Leyton & Sherrington, 1917; Woolsey, 1958). Maps, generated with low-intensity surface stimulation, showed that the motor cortex lying anterior to the central sulcus was organized somatotopically, with the feet and legs in the medial wall of the hemisphere, the hip, trunk and shoulder on the convexity and a continuous representation of the body extending laterally with the face and hands farthest from the mid-line. (Fig. 1). When punctate activation points were connected, the resulting picture resembled a distorted figurine or ‘simiusculus’ of the monkey. The same essential map was found in human patients (Penfield & Rasmussen, 1950).

Figure 1.

Delineation of motor cortex A, body movements resulting from 60 Hz AC stimulation of the motor cortical surface. B, stimulus intensities at which corresponding movements in A were just elicited. C, figurines created in primary and supplementary motor cortices from electrical stimulation as well as maps generated in primary and secondary sensory areas from peripheral stimuli. Figure from Clinton Woolsey published in Phillips & Porter (1977); reprinted with permission from Elsevier.

The reasons that the precentral sulcus is susceptible to this type of electrical activation are twofold. First, the primary motor cortex contains large, pyramidal Betz cells, which because of their size, shape and orientation, form dipoles that are efficient in sinking anodal surface current to generate action potentials. Second, some of the corticofugal output from motor cortex projects directly to the spinal cord to activate muscles.

So the textbook description of motor cortical function is based on these historical concepts. Specifically, neurons in the motor cortex are considered to be ‘upper motoneurons’, laid out in a somatotopic map, that when activated lead to an obligatory muscle contraction (online Supplemental material, Movie 1).

However, deeper probing of this issue shows that these concepts are limited (Phillips & Porter, 1977). The corticofugal projection from primary motor cortex (M1) projects to many sites throughout the neuroaxis. The motor cortex contribution is a fraction (40%) of the total cortical input to the corticospinal tract. Even the relatively few monosynaptic fibres projecting to α-motoneurons send collaterals to a variety of distributed motoneuron pools. Multiple routes to spinal motoneurons can be followed from each motor cortical output neuron and the vast majority of these are polysynaptic (difficult to estimate, but probably in the 80–90% range; Porter & Lemon, 1993). This adds up to a picture that argues against a discrete cortical neuron–muscle contraction linkage.

Physiology or function of the nervous system can be delineated during natural behaviour. A famous proponent of this viewpoint was the Russian psychophysicist N. A. Bernstein, who in the '30s developed an optical device that allowed him to measure natural movements in three dimensions (Bernstein, 1967). He studied trajectories – the time series of positions that describe a movement – and found a number of consistent rules. One of these findings is rudimentary to this discussion. When performing cyclic movements, subjects have a great deal of variability in the middle of the trajectory, but this narrows at the end-point. For instance, when hammering a nail, the path of the hammer differs on every stroke, yet the hammer hits the nail consistently. Bernstein used this finding as leverage against locationist dogma. Imagine that cortical neurons light up like bulbs when they are active. He argued that if each neuron corresponded to activation of a muscle, then there would appear to be a different zig-zag pattern appearing in the cortex for each hammer stroke. The central argument he put forth is that muscles must act to generate force that moves the body in context of the environment. The environment is constantly changing, so that in order to make the same movement, different muscle activation patterns would be needed for each repetition. If these repeated movements are considered as ‘conditioned reflexes’, then either the conditioned reflexes cannot be localized or muscles cannot be localized, since the relation between them changes upon each repetition (Movie 2), ‘…in other words, we arrive at the conclusion that the hypothesis of cellular localization of muscles necessarily leads to a denial of cellular localization of conditioned reflexes. One of the two chess pieces must be taken, and it is here a very pertinent question which of the two the old-fashioned localizationalist would rather kill.’ This is a good example of how psychophysical arguments based on dynamic variables can be used to guide a general approach to research on motor control.

Physiology as a science is derived from anatomy. By definition, anatomy deals with stationary observations, in that fixed tissue doesn't change. In fact, most studies of the physiology of motor control explicitly remove time as an experimental variable. Except for notable examples such as those from Jackson, Denny-Brown and Sherrington, most motor physiology experiments until the 1970s used decerebrate, deeply anaesthetized and/or neuromusuclarly paralyzed preparations. Even today, most neurophysiology experiments are carried out in pseudostatic conditions so that time can be considered constant. Movement is either absent or restricted.

Taking a historical survey of motor physiology (Fig. 2), there have been reports since the Hippocratic era of the contralateral relation between cortical injury and the observed deficit (Walker, 1957). Through most of recorded history until the 19th century, the tools used to observe cortical function consisted of sticking, cutting, burning, pinching or removing the tissue. With the advent of the galvanometer and reproducible electrical stimulation, it became possible to record from and stimulate the cortex, which led by the beginning of the 20th century to the idea of localized motor function. At this time, Sherrington was publishing his discoveries of spinal reflexes and building a theory of motor behaviour based on successively longer reflex loops. It should be emphasized that Sherrington's work was based on the observation of motor behaviour resulting from ablation and was akin to clinical observations of disease and injury-related neural lesions. His synthesis of reflex action and neural transmission led to the idea of a nervous system comprised of fundamental circuits that were elaborated as the neural axis was traversed from the spinal cord to the cortex. Most of Sherrington's insight and legacy came from careful observation of behaviour, a neurophysiological approach that was almost extinguished in the decades that followed.

Figure 2.

Historical description of neurophysiological investigation Interpretation presented using boxology.

In the 1950s and '60s, single-unit recordings with microelectrodes became well established. In order to maintain intra- and extracellular recordings of this neuronal activity, microelectrodes had to be placed critically within microns of their targets and this required, at least initially, the preparation to be absolutely still. While many important observations could be made related to the mechanisms of neural transduction and transmission, the relation of neural activity to behaviour was set aside. The most general attempt to relate this work to behaviour was carried out with electroanatomical approaches, in which responses driven by electrical stimuli were used to describe motor ‘circuits’. Through the late '60s and '70s, animal preparations that could move were combined with the single-unit approach. A group of Russians (Shik et al. 1969) developed a decerebrate locomoting cat preparation that could walk on a treadmill while single-unit activity was recorded from a variety of motor structures. Evarts (1968) developed a chronic monkey experiment, in which a skull-mounted chamber was used to access the animal's motor cortex with microelectrodes on a daily basis. These primates were completely awake and trained to make simple wrist movements as single units were recorded in primary motor cortex. Initially, this work suggested that motor cortical activity mimicked that of the muscles during this simple task, but these observations soon gave way to more elaborate experiments suggesting that other movement parameters were represented in this activity (Humphrey, 1972; Humphrey & Corrie, 1978). Work by Georgopoulos with monkeys making whole-arm reaching movements (Georgopoulos et al. 1982) showed that movement direction was well represented in motor cortical activity. The relation between movement direction is simple, but the relation between neural activity and movement direction, at least on first principles, is not. It should be noted, however, that there have been few attempts to characterize the widespread muscle activity that occurs during volitional arm movement, and those that have show that the activity is quite complex (Flanders, 1991; Flanders & Herrmann, 1992). The time-varying patterns of muscle activity used to generate the torques to move the limbs during volitional movements cannot be chosen in a simple way with deterministic equations, because many combinations of muscle activity lead to the same torque. Therefore, there is no easy way to reconcile the simple and robust representation of movement direction with an equally straightforward cortical representation of muscle activity. Those arguing for muscle activation are hampered by a paucity of data and the complexities of a redundant, ill-posed mechanical system, while those of us celebrating the success of a robust, simple description of kinematics found in neural activity should temper our enthusiasm with the knowledge that movement is effected with muscles.

More recently, much of neuroscience has focused on molecular and pharmacological methods using reduced preparations – experiments carried out with brain slices and cell culture. The experiments that are centred on behaviour, for instance cognitive neuroscience studies using brain imaging, are focused once again on which anatomical structures are responsible for a given aspect of behaviour. So history again repeats itself by bringing reductionism and locationism to the forefront. Admittedly, this historical snapshot is but one interpretation. By summarizing this description, I am drawing arbitrary boundaries with a sampling of statements representing one viewpoint. The labels of each box are ‘sound bites’ that could arguably be considered to be taken out of context to emphasize a particular point of view. By organizing these categories, or by making boxes, I am participating in what could be called ‘boxology’. To be more fair, I could point out that much of our understanding of the brain is based on anatomy and localization of function (for example, most of neurology). But I would argue that we know very little of the basic operations of the brain that lead to behaviour generation. To gain this knowledge, we must move beyond concepts based on static, discrete categorization.

Where do the arguments come from?

Another example of categorization or boxology is the common flow diagram used to merge physiology with anatomy. A great illustration of this is Van Essen's diagram of the visual pathways from the lateral geniculate nucleus (LGN) to centres of ‘higher processing’ such as the hippocampus (Felleman & Van Essen, 1991; Fig. 3). Based primarily on anatomical considerations, this diagram is quite complex. What can one take away from such an illustration? Compare the detailed structure on the right side of the figure (visual system) to the left side (the rest of the brain). Clearly this interpretation is biased, in that choices are made when categorizing a myriad of data to make the facts fit within a set of boundaries.

Figure 3.

Circuit diagram showing hierarchical organization of brain areas, arranged according to anatomical and receptive field properties Right side shows nodes of the visual system. Left side is for the somatosensory system. Modified from Van Essen, Anderson & Feldman (1991 used with permission).

Another box example comes from engineering. A typical engineering exercise is to describe the input and output of a system with the goal of writing an equation describing the derivation of the latter from the former. This is implicit in many descriptions of the motor system. A set of examples in Fig. 4 demonstrates this idea.

Figure 4.

Engineering boxology Top row shows a co-ordinate transformation scheme in which the target is sensed visually and the goal is represented in successive co-ordinate frames. Middle row shows inverse dynamics starting from a trajectory and ending with muscle force. Bottom row shows the anatomical schematic of output action.

The consideration of movement as a time-varying process is fundamental in physics and engineering. Co-ordinate-system transformations describe the positions of objects and the way they move through space by relating these parameters to different frameworks or perspectives (Pellionisz & Llinas, 1985; Helms Tillery et al. 1991). An example of this is outlined in the top row of Fig. 4. To move to a point in space or to trace a shape, the target or object is seen. Vision begins with registration of an image, in this case a target, on the retina. The location of the visualized goal can be represented as a two-dimensional position on the retinal sheet – a ‘retinotopic’ co-ordinate frame. This position may then be transformed to a head-centred co-ordinate system or the relative three-dimensional (3-D) position of the head to the target (the third dimension is estimated from stereo-disparity on the two retinal sheets, and other depth cues). The head-centred co-ordinates may then be transformed to a shoulder- or body-centred system (target relative to shoulder) and finally to a hand-centred system in which the control system moves the hand to the target with a velocity signal that can be thought of as a time series of points along a hand trajectory. The input and output (position in a co-ordinate frame) to each transformation is completely specified, and the straightforward procedure to carry out each transformation uses routine matrix algebra.

Another approach to describe motor control comes from robotics. The normal causal chain in mechanical systems is for an applied force to generate displacement. The reverse, in which a kinematic path is observed and the underlying forces calculated, is called ‘inverse dynamics’ (Fig. 4, second row). This has been considered for the multiple segments, joints and actuators of the arm (Hollerbach, 1982). In skeletal mechanics, two major categories of parameters are used: kinematics, consisting of positions and their temporal derivatives; and dynamics, consisting of forces and their derivatives. Consider the simple wrist flexion and extension experiment of Evarts (1968). The kinematics consisted of a single degree-of-freedom (DOF) at the wrist and the dynamics could be idealized as a pair of opposing springs. Force and displacement could be approximated with a simple linear equation, F =kx, where F is the force in the spring, x is the spring length and k is the spring constant. This is idealized as a single, pure antagonistic muscle pair used in the task. But in the real world, there are a number of muscles acting in this task, some of which span several joints, generating torque contributions that overlap each other. So even in this simple task, the inverse dynamics are complex. Imagine now, a two-dimensional trajectory recorded as the path of the hand evolves in time when moving on a table top with the elbow and shoulder in the same plane as the table top. Once again, the kinematics are fairly simple because there is only one DOF at the elbow and one at the shoulder; together their angles specify the wrist displacement uniquely. However, the inverse dynamics become even more complex, not only because of muscle redundancy but because motion around one DOF generates torque about the other (interaction forces). Finally, for natural arm movements, the kinematics are complex because each 3-D displacement of the wrist can be generated by many different joint combinations (joint redundancy). The corresponding inverse dynamics are very complex. Although it has a lot of pitfalls, this procedure can be carried out precisely, especially if physiological realism is not deemed essential. The typical scheme is to start with a trajectory, calculate a set of joint angles to get each position along the trajectory, choose a set of torques constrained by skeletal geometry and the laws of motion, and then find a set of muscle activations to get the muscle forces required for the chosen joint torques.

Another way to describe movement generation is with a conventional physiological/anatomical block diagram (Fig. 4, bottom row). This might start with a trajectory of the desired hand path in the motor cortex with cortical output projecting successively to the spinal cord and to muscles to generate displacement. At least from the anatomical perspective, this is a fairly well accepted point of view.

These three examples describe different approaches taken towards understanding the processes that underlie volitional movement. Each approach, in and of itself, is valid. The problem comes when one tries to fit a box from one row vertically into that of a different row. For instance, whether trajectory is first represented in motor cortex and then the next steps through the inverse chain are represented in ‘downstream’ anatomical structures is controversial (Kalaska & Crammond, 1992). These boxes represent models and as such are limited. Although it is likely that kinematic representations are transformed to a set of muscle contractions, it is unlikely that the steps to accomplish this are done discretely in neat boxes or modules. Nor is it likely that each transformation is confined to an individual anatomical structure.

In an attempt to describe a large body of work, I have formed a summary by placing these details in categories. Each detail can be interpreted from a number of viewpoints, so by placing boundaries around these categories, I am making an interpretation. The placement of these boundaries creates arguments. It is comforting to describe a system by its pieces, because of our experience with machines. No matter how complex the machine, it can always be broken into components. Historically, this machine analogy has been applied to the CNS, but at this point it is important to ask how this viewpoint has limited our understanding of neurophysiology. Adoption of a non-modular approach would free us from making arbitrary boundaries between brain structures and would emphasize the similarities between individual neurons and between anatomically identified structures. We have started down this road with the development of population analyses and the observations of the robust broad tuning of neural discharge to behavioural parameters. However, conventional input–output relations with clear-cut cause and effect descriptions will probably be a casualty of this new approach.

New ideas

Leading the way to this new consideration of the brain has been the development of the behaving animal model in which neural activity is recorded as cued response (movement) tasks are carried out. Viewing behaviour in the context of boxology (Fig. 5) illustrates a pitfall of the modular approach. A simple idea of behaviour is to consider the brain as a box with its input consisting of sensory stimuli and its output as movement. In this example, the brain generates behaviour. However, it is important to realize that movement changes the environment and generates sensation. Movement generates feedback, making this a closed-loop system. When looking at the brain as a black box, it becomes difficult to separate sensory- from motor-related neural activity. If we were to go back to the brain and puppet cartoon (Movie 3), we would see light bulbs sparkling over a wide range of the brain for a volitional movement. Behavioural models force us to confront the issue of causality in such a system.

Figure 5.

Schematic diagram of brain and behaviour Closed-loop feedback generated from action that is sensed.

How can we hope to interpret this complex pattern of activity? This interpretation is often referred to as ‘the decoding problem’. I like to use examples from the somatosensory system to illustrate this concept. Beginning in the '60s, Vernon Mountcastle carried out a set of experiments at Johns Hopkins University to describe the neural processing underlying vibrotactile sensation (Mountcastle, 1984). Recording action potentials from single afferent fibres of slowly adapting mechanoreceptors in the glaborous skin of the monkey finger, he showed that the firing rate of these receptors was linearly related to the depth of skin indentation (Figs 6 and 7, middle panel). Using the same stimuli, he also found a linear relation for neurons in the primary sensory cortex (Fig. 7, lower panel) and in human psychometric estimates (Fig. 7, top panel). It could be concluded that a simple linear relation between stimulus intensity and response describes the process all the way from initial mechanical encoding to verbal report. This is an example of a linear decoding scheme, in that the extraction algorithm used to predict the stimulus from the recorded neural signal is a linear equation. In fact, Ken Johnson, also at Johns Hopkins University, used this finding to demonstrate isomorphic encoding. He and John Phillips scanned embossed letters across the receptive field of a mechanoreceptor of the monkey's finger. The letters were shifted vertically in small increments in each pass over the receptive field. Single units were recorded in primary sensory cortex and action potential occurrences were aligned to the onset of each pass across the receptive field to construct a ‘spatial event plot’ (SEP; Phillips et al. 1988); each row of the SEP contains one pass. In the resulting plots, the images of the letters are clearly evident (Figs 8 and 9). The tendency for linear encoding of the stimulus throughout the system makes it simple to recover a robust, isomorphic representation of the stimulus.

Figure 6.

Spike train recorded from mechanoreceptor afferent fibre Ordered up-to-down, left-to-right as successively greater displacement is imparted to the receptive field. From Mountcastle (1984), used with permission.

Figure 7.

Linear encoding of mechanical stiumuli Plots of response versus intensity of mechanical stimulation. A shows responses from humans of estimated intensity of peripheral stimulus. B shows firing rate intensity of primary afferent in response to skin indentation. C shows post-central response to skin indentation. From Mountcastle (1984), used with permission.

Figure 8.

Spatial event plot (from Phillips et al. 1988) An embossed drum was used to scan letters over the receptive field of a mechanoreceptor in a monkey's finger. Action potentials recorded from a single unit in Area 3b are registered on the plot as tick marks. The beginning of each row is aligned to the beginning of the scan and different rows correspond to successive vertical shifts of the drum over the finger. This method shows that an isomorphic replica of the letter ‘K’ can be recovered from the neural activity.

Figure 9.

Different examples from Area 3b The method in Fig. 8 was used across a range of letters and different units.

In the early '80s Apostolos Georgopoulos, who was a member of the Bard Laboratories at Johns Hopkins University along with Mountcastle and Johnson, began to study the motor cortex. At that time the work of Evarts dominated the field. Georgopoulos designed a ‘centre-out’ task in which monkeys would reach in the different directions from a centre start position (Fig. 10). As monkeys made movements to targets in eight directions, single units were recorded in motor cortex. Remember that at the time, this was a bit heretical, since Evarts's work suggested that motor cortical discharge was related to single-joint movements with an implication that this cortical region is regulating simple muscle contraction patterns. The idea of finding a meaningful representation between neural discharge and a relatively unconstrained whole arm movement seemed unlikely, because the pattern of muscle activity underlying natural movement was thought to be complex, obviating (supposedly) a clear relation to movement direction. That more holistic aspects of arm movement might be contained in motor cortical discharge came from Georgopoulos's experience of recording in posterior parietal cortex as monkeys made arm movements to targets in extrapersonal space (Mountcastle et al. 1975). In this experiment, carried out by Mountcastle and colleagues in the early '70s, it was found that these neurons were responsive to arm movements directed away from the body to targets in the periphery. Georgopoulos noticed that these neurons that would fire vigorously with arm projection would stop firing when returning in the opposite direction, towards the start position, hinting at the directional nature of the response. These neurons were much more active when the movement was made purposively in the context of receiving a desired reward then when the same movement was made inadverdently, suggesting that these were not purely motor related. When demonstrating these experimental techniques to medical students, Mountcastle and Georgopoulos recorded motor cortical activity, since it was expected that these neurons would fire reliably with arm movements (A. P. Georgopoulos, personal communciation). They were surprised to see how similar these responses were to those in posterior parietal cortex. For instance, a motor cortical neuron that would fire vigorously while the monkey was reaching for a piece of food would not fire at all if the monkey was scratching himself. Also, in the early '80s it was becoming clear that reaching movements were being generated as whole behaviours with characteristic invariant features (Soechting & Lacquaniti, 1981; Morasso, 1981). Wanting to explore the possibility that general, behavioural features of reaching were represented in motor cortical activity, Georgopoulos designed the centre-out task based on a similar experimental design used originally by Paul Fitts (Fitts & Denninger, 1954). The first results of this experiment (Georgopoulos et al. 1982) showed that the discharge rate (mean calculated over the reaction and movement time) of motor cortical cells was directly related to the cosine of movement direction (Fig. 11). The peak of the cosine function is the movement direction with the highest discharge rate and was termed the ‘preferred direction’. Each tuned cell had a different preferred direction and there was a tendency for preferred directions to be uniformly distributed across motor cortical neurons. Since the tuning function spans all directions, a tuned neuron changes its activity for every direction. Conversely, for any given movement direction, all tuned cells respond together.

Figure 10.

Original centre-out model A monkey grasped the handle (‘C’) of a draftsman's arm and moved the sighting glass (‘D’) over a sequence of target lights embedded in the underlying table top. The sequence would begin with the capture of the centre target. This target would then be extinguished as one of the eight peripheral targets was illuminated, signalling the monkey to move towards and capture that target. Reprinted with permission from Georgopoulos et al. (1982), copyright 1982 by the Society of Neuroscience.

Figure 11.

Tuned directional response This is an example of a unit response recorded during the centre-out task. Top panel shows rasters arranged by movement direction. Each row in the raster is a trial aligned to movement onset. The unit fired intensely for upper (90 deg) movements and slowed for the opposite downward movement. Firing intensity was graded for movements between these extremes. Bottom panel shows cosine tuning of cortical response. The data from the top panel are redisplayed as mean firing rates in each direction and these are fitted with a cosine. The peak of the cosine is near 90 deg and is termed the ‘preferred direction’. Reprinted with permission from Georgopoulos et al. (1982), copyright 1982 by the Society of Neuroscience.

Can single units be used for decoding? Not very well! First, note that the code is ambiguous. Every discharge rate (except in the preferred direction) corresponds to two movement directions. Taking into account the intertrial variability (the vertical line at each data point on the tuning curve represents the standard deviation across repetitions), one would only be able to guess whether the impending movement would be towards or away from the preferred direction. This can be addressed by considering two cells at the same time (Fig. 12). Now there will be a unique pair of discharge rates (one from each curve) that matches a single movement direction. The success of this method will depend on how well the data fit the cosine function, but visual inspection indicates that as more neurons are added to the analysis, the determination of direction becomes more accurate.

Figure 12.

Decoding considerations Using a single tuning function to generate predicted movement direction does not work well because there are two directions for each discharge rate. This can be addressed by adding more units with different preferred directions and this combination will lead to a better solution (see text).

This was formalized by Georgopoulos with the population vector algorithm (PVA; Georgopoulos et al. 1983). In the initial version of this method, the contribution of each neuron in the recorded population was represented as a vector orientated in the preferred direction of the neuron. The length of the cell vector was weighted by the mean discharge rate of the neuron during the single movement (reach to a particular target) that was being modelled. The vectors all have a common origin and form a focused cluster (Fig. 13) for a movement. The next step is to add the vectors together and divide by the number of cells. Vector addition is simple; the x and then the y components from all the cell vectors are summed. The resulting x and y pair are the co-ordinates for the end-point of the resultant, or ‘population’, vector. The vector is a prediction of movement direction and was found to point accurately to each target in the centre-out task (Georgopoulos et al. 1983).

Figure 13.

Population vector algorithm The response of each unit in a population during movements to each target is represented by a vector in that unit's preferred direction weighted (length) by the unit's firing rate during the movement. The vectors in each cluster are summed to give a resultant population vector (dashed arrow), which is predictive of movement direction. From Georgopoulos et al. (1983), with permission.

A continuous control process generates volitional movement. One problem with using reaching movements to study directional control is that these movements are fairly straight, so that direction is approximately constant within the task. To study different directions within the same movement, I trained monkeys to make drawings with their finger tips. They traced a variety of shapes, such as sinusoids, spirals and figures of eight. After a single motor-cortical unit was isolated, the animal would first perform a centre-out task so that the unit's preferred direction could be identified. The monkey would then draw the projected shapes while isolation of the same unit was maintained. After recording several hundred units, I constructed population vectors. However, instead of calculating a single population vector for the entire movement, I calculated 100 vectors evenly spaced throughout the time course of the movement (Fig. 14 and Movie 4; Schwartz, 1994). Each successive population vector was placed at the end of the previous vector. Not only does the direction of the population vector match the instantaneous movement direction, but its length matches the corresponding speed of the movement. This means that the population vector corresponds to movement velocity (Moran & Schwartz, 1999) and is why the movement trajectory is recovered by arranging the time series of population vectors tip-to-tail. This arrangement is an integration, and in this case the integration of vectors corresponding to velocity gives the trajectory or path of the hand. One point of caution, however, is that even though population vectors represent trajectories very well, population vectors themselves do not exist. It is unlikely that there is a place in the brain where signals from many cortical neurons are gathered up to form a vector sum.

Figure 14.

Neural trajectory Population vectors are made every 10–20 ms throughout a single drawing task. Monkeys used their fingers to trace a template on a touch monitor. The blue cluster is composed of preferred direction vectors, one from each cell recorded population. The population vectors (yellow example) in the sequence are connected tip-to-tail to construct a neural representation of the trajectory, which in this case is a spiral. This figure corresponds to Movie 4.

So, in a way, this result is analogous to the isomorphic letters that Ken Johnson extracted from sensory cortex; we are extracting isomorphic trajectories from motor cortex. Although we could spend a lot of time arguing about what motor cortex ‘does’ at this point, this would be of little value. As outlined earlier, there has been a long-standing controversy as to whether movements or muscles are represented in the motor cortex. One could take our results and argue that kinematics are represented in the motor cortex. Interestingly, others have taken our data to argue exactly the opposite (Todorov, 2000). This is another version of boxology, in which attempts are made to assign a specific function to an anatomical structure. Rather than revisit this argument, we ask how these findings could be used to learn more about the control of volitional movement.

Using the new approach for system insight: physiological results

Over the years, we have developed a behavioural model in which monkeys work in a virtual reality environment (VR). A monkey, seated in a restraint chair, views a stereo monitor through a periscope Fig. 15. Its arm is tracked with a 3-D motion tracker and the position of the hand is represented as a spherical cursor in the display. Different shapes are presented on the monitor; these appear as tubular objects, and the animal is trained to place its hand in the object to trace the shape while keeping contact with the tube (Fig. 16). An example of the animal performing one of these tasks while a unit is recorded is shown in Movies 5 and 6. The animals make smooth, graceful movements in free space. From the movie, it can be seen that the unit increases its firing rate in a particular quadrant of the ellipse and shuts off in the opposite quadrant. This is a good example of the directional characteristics of a motor cortical neuron.

Figure 15.

Virtual reality model Monkeys viewed images projected in stereo through a periscope. The resulting 3-D image appeared to be floating in front of them. Their hand position was tracked in 3-D and this was used to move a cursor ball in real time. The animals were trained to place the cursor in the tubular object and trace it.

Figure 16.

The projected object and cursor from the monkey's perspective

Given that we have this kind of experimental model and the ability to extract isomorphic representations of the drawn object, how can we use this to learn more about movement processing? We looked into the net behaviour taking place in this kind of task and realized that vision was a major component. The monkeys were drawing a virtual object that they could only visualize and could not feel. The possible route and anatomical structures that visual information may traverse on the way to motor cortex are summarized in Fig. 17. Normally, vision of the movement corresponds completely to the execution of the movement. If you draw a circle, you will see your hand moving in a circular path. To separate the visual or perceived path from the executed path, we created a motor illusion. Subjects (both humans and animals were used in this experiment) normally performed the ellipse drawing task by moving their wrist cursor five times around the ellipse template. To create the illusion, we gradually increased the horizontal gain of the cursor through cycles 2–4. The final gain was 1.8, which was the eccentricity of the ellipse (Fig. 18). To keep the cursor within the visualized template, the subjects had to shorten the horizontal portions of their movement and ended up drawing circles. Since the gain change was gradual and the cursor continued to appear to be moving along the same oval, the subjects did not perceive the change in motion. With our trajectory read-out technique, we can sample neural activity from two locations along the visual pathway and test the corresponding neural trajectories from each for a match to the perceived or the real movement (Fig. 19). For this comparison, we chose the ventral premotor cortex and the primary cortex, two highly connected regions. In the beginning of the task, before the illusion took place, the neural trajectories in both structures looked identical, in that both sets of neurons were cosine tuned, with preferred directions and neural activity that could be used to create accurate neural trajectories. However, as the illusion was invoked and fully in place in cycle 5, the motor cortical neural trajectory matched the hand path, while the premotor neural trajectory followed that of the cursor path reflecting the visualized, perceived path. Thus two adjacent interconnected cortical areas can have movement representations that coincide in most instances but differ in others (Schwartz et al. 2004). This work also showed that perceived and real movement may be represented separately in different cortical areas.

Figure 17.

Schematic of brain structures between primary visual and motor cortex MT, middle temporal; MST, medial superior temporal; VIP, ventral intraparietal; LIP, lateral intraparietal; Cing, cingulate; PMv, ventral premotor; PMd, dorsal premotor; M1, primary motor; AIT, anterior inferotemporal temporal; PN, pontine nuclei; Cb, cerebellum; SMA, supplementary motor; pSMA, presupplementary motor; and VPLo, ventral posterior lateral oralis (thalamus).

Figure 18.

Motor illusion task Subjects drew five oval cycles in virtual reality. The top row shows what the hand did, starting from the first cycle, which uses normal gain, and proceeding through the subsequent cycles as the horizontal gain is increased to 1.8. The subject's hand trajectory became more circular throughout the task. The bottom row shows that the movement in the visual display of the cursor in the oval template continued to appear the same through the task.

Figure 19.

Results of the illusion task Top row shows data collected from the primary motor cortex. The red trace represents the neural trajectory from M1, blue is the hand trajectory and green is the cursor path. The gain is changed in cycles 3–4, and both the neural and hand trajectories become circular. Bottom row shows neural trajectories from the ventral premotor cortex. These data show that the neural trajectory matches the cursor path instead of the hand trajectory. From Schwartz et al. (2004).

The new approach applied to psychophysics

Psychophysicists have made many contributions to our understanding of movement generation. By identifying ‘invariants’ or rules of movement, they have described constraints that reflect features of system operation. Championed by Helmholtz in the late 1800s, psychophysics was originally used to map physical stimuli to perception. Motor psychophysics seeks to describe the decision-making process underlying behavioural action. For every volitional movement, information about the goal of the movement, the speed and path to be taken, which joints to move, which muscles to contract, etc., needs to be transmitted throughout the motor system. In spite of all these choices, movements are made in a stereotypical fashion. Psychophysical results describe these consistent characteristics.

Psychophysical laws have been described for drawing movements. As we draw closed shapes such as figures of eight, we have the feeling that we are making them with a continuous movement. However, kinematic inspection shows that the hand slows down for each curve and speeds up on the straight portions. A figure of eight can be divided into four segments with boundaries at the vertex where speed is maximal. This is referred to as segmentation (Viviani & Cenzato, 1985). We trained monkeys to make figures of eight and found that they obeyed the segmentation rule (Fig. 20). Neural trajectories made from cortical recordings during these tasks had the same segmentation, showing that the rule is contained in motor cortical activity (Schwartz & Moran, 1999). A related rule, the ‘⅔ power law’ is an elaboration of the general finding that we move slower in curves. Originally described in a handwriting study (Lacquaniti et al. 1983), in which it was found that angular velocity of the hand is proportional to the radius of curvature to the ⅔ power, the law has since been shown to be a general descriptor of animate movements, to have a perceptual equivalent (Viviani & Stucchi, 1989) and to form during maturation (Viviani & Schneider, 1991). Plots of angular velocity and curvature (Fig. 21) show that monkeys in the figure of eight experiments obeyed the ⅔ power law and that this was represented at the motor cortical level, indicating that this rule is incorporated in motor cortical processing.

Figure 20.

Segmentation during drawing Monkeys were trained to draw four different figures of eight. Their finger trajectories (right column) were divided into four segments at points of maximum velocity. The same segmentation was found in the neural trajectories (left column). From Schwartz & Moran (1999), used with permission.

Figure 21.

Power law When data from the figure of eight experiment were plotted as angular velocity versus curvature^−2/3, the relation was linear. This was true for both hand (top) and neural trajectories (bottom). From Schwartz & Moran (1999), used with permission.

Monitoring eye position can provide insight into the cognitive processing taking place during drawing. The point on the figure upon which the eyes are fixed gives an indication as to where the subject is attending. One might expect that during these tracing tasks, the eyes would pursue the hand as it moves around the template. However, this is not the case. The eyes tend to saccade between the corners or curved regions of the ellipse, in such a way as to catch the hand (image of the moving cursor on the fovea) as it passes through the curve (Reina & Schwartz, 2003; Fig. 22 and 23). This is consistent with the monkeys' performance, since they almost always make their mistakes (break contact with the template) in the curved portion of the template, which explains why these regions are attended to preferentially.

Figure 22.

Density plot of eye position during drawing The eyes tended to stabilize in either corner of the ellipse. The top of the color bar represents high probability.

Figure 23.

Density plot of the hand during ellipse drawing The hand slowed, spending more time, in the curves. The top of the color bar represents high probability.

Now we can weave a consistent story. Movements are made in segments. The hand slows precisely when traversing curves. These two invariants are represented in the cortex. Behaviourally, going straight is easy; mistakes are made in the curves, so the curves are attended to preferentially. In terms of information, few bits need to be transmitted when going straight. The command on successive time steps might be something like ‘keep doing what was done at the last time step.’ In the curve, more bits are needed to specify the directional changes. If the channel capacity is limited, it will take more time to transmit those bits, requiring the hand to slow down. This is consistent with a finite conduction time to accommodate increased processing overhead in the curves.

At this point, I hope I have convinced you that the motor control represented in motor cortex contains aspects of behaviour that range beyond the mechanical details of movement. The presence and details of these behavioural characteristics can give us insight into the process underlying volitional movement. From the population of active cells in the cortex, we can extract enough detail to predict the natural motion of the arm during a variety of different tasks. To substantiate this statement, we should be able to predict arm movement by recording a population of units, extract the movement signal and use it to move objects (arms). Incidentally, this is exactly what is needed to control a high-performance arm prosthesis.

Neural prosthetics based on populations

A key to successful, high-fidelity prosthetic control is the ability to record a population of neural activity. We have used a couple of different types of chronically implanted microelectrodes. As an example, the Michigan silicon probe is illustrated in Fig. 24. This is a planar silicon dagger with multiple recording contacts on each shank. Typically, we implant arrays of these probes, so that there are 96 recording sites in the monkey's motor cortex. Although this technology is still in its infancy, we can record 60–80 individual units with this arrangement. Preamplifiers are connected to the implanted arrays through skull-fixed microsockets during each experiment. The conditioned signal is processed by a multichannel spike sorter that discriminates action potentials from the recorded signals. The time stamps from each sorted spike are then used to find the tuning characteristics of the recorded unit. Once found, population vectors are calculated, typically at 30 ms intervals (Schwartz et al. 2006).

Figure 24.

Planar Michigan silicon array Photograph of a single multiprobe with four shanks, each containing four recording electrodes, displayed on the back of a penny. Courtesy of Dr Daryl Kipke, University of Michigan.

One of the earliest trials of this technology was to operate a robot arm in parallel with the monkey performing a 3-D centre-out task (reaching from the middle of a cube to push-buttons at the corners) with its own arm. The robot arm was in a separate room and the monkey had no knowledge of it. We refer to this as ‘open-loop’ control, since there was no feedback of the decoded movement to the animal. In this example from 1998 (Movie 8), the control, although rudimentary, moved the robot arm in approximately the same direction as the monkey's arm.

The next step was to have the animal work the 3-D centre-out task in VR. An infrared marker on the monkey's hand was tracked with a motion-capture system (Northern Digital Inc., Waterloo, Canada) and the wrist position was used to move a 3-D cursor in the stereo display. Initially, the animal performed the usual ‘hand-control’ centre-out task. After hitting a few of these targets, the cursor control was switched to the population vector algorithm output, which generated an updated output position approximately every 30 ms. The transition between hand- and brain-control is shown in Movie 9 as an increase in the target size. Initially, the animal moved both its arm and the cursor to the target as in hand control, but this was not always the case. Immediately after the switch to brain-control, the animal was fairly successful in hitting the target. If the algorithm perfectly extracted the intended movement, it could be reasoned that the animal would not distinguish between hand- and brain-control and would continue (unnecessarily) moving its own arm during brain-control.

Next, the brain-control state was made explicit by restraining both of the monkey's arms. After a day or two, when the animal was comfortable in the task, there was no detectable EMG in its arm during brain control. This experiment is displayed (Movie 9) as seen from the animal's perspective, but the movie is not in stereo. Target positions are three-dimensional, however, and are placed in and out of the plane of the display screen as well as left–right and up–down. The animal was able to move very well to these targets, moving to novel target locations successfully on the first try and moving from the centre-out and back to the centre again. The monkey could control its speed and move the cursor precisely to acquire the target. Performance was around 90% and the animal would perform this task for about an hour at a time (Taylor et al. 2002). In this work we found that the preferred directions of the recorded units would shift when switching between hand and brain control. Even though the shift-angle across units appeared to be random, the shift of an individual unit was consistent across days. Learning was an important aspect in the amount of success we saw in this demonstration. The algorithm was designed to be adaptive in calculating preferred directions. In the initial trials of brain-control, the estimates of preferred directions were iterated a number of times. In conjunction with these re-estimates, the animal's performance improved. This ‘coadaptation’ is a useful way to calibrate brain-controlled devices. Another feature of learning was the increased performance over days. Corresponding to a better fit of the cosine tuning function to the firing rate data, this learning led to better predictions from the PVA. Since the monkey could directly visualize the PVA output as cursor movement, it was able to modify its neural activity to achieve better performance.

The power of this learning is illustrated in Figs 25 and 26. The consistency of the population vectors from the original 3-D centre-out experiments performed in the Georgopoulos laboratory was evaluated with a boot-strap procedure to produce a 95% confidence angle around the population vector (Georgopoulos et al. 1988). With few cells in the population, the confidence angle is large (Fig. 26), but decreases exponentially as more cells are included in the population. The shoulder of the curve is around 100–150 units, at which point the confidence interval is about 25 deg. With 475 cells, the confidence interval was about 17.5 deg. When we began our prosthesis experiments, we looked at this curve and thought that it would take approximately 100 units recorded simultaneously to achieve good performance. When we performed these experiments, however, we found that we could achieve good performance with far fewer units. This result can be ascribed to the difference in using open- versus closed-loop control. In the recent closed-loop model, the monkeys were able to modify the discharge pattern of their recorded units to better fit the tuning functions assigned to them.

Figure 25.

Confidence interval of 3-D population vector A bootstrap technique was used to repeatedly calculate a population vector from its individual constituents. The results were rank ordered and represented as a 95% confidence interval around the mean population vector. This is represented by the radius of the red cone. The green vector is the movement vector for this 3-D centre-out example.

Figure 26.

Plot of confidence intervalversusnumber of cells included in the population Using the method shown in Fig. 25, 95% confidence intervals (radius in degrees) were calculated for populations of different sizes in the 3-D centre-out task. The line with the boxes is for data calculated in a hand-control task (Georgopoulos et al. 1988). The continuous line is from the same task performed with brain-control.

While operating computer displays for communication would help one segment of the patient population (Kubler et al. 2005), using brain-control to operate a real physical arm would expand the efficacy of neural prosthetics to enhance the quality of life for many more immobile individuals. To accomplish this, we were faced with a number of challenges. Since physical devices are subject to the laws of motion and construction limitations, they do not move as well as their virtual counterparts. In addition, training a monkey to operate a robot arm as a tool is difficult. With a human subject, the instruction could be given: ‘Imagine that this device is your own arm.’ No such instruction can be given to a monkey. Our first experiments with a robot arm were open-loop (see above). In later experiments, we partly closed this loop using a hybrid computer display–robot arm model (Movie 11). The robot arm was in a separate room from the monkey. The position of the robot's wrist was tracked with an Optotrak marker in 3-D space and the wrist position was used to move the 3-D cursor on the display in front of the monkey. The task was the same 3-D centre-out task that the monkey used in the brain-controlled VR experiment, but this time the cursor did not move as smoothly since the robot motion was imperfect. This particular arm suffered from sticktion and ataxia in particular movement directions. In spite of these effector problems, the animal was able to compensate for the visualized perturbation in cursor movement and use its cortical activity to acquire the targets successfully.

We had difficulties in our first attempts to get a monkey to interact directly with a robot arm. The animal was motivated by having the arm deliver food to its mouth. At first, the animal was frightened by the big, strange machine coming towards its face. However, we continued to have problems even after the animal was comfortable with the machine. We secured an orange slice to the end of the robot arm with a clothespin and held it a few inches from the animal's face. The idea was for the animal to use brain-control to move the orange to its mouth. Upon seeing the orange so close to its face, the monkey tried to reach it by projecting its head forwards. This was picked up in the neural recordings as a movement away from the body, which moved the device (with the orange) away from the animal, leading to great frustration. This is a good example of co-ordinate transformation gone awry!

By patiently presenting the food to the hungry monkey and by directing its robot arm towards the mouth, the monkey eventually got the idea and was able, in this one-dimensional task, to move the orange to its mouth (Movie 12).

Recently, we were able to demonstrate a complete self-feeding task using an anthropomorphic prosthetic arm with four DOF. The monkey's arms were restrained, and the shoulder of the robot was mounted near the monkey's shoulder. Population vectors constructed from the recorded motor cortical activity updated the position of the robot arm every 30 ms. The firing rates were calculated from the 300 ms of spike data that preceded the update. In the first experiment, an investigator would hold a piece of vegetable at one of four locations. Brain-control would be engaged, and as the arm approached the target zone around the food, the gripper would open. Once the animal had guided the arm/gripper into the target zone, the device would automatically home to the centre of the space and the gripper would close around the food. Brain-control would then be re-engaged and the animal would return the food to the mouth target zone, whereupon the gripper would automatically move to the mouth and release the food (Movie 13). Trajectories from this experiment are shown in Figs 27 and 28. The next set of experiments used continuous brain-control. Here there was no automatic homing within a target zone. As the animal used the device to reach for food in arbitrary locations, the slowing of the arm would trigger gripper closing. The animal would then move the arm back to its mouth and hold the food in a location where it could be eaten. The gripper again was under state control, in that when slowing on the way towards the mouth, the gripper would open (Movie 14). Notice that although the arm is under brain-control as the animal is chewing, it remains in nearly the same location until the animal decides to go for the next piece of food.

Figure 27.

Robot trajectories during brain control Trajectories from the mouth to each of four food positions along with their 95% confidence intervals are shown for the initial self-feeding task. Once the gripper entered the target zone (solid balls), the arm moved automatically to the food. The diameter of each target ball is 3 cm.

Figure 28.

Robot trajectories towards the mouth Same explanation as for Fig. 26 for the opposite movements. The diameter of each target ball is 3 cm.

At this point I think we have shown that neural prosthetic arm control has the potential to make a positive impact for immobilized patients. We are now extending this work with a mobile wrist to allow hand orientation and an anthropomorphic set of fingers that will enable dexterous manipulation. An example of the next-generation artificial hand, the ‘ACT’ hand from Yoky Matsuoka, is demonstrated with a fully articulated index finger (Movie 15). As our work continues, we hope to characterize motor cortical activity that can be used to control wrist orientation, hand shape, and accurate finger placement and force generation. We believe that the acquisition of this level of control will provide a serious improvement in quality of life for those who have lost arm and hand function.

Supplementary material

The online version of this paper can be accessed at:

DOI: 10.1113/jphysiol.2006.126698

http://jp.physoc.org/cgi/content/full/jphysiol.2002006.126698/DC1 and contains supplemental material consisting of 15 movies.

This material can also be found as part of the full-text HTML version available from http://www.blackwell-synergy.com