Encoding of kinetic and kinematic movement parameters in the sensorimotor cortex: A Brain-Computer Interface perspective

Abstract

For severely paralyzed people, Brain-Computer Interfaces (BCIs) can potentially replace lost motor output and provide a brain-based control signal for augmentative and alternative communication devices or neuroprosthetics. Many BCIs focus on neuronal signals acquired from the hand area of the sensorimotor cortex, employing changes in the patterns of neuronal firing or spectral power associated with one or more types of hand movement. Hand and finger movement can be described by two groups of movement features, namely kinematics (spatial and motion aspects) and kinetics (muscles and forces). Despite extensive primate and human research, it is not fully understood how these features are represented in the SMC and how they lead to the appropriate movement. Yet, the available information may provide insight into which features are most suitable for BCI control. To that purpose, the current paper provides an in-depth review on the movement features encoded in the SMC. Even though there is no consensus on how exactly the SMC generates movement, we conclude that some parameters are well represented in the SMC and can be accurately used for BCI control with discrete as well as continuous feedback. However, the vast evidence also suggests that movement should be interpreted as a combination of multiple parameters rather than isolated ones, pleading for further exploration of sensorimotor control models for accurate BCI control.

Graphical Abstract

All kinematic and kinetic movement parameters are to some extent represented in the sensorimotor cortex.

Individual finger movement, movement direction, complex hand movements and movement trajectories can be best discriminated in the SMC and potentially used for BCI.

Understanding how SMC encodes movements is particularly relevant for finding the best neural control signal for brain computer interface applications.

1. Introduction

Damage to the sensorimotor system caused by either trauma, stroke or neuromuscular disorder may result in severe paralysis or loss of motor function (World Health Organization, 2011; Armour et al., 2016). In the worst case, people can lose control over all their voluntary movements and may become locked-in (Smith & Delargy, 2005; Posner et al., 2007). In that situation, the means of communication are exceptionally limited. In the last decades, Brain-Computer Interfaces (BCIs) have been presented as a muscle-independent tool to restore communication of locked-in individuals, which could, eventually, improve their quality of life (Anderson, 2004; Rousseau et al., 2015). BCIs replace the lost motor control by bypassing the muscles and directly linking the brain to a computer (Daly & Wolpaw, 2008). In general terms, BCIs record neuro-electrical or hemodynamic signals from the brain, extract features from the recorded signal, translate the features into a control signal for the actuator, and provide feedback to the user, ideally without significant delay (Wolpaw, 2007). For a BCI to be of practical value to the user, the brain signal features must be detected and extracted in a highly reliable manner. An attractive brain region for BCI feature extraction is the sensorimotor cortex (SMC, Brodmann area 1–4) (e.g., Pfurtscheller & Lopes da Silva, 1999; Yuan & He, 2014; Vansteensel et al., 2016). The sensorimotor cortex consists of two areas: the primary motor cortex (M1), classically related to movement preparation and execution, and the primary somatosensory cortex (S1), related to afferent processing of sensory information. An interesting feature of the SMC is its somatotopical organization (Penfield & Boldrey, 1937), which entails an orderly and detailed arrangement of body parts. In this review, we focus on the SMC as a whole (M1 and S1) as the use of both areas can be beneficial for improving accuracy of BCI control. Of special interest for BCI is the relatively large portion of the SMC, called the ‘hand knob’ (Yousry et al., 1997), that is devoted to the control of the hand and finger movements, and plays a role in arm movements as well. Due to its large representation in the SMC and the straightforward measurement and interpretation of movement-related signal changes from this area, the ‘hand knob’ is the most commonly used cortical area for BCI control and, therefore, the one we focus on this literature review.

Importantly, the remarkable spatially organized representation of movements in the SMC does not provide information about which mechanical or physical parameters of the voluntary movement are actually encoded in this brain area. Human movement can be described through kinematic parameters and kinetic parameters (Riehle & Vaadia, 2005; Jones & Lederman, 2006). Kinematic parameters, also called ‘high-level control’ refer to the spatial and motion aspects of movement. The basic kinematic variables include the position, velocity, acceleration and direction of the movement, and their combination into a complete trajectory. Conversely, kinetic parameters, also called ‘low-level control’, refer to the control of individual muscles and forces. In the last decades, many studies have investigated the representation of separate and/or combined kinematic and kinetic features of hand, finger and arm movement in the SMC. In this review, we aim to define which features may be most suitable for use in BCI settings. For that, we take a BCI perspective to summarize and highlight the most striking and consistent findings in the field of movement encoding. We focus on both non-human primate and human studies that used different methods to record neuro-electrical and hemodynamic brain signals, including intracortical needle recordings, intracranial electrocorticography (ECoG) and functional magnetic resonance imaging (fMRI).

2. Literature selection and characteristics

We selected literature from Pubmed and Google Scholar using combinations of the search terms listed in Table 1. From 558 screened titles and abstracts, only English-written articles that focused on decoding executed hand and/or finger movements from brain signals recorded by intracortical needle electrodes, ECoG or fMRI from the sensorimotor cortex were included, resulting in 95 included papers, of which four addressed both kinetic and kinematic parameters and two combined more than one technique (Table 1). Stimulation studies were not included in this review, with the exception of Penfield and Boldrey (1937). Of note, many of the studies on primates and needle recordings involve reaching and grasping movements, which includes movement of both hand and arm. In this literature review we included these studies as the results provide further insight into how the hand, finger and arm movements are controlled by the cortex.

Table 1 – Search terms used in this study and overview of the number of papers included.

From the 95 included papers, four studies combined two types parameters (kinetic and kinematic), and two studies combined two techniques (ECoG and fMRI).

Concept	Search terms
Primate or human	Primates, non-human primates, human
Sensorimotor cortex	Sensorimotor cortex, Brodmann Area (BA) 1–4, primary motor cortex, M1, primary somatosensory cortex, S1, sensory cortex,
Hand or finger Movement	Hand movement, finger movement
Functional magnetic resonance imaging	fMRI, MRI
Electrophysiology	Local field potentials, spikes, arrays, electrocorticography, ECoG, recordings
Encoding	Encoding, decoding, decoded, mapped, mapping
Kinetic or kinematic parameter	Individual finger, posture, hand gesture, velocity, direction, position, acceleration, movement trajectories, force, muscle activity, electromyography, EMG, movement speed, movement frequency

Number of papers included
Parameters	Primate electrophysiology	Human ECoG	Human fMRI	Total
SMC mapping	5	4	8	17
Kinetic	21	3	11	35
Kinematic	24	21	4	49
Total	50	28	23

3. Sensorimotor mapping of the hand and fingers

The first evidence for hand and finger representation in the SMC was presented in 1937 by Penfield and Boldrey, who showed that electro-cortical stimulation of specific areas of the SMC elicits hand/finger movements (Penfield & Boldrey, 1937). This representation, also known as somatotopy, has been intensively investigated in the last decades in both non-human primates and humans. In 1993, Schieber and Hibbard were the first to suggest that individual neurons do not encode the movement of a single finger, but are, instead, involved in the movement of multiple fingers. Indeed, later primate studies showed that the movement of a finger results from the activity of a neuronal population distributed throughout the hand area in M1, and that a single neuron may contribute to the movement of multiple fingers (Georgopoulos et al., 1999; Poliakov & Schieber, 1999; Hamed et al., 2007). Later, fMRI studies in humans confirmed the broad spatial distribution of the representations of individual fingers in M1 and showed, additionally, that these were highly overlapping (Figure 1A) (Indovina & Sanes, 2001; Dechent & Frahm, 2003; Wiestler et al., 2011; Olman et al., 2012; Diedrichsen et al., 2013; Shen et al., 2014). Nevertheless, some studies have demonstrated that individual fingers can still be decoded with high accuracy (Hamed et al., 2007; Kubanek et al., 2009; Hotson et al., 2016) and show a consistent ventral to dorsal activation map from thumb to little finger, respectively (Miller et al., 2009b; Siero et al., 2014). In order to further investigate the representation of individual fingers, Schellekens et al., (2018) recently proposed a new approach to interpret the dispersed finger-related fMRI activation in the SMC, which is much in line with the early evidence of Gaussian population receptive fields (pRF) in the visual cortex (Dumoulin & Wandell, 2008). They showed that if we consider a model of population receptive fields for fMRI voxels in the SMC, a detailed ventral to dorsal somatotopic organization of individual finger representations emerges, where each voxel is mostly associated with a single, preferred, finger and to some degree with the adjacent fingers (Figure 1B). By assuming that a neuronal population integrates movements of distinct but related fingers, this method promises to offer new perspectives for the interpretation of cortical motor cortical representation, motor control and sensorimotor integration.

Figure 1 – Individual finger mapping in the sensorimotor cortex.

A) Direct mapping of individual finger movements in the left-hemispheric M1 hand area on magnified views of the inflated cortex for one subject shows considerable overlap between the representation of individual fingers. D1–5, thumb to pinky, respectively (adapted with permission from Dechent and Frahm, 2003). B) Gaussian population receptive fields (pRF) associated with finger flexion, projected on flattened surfaces of 2 subjects (S1 and S2). The light grey lines define the borders of the sensorimotor cortex and the triangle is aligned with the base of the central sulcus (CS). The top row shows the estimated Gaussian centers (pRF center), which represent the finger digit somatotopy, with red to blue representing thumb to little finger. The bottom row depicts the color-coded Gaussian spread, with red to blue representing small to large receptive fields. PreCG – Pre-central gyrus; PostCG – post-central gyrus (adapted with permission from Schellekens et al., 2018).

4. Kinetic parameters of hand and finger movement

Kinetic movement parameters describe the relationship between movement and its dynamics, more specifically the forces, torques and muscle activities. These parameters are intrinsically interconnected, since muscles produce the force applied by the body. Force is a vector with both a magnitude and a directional component. Whereas the magnitude part can conceptually be related to the activity of a single muscle, the directional part can be associated to the kinematic parameter ‘direction’ described later in section 5.1. To date, force and muscle activity have been studied under static isometric, dynamic isometric and movement conditions. The static isometric condition refers to paradigms where the applied force is constant and coupled with no movement, while the dynamic isometric condition refers to studies where the applied force magnitude is continuously changing, but no movement is made. The (dynamic) movement condition is a combination of continuously changing force magnitude coupled with actual somatic movement. The latter involves a more complex interpretation, as during movement multiple forces and interactions are present (Hollerbach & Flash, 1982).

4.1. Magnitude of force and muscle activity

The first attempt to investigate the relation between movement parameters and neuronal responses started by comparing the neuronal activity and force patterns during flexion/extension of the wrist (Evarts, 1968) and wrist static fixation (Evarts, 1969). In these pioneer studies, the author concluded that exerted wrist force (and the derivative of the force), rather than the position or direction of displacement of the wrist, was related to the discharge of neurons in primates. Although a remarkable finding, this result did not seem to provide the whole story about the SMC as, later on, evidence for the representation of direction and position in the SMC was also found. Some findings regarding static force encoding showed to be consistent and reproducible across studies. Namely, it was shown that different neuronal populations have different linear monotonic relations to force (Evarts, 1969; Thach, 1978). While for some cells the monotonic relation held for the whole force range, for other cells the relation between neuronal activity and static force did not hold for forces at the extremes of the range, yielding a typical S-shaped function (Cheney & Fetz, 1980; Evarts et al., 1983). Other interesting findings were that (1) more neurons seemed to respond with larger amplitude, and with greater regression slope, to extensor muscles (or forces) when compared with flexor muscles (or forces) (Cheney & Fetz, 1980); (2) the magnitude of force seemed to be accompanied by an increase in firing rate rather than an increase in the number of recruited cells (Cheney & Fetz, 1980; Evarts et al., 1983); and (3) encoding of the force seemed to depend on the task characteristics, such as task sequence, visual stimuli and force ranges (Hepp-Reymond et al., 1999).

Other groups explored the relation between static isometric and dynamic isometric force conditions in primates (Smith et al., 1975; Wannier et al., 1991), and showed that while some cells were related to both force and rate of force change, other cells correlated with either force or rate of force change (Smith et al., 1975). Also, they demonstrated that most recorded neurons display various (different) discharge patterns during dynamic and static force conditions, from phasic (onset-related) increases and decreases, to tonic (proportional to force) increases and decreases, and combinations of both (Wannier et al., 1991).

In humans, multiple studies showed that the fMRI signal (number of activated voxels and/or average signal intensity) in contralateral M1 increases as a function of increasing levels of isometric static grip force (Thickbroom et al., 1998; Dai et al., 2001; Peck et al., 2001; Cramer et al., 2002; Van Duinen et al., 2008; Keisker et al., 2009). When comparing static with dynamic (movement) grip forces, however, some studies showed that static force induces a significantly smaller blood-oxygen-level dependent imaging (BOLD) signal than dynamic force in contralateral M1, both during isometric (Kuhtz-Buschbeck et al., 2008; Keisker et al., 2010) and movement conditions (Thickbroom et al., 1999; Ehrsson et al., 2000). These studies suggest that the processing of repetitive transient force changes requires more metabolic activity in M1 than static forces, possibly due to the movement onsets originated during the dynamic (movement) conditions. Using ECoG signals and sophisticated regression algorithms, three groups have attempted to predict the time-varying force profiles during coarse and precision grasp and showed an accurate prediction of force magnitude, mostly using low-passed ECoG signals and high-frequency band power signals from M1 (Pistohl et al., 2012; Chen et al., 2014; Flint et al., 2014). Altogether, force seems to be represented in the SMC of both non-human primates and humans, although human fMRI studies showed a more obvious representation during dynamic force conditions.

Peck and colleagues (2001) argued that the increase in activations observed with higher force levels can be attributed to the recruitment of additional muscle groups to stabilize the arm. Indeed, they showed that after correction for muscle activity, the fMRI signal only weakly increased with increasing force levels, suggesting that force and muscle are not independently encoded in SMC. Expanding on this point, many groups showed that it is possible to predict electromyography (EMG) activity from neuronal discharges (Morrow & Miller, 2002; Koike et al., 2006), from summed M1 activity (Schieber & Rivlis, 2006), from averaged neuronal population activity using ECoG (Shin et al., 2012; Flint et al., 2014; Nakanishi et al., 2017) and from fMRI BOLD signals (Ganesh et al., 2008).

4.2. Directional component of force and muscle activity

In 1985, Kalaska and Hyde reported, for the first time, on disentangling the directional component of the static isometric force from its magnitude, using a single paradigm that involved multi-joint 2D forces, and compared the effect of magnitude and direction of force on the neuronal and electromyographic (EMG) activities. Their results, later further extended and confirmed by other studies (Kalaska & Hyde, 1985; Kalaska et al., 1989; Taira et al., 1996), showed that most cells recorded from the motor cortex respond exclusively to the direction of force, whereas most of the muscle activity was correlated to both the direction and magnitude of force. These results suggested that (1) in the muscles, the specification of magnitude is embedded within the directional signal, (2) the direction of force is most prominent in M1, and (3) the direction of force can be controlled independent from its magnitude. Furthermore, these results are in agreement with findings from studies that used more complex dynamic isometric conditions, and which showed that M1 cortical cells are directionally tuned and that their activity varies with the change in force and visually defined target-directions, rather than with the total force (and muscle activity) exerted by the subject (Georgopoulos et al., 1983, 1992). The above results indicate that M1 is preferentially involved in the control of the dynamic component of the force and that when dynamic conditions are superimposed on static ones, the dynamic process seems to assume primary importance in the motor cortex. Lastly, studies that examined movement conditions in the presence and absence of external loads showed a strong effect of the directional component of the movement on the motor cortical activity in both situations (Georgopoulos et al., 1982; Schwartz et al., 1988; Kalaska et al., 1989; Riehle & Requin, 1995) (see also section 5.1). The findings that directional tuning seems to occur in M1 in both isometric and movement conditions suggest that there is a common underlying factor for M1 activity, possibly related to a more abstract spatial representation of the motor output (Ashe, 1997).

5. Kinematic parameters of hand and finger movement

Kinematic parameters comprise the spatial and motion aspects of the movement. These parameters can describe: 1) “static” direction during point-to-point movements; 2) continuously varying position, velocity and acceleration, which can be further subdivided into its amplitude and direction components; or combinations of these, such as movement trajectories. In this section, we summarize the non-human primate and human findings organized in five topics, being direction of point-to-point movements, continuously varying velocity and acceleration, static vs. dynamic position, movement rate and movement trajectories.

5.1. Direction of point-to-point movement

One of the first groups investigating kinematic parameters of hand movement in primates were Georgopoulos and colleagues (1982). They used a so-called center-out task, which consists of a set of eight targets peripherally arranged with the same radial distance from the center, each representing the target position of the monkey’s hand. The authors correlated neuronal spiking activity in M1 to the direction in which the monkey moved its hand. In this case, direction was studied irrespective of the externally applied forces (for a comparison with forces see section 4.2). The authors provided the first evidence of directionally tuned cells, meaning that the neuronal discharge of a specific cell is most intense for hand movements in a particular (‘preferred’) direction, and gradually reduces for hand movements in other directions (Georgopoulos et al., 1982). This led to the coining of the term population coding, which refers to the concept that every directionally-tuned neuron contributes additively to movement in any direction. These findings were strengthened by other primate studies that showed a strong correlation between neuronal activity and movement direction in both 2D and 3D spaces (Georgopoulos et al., 1986, 1988; Schwartz et al., 1988; Kettner et al., 1988; Kalaska et al., 1989; Fu et al., 1993, 1995; Ashe & Georgopoulos, 1994; Moran & Schwartz, 1999a; Rickert et al., 2005; Golub et al., 2014). Interestingly, Kettner et al., (1988) reported that the relation between neuronal activity and movement direction was independent of where the movement was made relative to the body. Others, however, have shown that changes in arm posture significantly change the preferred direction of M1 neurons, both during reaching tasks (Scott & Kalaska, 1997) and wrist movements (Kakei et al., 2003). These findings are of great importance for BCI control settings, in which the user body posture may change in the course of a day, and should be investigated further within that context.

Besides from single neuron recordings, there is also evidence for directional tuning from neural population recordings, but less reported than in primate literature. Human ECoG studies, for example, showed accurate decoding of direction from SMC during center-out tasks (e.g., Ball et al., 2009; Wang et al., 2012) using low-passed filtered features, low-frequency components (0–2 Hz) and high-frequency components (> 50 Hz); and during two-target online control of Brain-Machine Interfaces (BMI) (Milekovic et al., 2012) using uniquely low-passed filtered signals. The limited amount of evidence for directional tuning from human ECoG studies is probably related to the fact that ECoG electrodes measure from large populations of neurons each with a different preferred direction, whereas the primate studies investigated the responses mostly from (groups of) single cells. Nevertheless, it can be concluded that there is very strong and consistent evidence from both monkey and human research for the presence of a representation of movement direction in the SMC.

5.2. Velocity and acceleration

Velocity and acceleration were initially investigated in primates using the center-out task described above. Some reports concluded that, although less clearly represented than direction of velocity, the magnitude of velocity or speed is encoded in the SMC, and that the magnitude of acceleration is the least represented (Schwartz, 1993; Ashe & Georgopoulos, 1994; Moran & Schwartz, 1999a; Golub et al., 2014). Importantly, however, Paninski et al., (2004) and Wang et al., (2007) expressed concerns about the center-out task, since its concept makes it difficult to study velocity as a separate movement parameter, due to its interdependence with other parameters, such as position. Wang and colleagues (2007) attempted to reduce the statistical dependencies between velocity and position, and compared the results of a standard center-out reaching task with a random reaching task, in which the starting position and end target were chosen randomly in the 3D space (Wang et al., 2007). They showed not only a representation of position but also confirmed the presence of a representation of velocity magnitude (speed) and velocity direction in M1 during movement during target-holding and movement periods.

Human ECoG studies have reported that speed is more clearly represented in M1 than velocity direction, and that specific frequency components of the signals are associated with each of these kinematic parameters (Hammer et al., 2013, 2016). The authors argued interestingly that the difference in representation of direction and speed between the single neuron level (where direction was better represented than speed) and large population level could be explained by the scale of the recording, in that the speed tuning may be constructively summed up across the neuronal population, and is, therefore, most evident in the average of large enough populations (> 10.000 neurons) (Hammer et al., 2016).

5.3. Static position vs. dynamic position

Using the center-out task, several groups studied the relationship between neuronal response and different static hand positions in primates and showed that the neuronal discharge varied with the (target, static) position and distance between hand position and the target (Georgopoulos et al., 1984; Fu et al., 1993, 1995). In order to interpret the seemingly similar correlation of position and movement direction with single neurons discharge (section 5.1), Fu et al., (1995) suggested that these were in fact temporally encoded in M1. That is, the specific timing of the firing rather than the firing rate itself would encode these parameters. Indeed, they found that the majority of the direction-related neuronal discharge occurs first during the pre-movement period, followed by a peak just before holding at the final position. These results demonstrate that single cells could encode multiple parameters in a serial manner, tentatively explaining the encoding of more than one movement parameter. This theory was later explored by other groups who found sequential encoding of the dynamically varying velocity and position (Paninski et al., 2004; Wang et al., 2007).

In humans, static position paradigms using finger and hand postures were investigated mostly with the ultimate goal of increasing the degrees-of-freedom of BCI systems. Both ECoG (Chestek et al., 2013; Bleichner et al., 2016) and fMRI (Bleichner et al., 2014; Leo et al., 2016) studies revealed accurate discrimination of a limited number of different hand/finger postures and synergies (i.e., hand postures that result from recruiting sets of muscles and joints simultaneously), using the averaged signals during the movement periods, even during online control of a prosthetic limb (Chestek et al., 2013). Remarkably, similar to what was found in primates (Fu et al., 1995), Bleichner et al., (2016) indicated that neuronal activity, especially the high frequency broadband (> 75 Hz), returned to baseline values during the static phase after movement. Exploring this, the same group later showed that temporal information was crucial for decoding, and that the combination of temporal and spatial features increased decoding accuracy (Branco et al., 2017) even in subjects for whom decoding results were initially poor (Bleichner & Ramsey, 2014). These results, together with those of Fu and colleagues (1995), suggest that static position itself is not represented in the SMC, but that SMC activity is more likely associated with the movement towards the position (i.e., the hand trajectory and movement direction).

5.4. Movement rate

Human fMRI and ECoG studies have also focused on describing hand/finger movement rate, that is, the frequency of repeated finger tapping movement, where an increase in finger tapping rate leads to an increase in both the velocity and acceleration amplitudes of the finger movement. In fMRI, a linear relationship between the movement rate and the BOLD amplitude was found for lower movement rates, but saturation occurs at higher movement frequencies (Sadato et al., 1997; Jäncke et al., 1998; Riecker et al., 2003; Siero et al., 2013). The range of frequencies for which the linear relation (and saturation thereafter) was identified varied considerably across studies, from 0–1 Hz (Siero et al., 2013) to 1.5–5 Hz (Jäncke et al., 1998), and there is currently no explanation for these differences in saturation points. The saturation itself has been associated with a decrease in the amplitude of movement-induced changes in spectral power in the high frequency band (> 65 Hz) of the electrical neuronal signal (Siero et al., 2013). Indeed, ECoG research has shown that, after the first movement, neuronal activity during each subsequent movement declines when multiple similar movements are made at a fast rate (Hermes et al., 2012b; Siero et al., 2013). Nevertheless, none of the previous studies has investigated the modulation of the ECoG signal’s amplitude with respect to movement rate and it remains to be determined if increasing movement rate is associated with consistent change in the amplitude of the recorded signal.

5.5. Movement trajectories

So far, most studies have focused on the description of kinematics variables as static, scalar quantities, by comparing the temporal average of the movement feature and the temporal average of the neuronal output. By combining information from position and velocity, however, one can reconstruct the trajectory of the movement, that is the path followed by the hand or fingers when moving towards a target or position. Indeed, investigating the representation of complete continuous, more natural, trajectories can be of great interest, not only for our understanding of the neuronal underpinnings of movement, but also for the design of BCIs.

The first evidence for the presence of a representation of dynamic position (i.e., trajectory) in the motor cortex was given by Georgopoulos et al., (1988), who analyzed data from two monkeys performing a 3D center-out-task. This study, later confirmed by (Schwartz, 1993; Moran & Schwartz, 1999a, 1999b; Schwartz & Moran, 1999), showed that although quantitatively less represented in M1 than target direction and velocity, the continuous change in position of the hand was highly correlated with the neuronal response and could be constructed using both directional and length information of the population vector. Later, several primate needle studies reconstructed hand and arm movement trajectories from the SMC and used them to control, in real-time, robotic device in 1D and 3D spaces, using both linear and non-linear algorithms (Wessberg et al., 2000; Paninski et al., 2004; Aggarwal et al., 2013). Aggarwal et al., (2013) showed, additionally, that spiking activity, rather than local field potentials (averaged signal between 0.7 and 175 Hz), was the most informative for decoding trajectories. Other studies in monkeys showed successful reconstruction of arm (Chao et al., 2010) and hand (Shimoda et al., 2012) trajectories, mostly from the higher-frequency components of ECoG (> 40 Hz). These results are, in fact, in line with the findings of Aggarwal and co-workers, since higher-frequency band signals are thought to be especially related to spiking activity (e.g., Miller et al., 2009a; Buzáki & Wang, 2012; Ojemann et al., 2013).

Successful reconstruction of 2D and 3D trajectories has also been demonstrated using human ECoG signals recorded during arm and finger movements (Schalk et al., 2007; Pistohl et al., 2008; Gunduz et al., 2009; Acharya et al., 2010; Nakanishi et al., 2014, 2017; Bundy et al., 2018). Several of these studies investigated which electrodes and temporal-spectral features of the ECoG signal were most useful for decoding of trajectories. In general, M1 was the most informative cortical region, followed by S1, and the high-frequency band signals together with the low-pass filtered ECoG signal, also known as local motor potential (Schalk et al., 2007; Pistohl et al., 2008; Acharya et al., 2010; Nakanishi et al., 2014, 2017; Bundy et al., 2018), were the most informative features. Remarkably, in one of the studies of Nakanishi and colleagues (2014), the authors even showed that the finer fingertip trajectories (Figure 2) can be predicted with very high precision (r > 0.92) from the upper part of the SMC.

Figure 2 – Fingertip trajectories.

Fingertip trajectories (x, y, z coordinates) were decoded from one (epilepsy) patient implanted with ECoG grids (1 cm inter-electrode distance) over the left hemisphere sensorimotor cortex. In this study signals from nine frequency bands were used: 0–4 Hz, 4–8 Hz, 8–14 Hz, 14–20 Hz, 20–30 Hz, 30–60 Hz, 60–90 Hz and 90–120 Hz. Left panel: 3D view of the finger trajectories. The delta and high-frequency bands (> 90 Hz) contribute most significantly to the trajectory prediction. Predicted (red lines) and actual trajectories (blue lines) for all trials are displayed. Right panel: Examples of the predicted (red lines) and actual trajectories (blue lines) for three individual fingers (thumb, index and middle finger) compared using correlation coefficient (CC) and normalized root-mean-square-error (nRMSE) values. The graphs express changes in x, y, and z coordinates over time, as well as the x–z plane projections (bottom row) of curves in the 3D view (adapted with permission from Nakanishi et al., 2014).

Notably, the aforementioned primate and human studies investigating the representation of movement trajectory used sophisticated regression models to optimally predict the position and velocity vectors, which incorporated not only multiple kinematic variables, but also their temporal dynamics. The adequate decoding of movement trajectories obtained with these models indicates that the use of movement trajectories for BCI purposes may provide users with a precise interaction with the environment.

6. Discussion

Brain-Computer Interfaces have been proposed as a technology to replace, restore, enhance, supplement or improve (lost) natural central nervous system output (Wolpaw, 2007; Brunner et al., 2015). The sensorimotor cortex (SMC) is frequently taken as a source of signals for BCI control, as it shows large and consistent signal changes related to movement. However, a further improvement of the accuracy and specificity of BCI control will benefit from a thorough understanding of how the SMC cortical activity specifies the spatiotemporal properties of the movement. To what extent kinematic and kinetic parameters contribute to the output movement is still a topic of great discussion (Kalaska, 2009; Todorov, 2000). Therefore, we summarized here the most consistent evidence on the representation of kinematic and kinetic parameters of arm, hand and finger movement control in the SMC.

6.1. Using kinetics or kinematics for BCI?

In general, evidence for a clear representation of most movement parameters was found, from low-level forces and muscle activities (kinetics) to high-level spatial and motion aspects (kinematics). The first studies on movement parameters compared force to neuronal output (Evarts, 1968), based on the assumption that the motor cortex has a direct relation with muscle output. Later on, multiple studies focused on the direction component during isometric force and movement conditions, both revealing a strong evidence (mostly in monkeys) for direction tuning of the SMC neurons (Georgopoulos et al., 1982, 1986, 1992; Kalaska & Hyde, 1985; Kettner et al., 1988; Schwartz et al., 1988; Taira et al., 1996; Moran & Schwartz, 1999a). Other interesting findings were that movement parameters (e.g., movement direction) did not seem to be coded by single-neuron patterns but by neuronal ensembles (Georgopoulos et al., 1982), and that the neurons may temporally encode different parameters at different stages of the movement (Fu et al., 1995; Sergio & Kalaska, 1998; Moran & Schwartz, 1999a; Elsayed et al., 2016; Branco et al., 2017), pleading for the prevailing encoding of spatial-temporal features of the movement, such as trajectories, rather than separate parameters. Indeed, the prediction of trajectories (combination of hand/finger position and velocity) showed impressive results both in non-human primates and humans (e.g., Wessberg et al., 2000; Shimoda et al., 2012; Nakanishi et al., 2014), suggesting that the SMC provides enough discriminative information to replicate the intended arm and finger movements, a concept which can be successfully used for online BCI control of prosthetics and robots (Wessberg et al., 2000; Paninski et al., 2004; Aggarwal et al., 2013).

Even though the decoding of continuous variables, such as movement trajectories, is of great value for the BCI control of robotic arms and hands, BCIs can also be benefit from the decoding of discrete classes of movement, in that each class could control an independent degree-of-freedom of the system. In that regard, the most discriminative parameters represented in the human SMC appear to consist of individual finger movement (either single (Miller et al., 2009b; Siero et al., 2014) or repeated movements (Hermes et al., 2012b; Siero et al., 2013)), movement direction (Milekovic et al., 2012) and complex movements (such as hand postures, configurations or muscle synergies (Chestek et al., 2013; Ejaz et al., 2015; Bleichner et al., 2016; Leo et al., 2016)). Using ECoG and fMRI recordings in humans, the optimal spatial location and the best temporal-spectral signal features for decoding were investigated. Most discriminative information was located in the SMC, with some studies indicating significant information from supplementary motor areas (Indovina & Sanes, 2001), pre-motor cortex and pre-frontal cortex (Chao et al., 2010; Shimoda et al., 2012; Nakanishi et al., 2017); and most discriminative features were the low-passed filtered and high-frequency band signals (Chao et al., 2010; Shimoda et al., 2012; Nakanishi et al., 2017).

Altogether, the extensive variability of parameters represented in SMC supports the presence of a complex relationship between behavior and neuronal activity, likely because the SMC is responsible for the generation of complex movements composed of different speeds, forces and directions, and interacts with many different cortical and subcortical areas and spinal cord. In 2009, Kalaska raised several interesting questions, such as: “Which movements should we be looking at?”, “Can we really decouple movement parameters?”, and “What is the role of the motor network on the SMC?” Indeed, most studies looked at correlations between neuronal output and specific parameters, which can be easily confounded by the inevitable correlation to other parameters (Reimer & Hatsopoulos, 2009), and therefore are unlikely to unveil the causality between any of the parameters and the neuronal source. As an example, most of the individual finger representation studies used finger flexion and extension paradigms, which also elicit changes in finger velocity, position and direction. Perhaps, a better way to interpret these results is to consider the existence of a motor mechanical and behavioral model that incorporates the complete motor output (Ebner et al., 2009), as discussed in the following section.

6.2. Evidence from sensorimotor control models

One important question in the study of movement parameters is whether the concept of parameterization (i.e., describing the movement as a series of parameters) is valid (Ebner et al., 2009). In recent years, the view on, and study of, movement parameters shifted from single parameter (e.g., direction, position, velocity) and simple conditions (single-joint, static isometric paradigms) to multiple parameters and complex conditions (trajectories, multi-joints, dynamic movement). As mentioned above, almost all movement parameters are associated with a detectable response in the SMC. This could be a result of the intrinsic correlation between movement parameters created by the musculoskeletal biomechanics and anatomy constraints. For instance, hand movement automatically involves a combination of direction and magnitude of muscle contraction and forces, movement velocity and acceleration. However, these parameters are all interconnected and can be modelled together as part of one movement. Of note, two frameworks have been widely used to model the human movement, the Optimal Feedback Control (OFC) and Active Inference (AI) models (Friston, 2011). Even though other models are being used to explain motor control, we focus on these two as these are the most prominent frameworks and present contrasting theories of how the brain and in particular the sensorimotor cortex controls movement. The OFC framework is an engineering model based on the principle of optimization. In this framework, M1 is assumed to be the cortical region in charge of sending top-down motor commands, where the motor command estimation is created by minimizing the error between the feedback signals (sensory feedback provided by S1) and an efferent copy of the motor commands created by an optimal feedback law (Scott, 2004). In other words, the OFC is a closed-loop process that minimizes the variability in the output parameters by continuously measuring the feedback and correct the motor output. Conversely, the AI framework is a statistical hierarchical model based on the principle of energy minimization. In this framework M1 is thought to provide top-down predictions (rather than send motor commands), while other lower-level structures (e.g., spinal cord) are responsible for producing the motor commands. That is, AI is based on prediction of the motor output that relies on prior predicted beliefs and minimizes the prediction error by adjusting the internal model (Adams et al., 2013). Whereas in OFC both forward and inverse models are computed by the motor cortex, in AI the inverse mapping is left to the spinal cord and the (generative) forward model is estimated from the sensory data. Even though there is no consensus over which model best represents reality, the OFC model has been explored more intensively in the last decades. In an effort to explain that the correlations found between neuronal signals and multiple movement parameters are not mutually exclusive, Todorov (2000) developed an interesting OFC mechanistic model to explain the representation of motor behavior in the SMC, which incorporates properties of muscle and multi-joint mechanics. The model shows how M1 activity can cause motor behavior, just by taking into consideration the physical (muscle and joint) constraints associated with movement. He showed that the observed correlations of parameters simply emerge from the model and were consistent with the ones found in the literature, such as the representation of force magnitude in isometric tasks and of velocity in movement tasks, the dominance of velocity and force direction over magnitude, and the temporal multiplexing of direction and target position signals. This interesting theory, recently corroborated (Teka et al., 2017), suggests that the relative contribution of different movement parameters are not physiological characteristics, but rather depend on the actual motor behavior. Based on this theory, recent studies have attempted to use OFC to control Brain-Machine Interfaces in primates, by developing decoders that combine the target information, external feedback and the neuronal spiking activity during movement (Shanechi et al., 2013a, 2013b; Benyamini & Zacksenhouse, 2015). Results of these studies show not only that this model closely mimics the sensorimotor control system, but also that the use of more advanced models can be very promising for neural-control.

6.3. Sensorimotor target areas for BCI control

This review focused on the results obtained from the sensorimotor cortex as a whole, up until now with no distinction made between M1 and S1 cortices. However, the specific role of M1 and S1 cortices and their interactions are a matter of great discussion. Despite the fact that M1 was classically defined as the center of motor control and S1 the center for sensory feedback, new evidence has shown that the role of S1 goes beyond sensory information processing. For example, a recent study using ECoG in humans has shown that S1 activates before M1 during hand movement (Sun et al., 2015). In addition, high accuracy decoding of hand and finger movements from S1 has been demonstrated by recent BCI studies performed with paralyzed patients (Wang et al., 2013; Degenhart et al., 2018) and people with arm amputation (Kikkert et al., 2016; Bruurmijn et al., 2017). These results support data from earlier fMRI and electrocortical stimulation studies, in particular the findings 1) in individuals with spinal cord injury, who show fMRI activation of S1 during attempted movements (Cramer et al., 2005; Hotz-boendermaker et al., 2008, 2011); 2) in individuals with induced ischemic nerve blocking, who had preserved fMRI responses without sensory feedback (Christensen et al., 2007); and 3) in individuals with epilepsy who show isolated and complex hand responses upon cortical stimulation of S1 (Nii et al., 1996; Haseeb et al., 2007). Interestingly, the two frameworks discussed in the previous section have contrasting views regarding the role of S1. While OFC considers S1 uniquely as a source of sensory information and describes an efferent copy from M1 to S1 for control optimization, in AI the efferent copy comprises proprioception information and is sent from S1 to M1. That is, while M1 is involved in modeling the intended body state, S1 is involved in predicting the current (proprioceptive) body state (Adams et al., 2013). Regardless the role S1 plays in motor control, the above evidence suggests that BCIs may benefit from exploiting both M1 and S1 cortical regions for control and therefore the study of movement parameters should not be restricted to M1.

6.4. Other BCI considerations

Besides a search for movement parameters that are associated with the most consistent and large signal changes in the SMC, there are other considerations that need to be addressed when designing a BCI. First, it should to be noted that most BCI studies described in this manuscript focus on offline decoding of multiple movement parameters. The question is whether the interpretation of the results between offline and online experiments would differ. Several studies have investigated this topic, and showed that the offline decoding of hand movement is highly predictive of BCI performance in real-time applications that use, for instance, prosthetic devices for feedback (Gharabaghi et al., 2014; Yanagisawa et al., 2011) or to control a one-dimensional ‘brain-click’ on a computer program (Vansteensel et al., 2016).

Second, the target population for BCIs are typically individuals with severe paralysis and/or communication problems (Pels et al., 2017). In most cases, actual movement is not possible, leaving imagined and/or attempted movement as alternative options to control a BCI. In this review we focused on studies where actual movement was studied, and for these studies to be of optimal value for the BCI field, we need to be aware of the similarities and differences in neuronal representation of executed, imagined and attempted movement. Importantly, a great amount of evidence shows that attempted movements generate similar activation patterns in the SMC as actual movement. Examples are studies that showed that attempted and phantom hand movements could be accurately and robustly classified from the high-frequency band SMC signals using a real-time closed-loop feedback control, from amputated or paralyzed individuals (Wessberg et al., 2000; Yanagisawa et al., 2011; Wang et al., 2013; Gharabaghi et al., 2014; Degenhart et al., 2018); two extended fMRI studies that showed that finger representation was intact (Kikkert et al., 2016) and that attempted complex hand gestures (6 in total) can still be decoded accurately from individuals with an amputated arm (Bruurmijn et al., 2017). Moreover, the results from the human pilot clinical trial BrainGate Neural Interface System (http://www.clinicaltrials.gov/ct2/show/NCT00912041) have demonstrated successful decoding of hand kinematics from SMC in paralyzed individuals (Zhuang et al., 2010; Simeral et al., 2011; Ajiboye et al., 2012; Hochberg et al., 2012; Jarosiewicz et al., 2015; Pandarinath et al., 2017). An interesting question remaining to be answered is whether the OFC and AI motor control models described above (section 6.2) change when no actual movement is performed and no feedback is provided.

Third, we saw that hand and fingers are well represented in the SMC, although with some overlap. Considering that the brain is a plastic organ, an interesting question would be if any other body part, such as lips, eyes or legs, produce activity in the same cortical location as the hand and fingers. This is especially relevant for users that still have residual muscle movement or control eye-tracking systems using their eyes. In these situations, it is not known whether there is overlapping activity and if so, how are these different from the cortical presentations known to date.

Fourth, as shortly mentioned above, there are indications that arm posture influences the cortical representation of some movement parameters, such as movement direction (Scott & Kalaska, 1997; Kakei et al., 2003). In other words, the representation of hand movement may change with different arm positions or orientations. This result may have consequences for home-use BCI control of severely paralyzed individuals, as a new cortical representation implies re-interpretation of the parameters (i.e., system calibration) each time the position of the arm changed (e.g., after caretaking). Clearly, the extent to which this phenomenon affects BCI performance should be addressed in future studies, since stable and reliable BCI accuracy in real-life home situations is crucial for BCI acceptance and use by the target population (Huggins et al., 2011).

Lastly, with regard to the concept of learning (Paz & Vaadia, 2004; Ebner et al., 2009), it remains unknown whether the encoding of any parameter is predetermined or is a learned association. It is expected that the brain adapts to relevant behavioral variables, meaning that frequently used features should be encoded stronger than less frequently used features (Ebner et al., 2009). Moreover, considering the fact that the motor structures are plastic and people are able to learn to control strategies, it is unclear whether chronic use of a particular movement (parameter) in BCI control settings affects the representation of the movement parameter and the long-term BCI performance. From literature, some evidence supports the idea that that frequently used finger configurations are more strongly represented than less frequently used configurations (Leo et al., 2016), but this topic needs further investigation from a BCI perspective.

6.5. Limitations

Some of the diversity between the results of different studies may be explained by the different nature of the recording techniques. In this review we focused on studies using intracortical needle, intracranial ECoG and fMRI recording techniques, since these provide the most accurate spatial and/or temporal resolution of the acquired signals, which is especially important for the mapping of movement parameters to concise cortical regions (order of 1 neuron to 10.000 neurons) and in short time windows (order of milliseconds) (Nicolas-alonso & Gomez-gil, 2012). Most intracortical research was performed on non-human primates, which may not directly extrapolate to humans (Passingham, 2009). Yet, consistent conclusions were drawn across studies in multiple parameters, such as movement direction and trajectory. Moreover, less literature focused on fMRI measurements, possibly since these are considerably susceptible to movement artefacts, leading to limitations regarding the study of certain movement parameters, and because of the slow hemodynamic response. Nevertheless, fMRI studies provide a broader mapping of cortical and sub-cortical areas than electrophysiological techniques, and fMRI activation patterns have been shown to be highly correlated to changes in high-frequency ECoG signals (e.g., Hermes et al., 2012a). As such, fMRI is recognized as a valuable technique to study the brain, also for BCI purposes.

Finally, there was quite some variability in the paradigms used, even when authors attempted to examine the same parameter of the movement. The reason for this difference is likely due to the considerable differences in setup between primate and human studies, and between electrophysiological and hemodynamic studies. In order to provide consistent and reproducible findings across all fields, an effort should be made to standardize paradigms to study specific parameters, allowing for a more accurate comparison between techniques.

7. Conclusion

This literature review provides a summary of existing literature on the representation of different kinetic and kinematic parameters of movement in the sensorimotor cortex, with the goal of providing the research community with information about which parameters of the movement are promising candidates as a BCI control paradigm. We conclude that all evaluated parameters are to some extent represented in the sensorimotor cortex. Nevertheless, we show that some strategies, such as individual finger movement, movement direction, complex hand movements and movement trajectories can be most accurately discriminated from the SMC and are, therefore, the most promising candidates for BCI control. The broad evidence for the encoding of multiple parameters suggests that movement should likely be interpreted as a combination of multiple parameters and that more complex mechanistic models may be the key to describe motor behavior. Future work should evaluate the decoding performance using these strategies, its stability in chronic long-term BCIs, as well as the online implementation of control models to improve decoding of natural movements.

Abbreviations

1D

One-dimensional

2D

Two-dimensional

3D

Three-dimensional

AI

Active inference

BA

Brodmann area

BCI

Brain-computer interface

BMI

Brain-machine interface

BOLD

Blood oxygen level dependent

CC

Correlation coefficient

CS

Central sulcus

ECoG

Electrocorticography

EMG

Electromyography

fMRI

Functional magnetic resonance imaging

M1

Primary motor cortex

MRI

Magnetic resonance imaging

nRMSE

Normalized root-mean-square-error

OFC

Optimal feedback control

PreCG

Pre-central gyrus

PostCG

Post-central gyrus

pRF

Population receptive fields

S1

Primary somatosensory cortex

SMC

Sensorimotor cortex