Decoding and Cortical Source Localization for Intended Movement Direction With MEG

Abstract

Magnetoencephalography (MEG) enables a noninvasive interface with the brain that is potentially capable of providing movement-related information similar to that obtained using more invasive neural recording techniques. Previous studies have shown that movement direction can be decoded from multichannel MEG signals recorded in humans performing wrist movements. We studied whether this information can be extracted without overt movement of the subject, because the targeted users of brain-controlled interface (BCI) technology are those with severe motor disabilities. The objectives of this study were twofold: 1) to decode intended movement direction from MEG signals recorded during the planning period before movement onset and during imagined movement and 2) to localize cortical sources modulated by intended movement direction. Ten able-bodied subjects performed both overt and imagined wrist movement while their cortical activities were recorded using a whole head MEG system. The intended movement direction was decoded using linear discriminant analysis and a Bayesian classifier. Minimum current estimation (MCE) in combination with a bootstrapping procedure enabled source-space statistical analysis, which showed that the contralateral motor cortical area was significantly modulated by intended movement direction, and this modulation was the strongest ∼100 ms before the onset of overt movement. These results suggest that it is possible to study cortical representation of specific movement information using MEG, and such studies may aid in presurgical localization of optimal sites for implanting electrodes for BCI systems.

INTRODUCTION

Technologies for augmentative communication and mobility are needed to replace motor functions lost after damages to the CNS, such as those caused by stroke, spinal cord injury, and amyotrophic lateral sclerosis (ALS). Unfortunately, these motor impairments limit the user's ability to use many currently available (i.e., muscle-based) communication aids or mobility devices. This creates a need for brain-controlled interfaces (BCIs) that express motor commands via neural signals recorded directly from the brain (Hochberg et al. 2006). Previous studies have shown that various hand movement parameters can be decoded from the activities of motor cortical neurons (Georgopoulos et al. 1986; Moran and Schwartz 1999; Paninski et al. 2004; Wang et al. 2007), enabling real-time brain control of computer cursors and robotic arms (Serruya et al. 2002; Taylor et al. 2002; Velliste et al. 2008; Wessberg et al. 2000). Recent human studies showed that a significant amount of movement-related information can also be extracted from macroscopic recordings obtained from the brain surface with electrocorticography (ECoG) (Ball et al. 2009; Crone et al. 1998; Leuthardt et al. 2004; Pistohl et al. 2008). Recent studies showed that subjects are able to gain accurate control of cursor movements using ECoG signals (Leuthardt et al. 2004; Schalk et al. 2008), making ECoG a promising alternative to penetrating microelectrode arrays for BCI applications.

Noninvasive methods for measuring brain activity, such as EEG and magnetoencephalography (MEG) have also been used for BCI applications (Mellinger et al. 2007; Wolpaw and McFarland 2004). MEG studies can support the development and implementation of implantable BCI systems by providing presurgical localization of cortical sources for movement information and user training on BCI control. MEG is well suited for these tasks, because, putatively, the spatial and temporal characteristics of MEG may be comparable to ECoG, especially after MEG signals are mapped appropriately from the sensor space (MEG sensor signals) to the source space (cortical surface activity) (Dalal et al. 2008; Gharib et al. 1995; Korvenoja et al. 2006). Several groups showed that hand movement can be decoded from MEG signals during figure drawing (Georgopoulos et al. 2005) and center-out (Waldert et al. 2008) movements. Furthermore, a recent study showed that the speed (fast vs. slow) of imagined wrist movement can be extracted from EEG recorded in individuals with ALS (Gu et al. 2010). In a recent real-time MEG-BCI study, individuals with severe upper extremity paralysis caused by stroke were enabled to control a hand orthosis by modulating their sensorimotor rhythm with imagined hand movement (Buch et al. 2008). This study further examines human cortical activities as recorded with MEG during overt and imagined movements to investigate whether intended movement direction can be decoded in the absence of overt movement, because the candidates for BCI technology are individuals with severe motor impairments. In addition to sensor-space analysis, this study also performs statistical analysis on cortical sources found using a minimum current estimation (MCE) source localization algorithm in combination with bootstrapping to identify cortical areas that encode significant information about intended movement direction in individual subjects. The capability of MEG to localize noninvasively the cortical areas most strongly modulated by intended movement direction may enable presurgical user training and source mapping of the whole brain, with the goal of finding optimal electrode implantation sites.

METHODS

Participants

All procedures were approved by the Institutional Review Board at the University of Pittsburgh, and all experiments were performed in accordance with the approved protocol. This study recruited 10 able-bodied subjects, 7 men and 3 women, with no previous history of nervous system diseases. Their ages ranged from 25 to 45 yr old. All subjects gave informed consent before participating in this study.

Experiment setup

A nonmagnetic back-projection screen was placed in front of the subjects to present visual feedback during behavioral tasks. A 306-channel whole head MEG system (Elekta Neuromag, Helsinki, Finland) was used to record brain activity. This system has 102 sensor triplets, with each triplet containing one magnetometer, one longitudinal gradiometer, and one latitudinal gradiometer. Muscle activity (EMG) of wrist flexor and extensor muscles (flexor carpi radialis and extensor carpi radialis) was recorded in all sessions. Electrooculography (EOG) was recorded with electrodes placed above, below, and lateral to the eyes. EOG captured horizontal and vertical eyes movements, as well as eye blinks. All MEG, EMG, and EOG signals were band-pass filtered between 0.1 and 300 Hz and sampled at 1,000 Hz. Additionally, four head position indicator (HPI) coils were placed on the subject's scalp to record head position relative to the MEG helmet at the beginning of each session. These coils, along with three cardinal points (nasal, left, and right preauricular), were digitized and used for head movement compensation, co-registration with structural MRI data, and spatial filtering. Subjects performed wrist movements while holding a MEG-compatible joystick (Current Designs, Philadelphia, PA). BCI2000, a general purpose BCI software package (Schalk et al. 2004) running on a high-performance personal computer, was used to control the experiment paradigm and track joystick movement. The BCI2000 software also sent digital output to the MEG system through the parallel port to synchronously mark the occurrence of various behavioral events (e.g., target onset) in the MEG data.

Experiment protocol

Subjects performed a two-dimensional (2-D) center-out task using wrist movement in four directions (radial deviation, ulnar deviation, flexion, and extension) following the corresponding visual target (up, down, left, and right). Wrist movement was chosen to minimize contamination of recorded MEG signals by arm and shoulder muscle activities, and subjects were instructed to make movement solely with the wrist, keeping the shoulder and arm at rest. During the overt movement task, subjects controlled a 2-D cursor using the wrist to perform the center-out task. Visual feedback of the cursor was provided continuously. Movement to the left/right targets required ∼40° of wrist flexion/extension, whereas the up/down targets required ∼25° of radial/ulnar deviation. A total of 120 successful repetitions (i.e., cursor hit the target) were made for each target. To minimize eye movements, subjects were instructed to fixate their eyes on a cross-hair in the center of the screen throughout each trial. The time course of a typical trial is shown in Fig. 1. Each trial started after the subject held the cursor at the center target for a random holding time (1–2 s), which triggered the appearance of one of the four peripheral targets. To complete a trial successfully, the subject needed to move the cursor to the target and hold for a random holding period (0.5–1.5 s). If the subject did not hit the target within 1 s, the trial was aborted. For five subjects (subjects S6–S10), a delay period was inserted between target onset and the go-cue. At the end of the center-hold period, one of the peripheral targets appeared, but the center target did not disappear. The subject was instructed to move only after the center target disappeared (go-cue), which happened within 0.5–1.5 s after the presentation of a peripheral target.

Fig. 1.

The center-out task with accompanying speed profile during overt wrist movement. The subject controls the 2-dimensional (2-D) cursor position using wrist movements. The cursor needs to go to the center and stay there for a hold period until the peripheral target appears. Then the cursor moves to the target and stays there for another hold period to complete the trial successfully. The target changes color when hit by the cursor and disappears when the holding period has finished. There was a 1-s interval between trials. The bottom trace shows the speed profile of the cursor from a representative trial, and the dotted lines delimit the premovement/planning period within which a time window was identified and used for decoding the intended movement direction for overt movement trials.

During the imagined movement task, subjects were asked to imagine performing the center-out movement using the wrist. When a peripheral target was presented, after a simulated reaction time (0.5–1 s), the cursor started moving with constant speed toward the target. To keep subjects engaged in the imagined movement task, catch trials were inserted. During a catch trial, the cursor stopped moving before it reached the target. The subject needed to press a button to inform the experimenter that a catch trial was recognized. EMGs of wrist flexor and extensor muscles were recorded at all time to ensure that subjects did not generate covert muscle activity during imagined movement sessions (see Supplemental Fig. S1 for further details on the EMG analysis).¹ Blocks of 20 successful repetitions (i.e., cursor hit the target and no EMG activity for wrist was recorded, catch trials excluded) per target were used, with a few minutes of rest between blocks when necessary. Blocks of overt and imagined movements were intercalated.

During a separate visit, a standard head structural MRI scan using magnetization-prepared 180° radio-frequency pulses and rapid gradient-echo (MP RAGE) protocol (Brant-Zawadzki et al. 1992) was performed for each subject for co-registration with MEG data and source localization.

Data preprocessing

Data analysis was performed on successful trials only. First, spatial filtering was performed on the raw MEG data using the signal space separation (SSS) method (Taulu et al. 2005). Briefly, the SSS algorithm decomposes MEG signals recorded from all sensors into linearly independent internal and external terms. Internal terms correspond to sources inside the MEG helmet and external terms to sources outside the helmet. The SSS algorithm is able to spatially filter MEG data so that only signals coming from sources inside the MEG helmet are preserved and signals coming from sources outside the MEG helmet are suppressed. In addition, the SSS algorithm also corrects for head movement between multiple experiment sessions and minimizes effects of sensor noise. Second, trials with eye movement or eye blinks were rejected. A threshold of 150 mV was applied to each repetition, and whenever the peak-to-peak amplitude of the signal of either EOG channel (i.e., horizontal or vertical) crossed this threshold, the repetition was rejected from further analysis. We further excluded data from subjects whose EOG was significantly correlated with target direction, i.e., EOG contains information about intended movement direction, to ensure that any directional modulation of MEG signals identified by this study was not caused by eye movement (see Supplemental Materials for detailed discussion). Third, for each trial, a Gaussian kernel of 25 ms in width (σ) was used to smooth MEG signals (Fig. 2). Finally, all MEG data for overt movement were aligned to movement onset (time point when cursor speed 1st exceeded 10% of peak velocity), and data for imagined movement were aligned to target onset. MEG data were analyzed in both the sensor space using MEG sensor signals and the source space using cortical activity obtained from the source localization algorithm. Please note that, although additional features in MEG signals, such as amplitudes of different frequency bands, also convey movement-related information, this study focused on the time-domain MEG signals. This is similar to the study performed by Georgopoulos et al. (2005) and is motivated by previous MEG and ECoG studies. Those studies have shown that the time-domain signal, especially the low-frequency component (LFC), also called local motor potential (LMP), carries a significant amount of information about hand movement direction (Ball et al. 2009; Pistohl et al. 2008; Schalk et al. 2007; Waldert et al. 2008).

Fig. 2.

Effects of signal smoothing (A) and magnetoencephalography (MEG) signal modulation by movement directions (B and C) for 1 MEG sensor (gradiometer) above the contralateral sensorimotor area. Data were collected from subject S1. A: raw MEG trace (solid gray line) overlaid by its smoothed version (dashed black line) during a single overt trial. Time 0 represents movement onset. B: each trace represents the average of smoothed signals of all repetitions for each movement direction for 1 subject. Data were aligned to movement onset (time 0, vertical dashed line). The vertical solid line indicates average target onset time. The 4 traces were normalized to the maximum of absolute amplitude. There is a clear downward deflection of MEG signals ∼100–200 ms before movement onset. The amplitude of this downward deflection varies across different movement directions. Furthermore, the gray area indicates the average time window used for sensor-space decoding analysis (time of interest) across all subjects. C: same as B, except that the data were from imagined movement and aligned to target onset (time 0, vertical solid line). The vertical dashed line indicates average movement onset time. Position of this MEG sensor is shown in the inset, and the gray area represents the whole head helmet. F, front, L, left, R, right.

Sensor-space decoding analysis

Intended movement direction was decoded from MEG signals recorded from 87 sensors located above the sensorimotor area (Fig. 3, inset). The decoding analysis included three main steps: 1) identification of the time window (i.e., time of interest) during which MEG signals contained the most significant information on intended movement direction using multivariate ANOVA (MANOVA); 2) dimension reduction and data transformation using linear discriminant analysis (LDA) that maximizes linear discrimination among different movement directions; and 3) prediction of intended movement direction using a Bayesian classifier.

Fig. 3.

Modulation of MEG responses examined in 2-D space using the first 2 components of linear discriminant analysis (LDA). The LDA analysis projects activities of all MEG sensors above the sensorimotor area (sensor positions shown in the inset) into a low-dimensional space, where modulation of MEG responses by intended movement direction can be easily visualized. The figure shows a 2-D projection of single-trial MEG responses averaged over −200 to 0 ms relative to movement onset (subject S1). They clearly form 4 distinct clusters corresponding to 4 different intended movement directions. The green plus signs, black squares, blue circles, and red triangles represent single-trial MEG responses for movement in the up, down, left, and right directions, respectively.

SENSOR-SPACE MANOVA.

MANOVA (Johnson and Wichern 1992) is a multivariate extension to ANOVA. Although ANOVA characterizes the difference among groups based on measurement from a single dependent variable (here, 1 MEG sensor), MANOVA examines differences among groups given multiple dependent variables (e.g., multiple MEG sensors). MANOVA was used in this study to quantify the difference among different intended movement directions (groups) by considering recordings from all MEG sensors above the sensorimotor area (dependent variables). In ANOVA based on a single sensor at each time point, we would have a two-way array of sensor values across conditions and across trials within conditions. For our MANOVA, at each time point, we had an analogous two-way array of vectors (a 3-way data array) with each vector containing the signal values at different sensor locations. To obtain the F statistic in MANOVA, it is necessary to calculate the inverse covariance matrix of the sensor signals. The covariance matrices in this case were 87 × 87 and were singular (noninvertible). Principal component analysis (PCA) was used to project the data onto a lower-dimensional space so that the resulting covariance matrix would be invertible. Before computing the principal components, we standardized the signals at each sensor location to have mean 0 and variance 1. MANOVA analysis was repeated across time using 10-ms intervals. For each 10-ms interval, MANOVA analysis was performed on MEG signals recorded during that interval to calculate the corresponding F statistic, which indicates the significance of MEG signal modulation by intended movement direction. The MANOVA F statistic was plotted as a function of time.

Next, the time window (or time of interest) that contains the most significant information regarding intended movement direction was identified based on the temporal profile of MANOVA F statistic. The center of this time window was defined as the time point with the highest F statistic, and the width was kept constant at 200 ms. Additionally, to ensure that the decoding analysis was performed on MEG data not accompanied by overt movement, we limited our search for the time window with strongest directional tuning to the premovement period between target onset and movement onset for overt movement trials. If the length of the time window exceeded movement onset, the excess data were ignored. However, for imagined movement, the search for the time window was not limited to premovement period, because there was no overt movement. Figure 2, B and C, shows the time window identified by this step (averaged across all subjects).

DIMENSION REDUCTION USING LDA.

LDA finds a linear projection of the data that maximizes the variability between groups (i.e., movement directions) relative to that within groups (within directions). For this analysis, MEG signals in each sensor were averaged over the time window identified by the MANOVA analysis, yielding a single time-averaged value for each trial. The time-averaged data were fed into the LDA analysis to project the high-dimensional sensor data into a low-dimensional space (3-D space) that represents MEG signal features with the best linear discrimination among the intended movement directions. Similar to MANOVA, LDA requires the inversion of a within-class scatter matrix (or covariance matrix) that would be singular if the dimension of input variables exceeds the number of trials available. To ensure full-rankness, we projected the within and between class matrices to a lower-dimensional space via PCA before the LDA. Hence, for each trial, the time-averaged MEG signals recorded from all 87 sensors above the sensorimotor area were reduced to a three-element vector by LDA (3 is the maximal dimension given by LDA, because there are 4 different groups or movement directions; see Supplemental Materials for details). For convenience of discussion, we call this three-element vector the MEG response. In addition to facilitating the decoding analysis, the LDA procedure allows the high-dimensional multisensor MEG data to be viewed in 2-D or 3-D space to visualize modulation of MEG signals by intended movement direction.

DECODING USING A BAYESIAN CLASSIFIER.

The posterior probabilities for different movement directions given the MEG response from a single trial were calculated using a Bayesian classifier. Given that LDA in the last step outputs MEG responses in 3-D space, the Bayesian classifier assumes a 3-D Gaussian distribution for MEG responses for a specific movement direction. The decoded movement direction was set to the direction with the highest posterior probability. Leave-one-out cross-validation method was used to estimate decoding accuracy. More specifically, for every iteration of the cross-validation, a new decoding window was found based on the MANOVA only using the training data, and this decoding window was used to test the remaining repetition.

CHARACTERIZING TEMPORAL DYNAMICS OF DECODING ACCURACY.

To further characterize the temporal evolution of decoding accuracy over the course of an overt or imagined wrist movement, instead of using MEG responses within time windows determined by MANOVA, for this analysis, decoding accuracy was calculated with MEG responses within every 10-ms interval and plotted as a function of time.

Localization of active cortical sources

This study localized cortical areas modulated during the behavioral tasks using the minimum current estimates (MCE) algorithm (Uutela et al. 1999) provided by Elekta Neuromag MCE software after co-registering MEG data with structural MRI images for each subject. MCE estimates the best current distribution on the cortical surface (or produces a source image) based on the measured MEG sensor data by constraining the sum of the absolute values of source currents over the cortical surface (Uutela et al. 1999). The MCE software uses a standard brain surface mesh with 853 triangle patches representing 853 cortical areas in a source image, and it returns electrical current values for each of those surface patches. For each movement direction, MEG signals were averaged over all repetitions and fed into the MCE algorithm to estimate active cortical sources within the time window used for sensor-space decoding analysis.

Localization of cortical sources modulated by intended movement direction

In addition to localizing cortical areas that are active during wrist movement, it is more informative to identify cortical areas that actually encode significant information regarding movement direction, i.e., areas showing differential activities for different movement directions. In the last section, MEG data were averaged across all repetitions for each movement direction and fed into the MCE algorithm to obtain one source image for each movement direction. It is not possible to perform statistical hypothesis tests based on this information alone. To overcome this limitation, we applied a bootstrap procedure (Efron 1979) on the sensor data before source localization and generated multiple variations of source images, which allowed statistical testing to be performed in the source space. This procedure was similar to that described by an earlier study except that they used the bootstrap mainly for estimating dipole locations (Darvas et al. 2005). Specifically, for each movement direction, all repetitions were sampled with replacement to create a new data set with the same number of repetitions as the original set (some of the individual repetitions being repeated so that the new random sample was distinct from the original data). This new data set constituted one bootstrap sample. Each bootstrap sample was averaged and fed into the MCE algorithm to generate one estimate of cortical activity or one source image. For each movement direction, the bootstrapping and MCE procedures were repeated 50 times to create 50 bootstrap samples and thus 50 estimates of cortical activity maps. Subsequent source-space statistical analysis was based on these 200 (50 × 4) MCE images (a small sample size was used to reduce computation time, although by examining the samples, the nonzero source currents were found to follow the normality requirement of hypothesis testing as described in the Supplemental Materials).

LIKELIHOOD RATIO TEST.

To examine cortical areas significantly modulated by movement directions, we performed a likelihood ratio test (Behseta et al. 2007) for each of 853 cortical sources (section 2.1 of Behseta et al. 2007). Although ANOVA is typically used for this type of analysis, we used the likelihood ratio test because it does not rely on the assumption of equal variances in MEG data across experimental conditions (i.e., movement directions), thus maximizing our power to identify individual cortical sources that are modulated by movement direction. We performed the likelihood ratio test for each cortical area based on its source currents within the time window used for sensor-space decoding analysis. The null hypothesis was that there would be no difference in the source current under different movement directions. We obtained a χ² test statistic for each source. We further displayed these test statistics on the 3-D cortical surface by mapping the χ² value to each corresponding cortical area. We were thus able to directly visualize regions that have significant modulation.

SOURCE-SPACE MANOVA.

The temporal dynamics of cortical representation of movement direction was further examined in the source space using a MANOVA analysis similar to the sensor-space MANOVA. The only difference is that the source-space MANOVA tests the difference among different movement directions (groups) by considering activities of all cortical sources (dependent variables) instead of MEG sensors. Because the number of cortical sources (853) far exceeded the number of repetitions obtained from bootstrapping (200 repetitions in total, 50 repetitions per movement direction), PCA was applied to the cortical source activity data to reduce the data dimension while preserving 99% of the variance in cortical activity data. Supplemental Fig. S4 shows the number of principal components needed to preserve 99% of the variance in cortical activity. The output from PCA was fed into the MANOVA test. As in the sensor-space MANOVA analysis, the source-space MANOVA F statistic was plotted as a function of time to examine the temporal dynamics of cortical representation of intended movement direction.

RESULTS

Characterizing the sensor-space MEG signals

MEG recordings and structural MRI images were collected from 10 subjects. Each subject made 120 movements per direction for both overt and imagined center-out tasks. After data rejection using EOG, on average, for each subject, 96 ± 29 (SD) repetitions per movement direction were further analyzed in this study. Additionally, one subject (subject S10) was excluded from the data analysis because this subject's eye movement was significantly correlated with target direction. After data alignment based on movement/target onset, each overt movement trial contains MEG signals from 1 s before movement onset to 0.5 s after movement onset, and each imagined movement trial contains MEG signals from 0.5 s before target onset to 1 s after target onset (Fig. 2). MEG signals were smoothed using a 25-ms-wide Gaussian kernel, and the effect of signal smoothing on a typical trial in the overt movement task is shown in Fig. 2. Also shown are the temporal dynamics and modulation by intended movement direction for MEG signals recorded from a single MEG sensor above the sensorimotor area. For overt movement trials, MEG signals remain flat until ∼100–200 ms before movement onset. A prominent negative peak appeared right before movement onset, and the amplitude of this peak varied depending on intended movement direction. A similar negative peak also appeared for the imagined movement task ∼200–300 ms after target onset. This negative deflection in cortical signals was also observed in previous EEG and MEG studies, and it has been suggested that this waveform is related to the readiness potential, which might be modulated by movement direction (Brunia et al. 2000; Kornhuber and Deecke 1965; Waldert et al. 2008).

Sensor-space decoding analysis

The sensor-space MANOVA analysis was used to identify the time window (or time of interest) during which MEG signals encoded the most significant information for decoding intended movement direction. This search for the decoding time window was restricted to the premovement interval, as explained in methods. The gray areas in Fig. 2, B and C, show the average time windows identified by the sensor-space MANOVA test for overt and imagined movements across all subjects. For each trial, LDA was used to transform MEG signals recorded from all 87 sensors above the sensorimotor areas to a three-element vector, the MEG response, which best summarizes or depicts modulation of MEG signals by movement direction. Figure 3 shows the MEG responses in 2-D space in a time window from 200 to 0 ms before the onset of overt movement. The MEG responses clearly clustered into four different groups, corresponding to four different movement directions. This figure shows that sensorimotor cortical activity captured simultaneously by multiple MEG sensors was modulated by intended movement direction and that a classifier can be used to decode intended movement direction from MEG responses. Furthermore, within each cluster, MEG responses tended to be normally distributed. Hence, a multivariate Gaussian distribution was used to model MEG responses for decoding analysis with a Bayesian classifier. Table 1 shows the results for single-trial decoding using leave-one-out cross validation for all the subjects. The chance level is 25%. For overt movement, the intended movement direction was predicted with an average accuracy of 67% using MEG data within a short time window before movement onset. Similarly, an average decoding accuracy of 62.5% was achieved using sensorimotor cortical activity recorded with MEG when subjects simply imagined wrist movement without any overt movement or muscle contraction.

Table 1.

Summary of single-trial decoding accuracies (%) across all 9 subjects for overt and imagined movements

	S1	S2	S3	S4	S5	S6	S7	S8	S9
Overt	88.6	69.0	72.4	77.8	57.9	57.0	51.3	82.2	56.6
Imagined	95.0	69.9	71.9	45.8	62.5	39.6	73.7	44.3	45.2

Subjects S1–S5 performed the nondelayed center-out task, and subjects S6–S9 performed the delayed center-out task. MEG sensors above the sensorimotor area were used. The time window for decoding analysis was determined based on the sensor-space MANOVA analysis. MEG signals within that time window were averaged. Intended movement direction was decoded using a Bayesian classifier with leave-one-out cross validation. The chance level is 25%. The average decoding accuracies across all subjects are 67 and 62.5% for overt and imagined movement, respectively. MEG, magnetoencephalography; MANOVA, multivariate ANOVA.

While Table 1 shows the decoding analysis results using specific time windows that center on the moments when MEG signals were the most discriminative for intended movement direction, it is desirable to see how decoding accuracy actually evolved over the course of an overt or imagined movement and compare temporal dynamics of decoding accuracy between the nondelayed and delayed tasks. For the overt nondelayed task (Fig. 4A), the decoding accuracy started near the chance level, and it increased substantially before movement onset, suggesting that a significant amount of information regarding the intended movement was represented in sensorimotor cortical activity well before actual movement was initiated, as described in previous animal studies using single-neuron recordings (Moran and Schwartz 1999; Santhanam et al. 2006; Wang et al. 2007). Similarly, for the imagined nondelayed task, the decoding accuracy increased above the chance level shortly after target onset, and it stayed above the chance level throughout the movement. For the delayed tasks, although their decoding accuracies rose above the chance level after target onset, they increased more gradually over time compared with the nondelayed tasks. For both overt and imagined movements, the decoding accuracy for delayed movement is typically lower than for nondelayed movement. This is accompanied by the absence of fast increase in decoding accuracy, as seen in Fig. 4, C and D. As more time is allowed for motor planning during the delay period, neuronal processing of information about intended movement may be spread over a longer period. This potentially leads to weakened macroscopic MEG signals, making it harder to detect population activities related to intended movement.

Fig. 4.

Temporal dynamics of decoding accuracy during center-out movement. Decoding accuracies were calculated every 10 ms using MEG data within that 10-ms interval. A: overt nondelayed. B: imagined nondelayed. C: overt delayed. D: imagined delayed. For each plot, the thin gray lines represent decoding accuracies for individual subjects, and the thick black line represent the decoding accuracy averaged over all subjects. Time 0 corresponds to movement onset for overt movement and target onset for imagined movement. For movement with delay (C and D), average delay periods were marked with double-arrow lines.

Source-space analysis

Decoding analysis showed that a significant amount of information can be extracted from MEG sensor signals to predict intended movement direction in the absence of overt movement. From a pure engineering perspective, this result, to a certain degree, is sufficient for developing an MEG-based BCI system. However, from a neuroscience perspective, it is worth further investigating and understanding where are the actual cortical sources that lead to modulated MEG sensor signals and make it possible to decode intended movement direction from MEG sensor signals. Figure 5 shows the source localization results obtained from the standard MCE algorithm, which inversely maps whole head MEG recordings from 306 channels to the cortical surface of a standard brain with 853 cortical sources (surface patches). The color of each patch represents the intensity of cortical source activity during the time window used for decoding analysis in the sensor space (overt movement: t = −200 to 0 ms, relative to movement onset; imagined movement: t = 300–420 ms, relative to target onset). For overt movement, the contralateral (left) motor cortical area was strongly activated during wrist movement, accompanied by weaker activity in ipsilateral motor area. In addition, both the visual and prefrontal cortical areas were also activated, because the task is visuomotor in nature. More interestingly, a very similar cortical activation pattern was observed for the imagined movement as well, with a clear motor cortical activation, although to a lesser extent. This suggests that imagining wrist movement actually engages cortical areas and processes very similar to those used by overt wrist movement and is consistent with previous findings of motor cortical activation during motor imagery (Crammond 1997; Porro et al. 1996; Sabbah et al. 1995; Stephan et al. 1995).

Fig. 5.

Localization of active cortical areas for overt (A and C) and imagined (B and D) movements. Minimum current estimation (MCE) was used to localize active cortical sources during the time period used for decoding analysis in subject S1. Data were averaged over all repetitions for leftward movement. A and C: cortical activity from −200 to 0 ms relative to the onset of overt movement. B and D: cortical activity from 300 to 420 ms relative to target onset for imagined movement. The contralateral motor cortical area shows strong activation. In addition, there is a certain degree of visual and ipsilateral motor cortical activity.

While Fig. 5 shows cortical areas that were active during overt and imagined wrist movements, it does not tell us what information is being represented in each cortical area and which cortical area is actually encoding intended movement direction. Through a combination of bootstrapping and MCE, we obtained multiple estimates of cortical activity maps for each movement direction, allowing statistical analysis that answers both questions together. Cortical activity within the time window used for decoding analysis in the sensor space were examined to further localize cortical areas that encode significant information on intended movement direction. A likelihood ratio test was performed for each cortical source. The resulting χ² statistics from all cortical sources were mapped to a 3-D cortical surface plot. Each surface patch represents 1 of the 853 cortical sources, and the color of each surface patch represents its χ² statistic (Fig. 6). The modulation is bilateral, with stronger modulation for the contralateral (left) hemisphere. The contralateral motor cortex showed the strongest modulation by intended movement direction for both overt and imagined wrist movement. The visual cortices also showed a certain degree of modulation, but weaker than the motor cortical area. Interestingly, the left superior temporal lobe and the left inferior frontal gyrus, part of which overlaps the Broca's area, also showed strong modulation by intended movement direction during the overt movement task.

Fig. 6.

Localization of cortical sources modulated by intended movement direction. Combination of MCE with bootstrapping created multiple estimates of cortical activity for each movement direction, and a likelihood ratio test was used to examine whether a cortical source (a triangular surface patch on brain surface plots) was modulated by intended movement direction. Surface patch color represents the significance of modulation, as measured by the χ² statistic calculated within the time period used for decoding analysis in subject S1. Hotter (red) color indicates stronger modulation. A and C: left/contra and right/ipsilateral hemisphere activity during overt movement. B and D: left/contra and right/ipsilateral hemisphere activity during imagined movement. P values of 10⁻³ and 10⁻⁵ correspond to χ² statistics of 16.2 and 25.9, respectively. The contralateral motor cortical area is significantly modulated by intended movement direction. In addition, the visual cortical areas and the left inferior frontal gyrus also show modulation.

While Fig. 6 shows the modulation of cortical activity within specific time windows used for decoding analysis, this study further examined temporal dynamics of cortical activities representing intended movement direction using the source-space MANOVA analysis, as shown in Fig. 7 (results for similar analysis in sensor space are provided in Supplemental Fig. S2). Different from single-source analysis using tests like ANOVA or the likelihood ratio test used in Fig. 6, a MANOVA test is able to better depict the overall modulation of all cortical sources by intended movement direction, i.e., the “population response.” Figure 7, A and B, shows the temporal evolution of source-space MANOVA F statistic. F statistic indicates strength of cortical activity modulation by movement direction. For overt wrist movement, the F statistic shows a very prominent peak ∼100–150 ms before movement onset, similar to firing rate patterns of many motor cortical neurons observed in previous nonhuman primate studies. This interval also agrees with the neural delay between motor cortical representation of movement and its actual expression reported by those animal studies (Moran and Schwartz 1999; Wang et al. 2007). Similarly, for imagined wrist movement, the F statistic peaks ∼400 ms after target onset and before the start of computer-controlled cursor movement (the cursor starts moving after a simulated reaction time that varies from 0.5 to 1 s). In summary, the source-space statistical analysis suggests that the modulation seen in the sensor space is largely caused by modulated activity in motor cortical areas. It also shows that temporal dynamics of cortical activity representing intended movement direction as recorded noninvasively using MEG in humans agrees well with previous findings using microelectrodes implanted in the motor cortex of nonhuman primates.

Fig. 7.

Temporal dynamics of cortical activity modulation by movement direction. The source-space multivariate ANOVA (MANOVA) test was performed in 10-ms intervals to characterize movement modulation of cortical activity, and the resulting F statistics were plotted as a function of time for overt (A) and imagined (B) movements. P value of 10⁻⁵ approximately corresponds to an F statistic of 25.3. Thin gray lines represent the F statistics for individual subjects (nondelay task), and the thick black lines represent the average over all subjects. Cortical activity becomes highly modulated right before movement onset for overt movement and 300–400 ms after target onset for imagined movement.

DISCUSSION

Encouraged by previous studies that decoded movement direction from MEG recordings of brain activity in humans performing overt movement (Georgopoulos et al. 2005; Waldert et al. 2008), this study was designed to further our understanding in using MEG to study cortical representation of intended movement and its application in BCI. This study has several features. First, the classic center-out task was used, allowing direct comparison between cortical activities recorded noninvasively in humans using MEG with neuronal activities recorded invasively in nonhuman primates using microelectrodes. The behavioral paradigm also made it possible to directly compare cortical activities between overt and imagined movements. Second, statistical techniques used in this study facilitated data analysis in visualizing and understanding MEG data, which typically have a high number of dimensions but a low signal-to-noise ratio. Third, this study showed the feasibility of decoding intended movement direction from MEG signals without any overt movement from subjects. Last, this study applied source localization techniques to inversely map MEG sensor signals to cortical activity, and source-space statistical analysis quantitatively identified cortical sources that encode significant information about intended movement direction.

MCE with bootstrapping for source-space statistical analysis

Source localization algorithms, such as MCE, map whole head MEG recordings to cortical activity. However, given the low signal-to-noise ratio of MEG recordings, to get a robust estimate of cortical activity, the general practice is to use averaged MEG data instead of single-trial data for source localization, which makes it difficult to perform statistical analysis in the source space. This study introduced the MCE with bootstrapping technique that enables statistical analysis in the source space (Fig. 6). Furthermore, this technique itself is not bound to MCE, and it is applicable to any source localization algorithm. A similar technique was used to examine dipole localization accuracy (Darvas et al. 2005).

Decoding intended movement direction from MEG signals during motor planning and motor imagery

Considering that targeted users of BCI systems are individuals with difficulties in generating overt limb movement, it is desirable to show that movement information can be decoded from cortical activity in the absence of overt movement. A recent EEG study in individuals with ALS has shown that the speed of imagined wrist movement can be decoded from EEG signals (Gu et al. 2010). Using a real-time MEG-based BCI system, Buch et al. (2008) enabled individuals with hand paralysis to control the opening and closing of a hand orthosis by modulating their sensorimotor rhythm with imagined hand movement. Our study focused on the time domain signal of MEG recording, which could be related to the lateralized Bereitschaftspotential (LBP)/readiness potential and readiness field (Deecke et al. 1982; Kornhuber and Deecke 1965; Pedersen et al. 1998; Takahashi et al. 2004). It was suggested that characteristics of LBP could be modulated by various movement parameters (Birbaumer et al. 1990), and several groups have shown that laterality of intended movement can be decoded reliably from LBP (Brunia and van den Bosch 1984; Coles et al. 1990). This study further showed that intended movement direction can also be decoded from MEG signals during the planning period before movement and EMG onset and during motor imagery using only MEG sensors above the sensorimotor area. From a neural engineering perspective, this study suggests that it may be possible for a user to operate an MEG-based BCI system by imagining moving the cursor toward the desired direction. From a neuroscience perspective, decoding analysis performed in this study provides strong evidence that intended movement direction is well represented in cortical activity recorded noninvasively by MEG during motor planning and motor imagery.

Modulation of neural activity providing information about movement direction evolves over different time scales in the delay and nondelay tasks (Fig. 4). Decoding accuracies in the delayed task for both overt and imagined conditions show more gradual changes and peak at lower values than those of the nondelay task. This is possibly because of temporal dispersion of cortical processes related to motor planning and motor execution during the delayed task (Churchland et al. 2006; Crammond and Kalaska 2000; Flanders et al. 1992; Ghez et al. 1997). This temporal separation may spread movement-related information over delay and movement time, reducing the amount of information available for movement decoding at any instant of time. On the contrary, cortical processes for motor planning and execution might overlap in the nondelayed task, leading to stronger cortical signals for decoding movement direction. The difference in temporal profiles of decoding accuracies between delayed and nondelayed tasks is certainly worth further examination in the future. Nevertheless, this study showed that movement direction can be decoded in the absence of overt movement in both delayed and nondelayed tasks.

Source-space analysis identifying cortical sources for MEG signal modulation

This study further provides answers to what type of information is being encoded by active cortical areas and when movement-related information is best represented in cortical activity. The contralateral (left) motor cortex clearly encodes intended movement direction (Fig. 6; see Supplemental Fig. S3 for a side-by-side comparison between Figs. 5 and 6). Interestingly, the left inferior frontal gyrus (LIFG) also exhibits a certain degree of modulation by intended movement direction. This area has been previously suggested to be part of the human mirror-neuron system, which is critical for recognition of movement intention (Rizzolatti and Craighero 2004).

F statistic in the source-space MANOVA reflects the degree to which cortical activity differs across different movement directions. The temporal profile of F statistic from source-space MANOVA peaks right before movement onset, suggesting that this is the moment when cortical activity varies the most for different movement directions (Fig. 7). Previous nonhuman primate studies showed that firing rates of a typical motor cortical neuron also varies the most for different movement directions near movement onset. The firing rate reaches its maximum and minimum around onset of movements in the neuron's preferred and antipreferred directions, respectively (Crammond and Kalaska 2000; Moran and Schwartz 1999; Wang et al. 2007). The temporal profile of source-space F statistic in Fig. 7 shows that movement-related cortical activity recorded with MEG shows a temporal pattern very similar to that of motor cortical neuronal firing rates.

Future work and summary

Previous studies have shown that the time-domain signal, especially the LFC, also called LMP, contains significant information regarding movement direction (Ball et al. 2009; Georgopoulos et al. 2005; Pistohl et al. 2008; Schalk et al. 2007; Waldert et al. 2008). The capability of this study to decode intended movement direction from time-domain MEG signals in the absence of overt movement further suggests that LFC/LMP signals carry significant movement-related information. It is worth studying whether other MEG signal features are also modulated by intended movement direction, because previous studies suggested that various frequency bands contain significant movement-related information (Ball et al. 2009; Cheyne et al. 2008; Crone et al. 1998; Heldman et al. 2006; Jurkiewicz et al. 2006; Leuthardt et al. 2004). Future studies can use frequency-domain source localization techniques to further explore movement information embedded in various frequency bands of MEG signals (Gross et al. 2001; Jensen and Vanni 2002).

It will be interesting to directly apply source localization algorithms on a single-trial basis, as done in several previous studies (Gaetz and Cheyne 2003; Jerbi et al. 2007), and to perform source-space decoding analysis. One question is whether decoding accuracy will be improved if movement direction is decoded in the source space rather than in the sensor space. Arguably, in the source space, better feature selection procedures may be performed based on both statistical analysis and prior knowledge on functions of various cortical areas. This may help remove noise and artifacts in the MEG data and lead to better decoding accuracy.

The capability to decode intended movement direction from MEG signals may enable a noninvasive high-performance BCI system based on MEG. Although debatable, several previous studies have suggested that MEG and ECoG might share similar spatial and temporal characteristics in terms of source localization accuracy and capability to resolve cortical activity represented by amplitudes of different frequency bands (Dalal et al. 2008; Gharib et al. 1995; Korvenoja et al. 2006). Although it seems clear that both MEG and ECoG signals contain movement-related information, future studies should examine the efficacy of MEG for potential BCI and rehabilitation applications. For example, although MEG itself is not portable, it might serve as a powerful tool for presurgical localization of cortical areas that are significantly modulated by intended movement direction, and intracranial electrodes can be implanted at those cortical sites.

It can be challenging to accurately localize the motor cortex after cortical reorganization induced by cortical lesions (e.g., caused by stroke) or spinal cord injury (Cramer 2008; Kokotilo et al. 2009). However, previous studies have shown that individuals with stroke or spinal cord injuries can at least partially activate their sensorimotor cortical areas with attempted or imagined movement (Buch et al. 2008; Shoham et al. 2001; Wang et al. 2010), and MEG may be indicative of cortical areas whose activity can be modulated volitionally by an individual for BCI control. Furthermore, even with intact brain, given that the implanted electrode array will typically cover a very small cortical area [e.g., 4 × 4 mm² for the intracortical microelectrode arrays (Hochberg et al. 2006) and ∼15 × 15 mm² for a minimally invasive custom-ECoG grid that we have been studying (Wang et al. 2009)], it is still desirable to accurately localize the targeted implantation site (e.g., hand area of the motor cortex) before surgery. The goal of this study is to develop the appropriate behavioral paradigms and algorithms that will allow us to accurately identify the cortical area that contains information about intended movement direction given the potential variability in cortical organization across individuals. Future studies should be conducted in individuals with motor impairments (i.e., targeted end users of BCI), and the paradigms developed in this paper provide the basis for those studies.

In addition, users can be trained noninvasively with an MEG-based BCI system before undergoing surgery. This training can at least familiarize users with the BCI paradigm, and it can potentially improve cortical activity modulation for BCI control. Furthermore, by providing real-time feedback of cortical activity, MEG-based BCI systems can serve as a rehabilitation tool for promoting cortical plasticity after stroke and spinal cord injury, which is another area of great scientific and clinical value (Birbaumer and Cohen 2007; Buch et al. 2008; Wang et al. 2010). These functionalities demand real-time acquisition, processing, and decoding of MEG signals, which we are actively working on. We developed a new real-time software package (rtMEG) for real-time MEG signal streaming directly from the real-time digital signal processing (DSP) units of our MEG system (//www.elekta.com/) with a minimal delay (Sudre et al. 2010). This software is well integrated with BCI2000, a general purpose software package for real-time BCI research (Schalk et al. 2008). We also developed a novel real-time MEG signal processing algorithm, robust SSS (rSSS), for on-line MEG signal filtering and noise removal (Guo et al. 2010).

In summary, this study showed that intended movement direction can be decoded from human cortical activity recorded noninvasively using MEG during motor planning and motor imagery, and source space analysis showed that the contralateral motor cortex has the strongest modulation by intended movement direction. This modulation is the most significant immediately before movement onset, with a temporal profile similar to that of motor cortical neuronal activities observed in previous nonhuman primate studies and also present during imagined movement. This representation of intended movement in human cortical activity can serve as a critical neural substrate for brain-controlled interface applications.

GRANTS

This work was partially supported by the National Science Foundation under Cooperative Agreement EEC-0540865, Telemedicine and Advanced Technology Research Center of U.S. Army Medical Research and Material Command Agreement W81XWH-07-1-0716, and Grant 5 UL1 RR024153 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH) and NIH Roadmap for Medical Research. This work is also supported by a special grant from the Office of the Senior Vice Chancellor for the Health Sciences at University of Pittsburgh, as well as a student travel fund from Center for Neural Basis of Cognition. Additional funding support was provided by NIH Grants 1R01 EB-007749 and 1R21 NS-056136 to D. J. Weber and R01 MH-064537 and R01 EB-005847 to R. E. Kass and Y. Xu.

DISCLOSURES

The contents of this work are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or National Institutes of Health. Information on NCRR is available at //www.ncrr.nih.gov/. Information on Re-engineering the Clinical Research Enterprise can be obtained from http://nihroadmap.nih.gov/clinicalresearch/overview-translational.asp. No conflicts of interest, financial or otherwise, are declared by the authors.