Brain–machine interface via real-time fMRI: Preliminary study on thought-controlled robotic arm

Abstract

Real-time functional MRI (rtfMRI) has been used as a basis for brain–computer interface (BCI) due to its ability to characterize region-specific brain activity in real-time. As an extension of BCI, we present an rtfMRI-based brain–machine interface (BMI) whereby 2-dimensional movement of a robotic arm was controlled by the regulation (and concurrent detection) of regional cortical activations in the primary motor areas. To do so, the subjects were engaged in the right- and/or left-hand motor imagery tasks. The blood oxygenation level dependent (BOLD) signal originating from the corresponding hand motor areas was then translated into horizontal or vertical robotic arm movement. The movement was broadcasted visually back to the subject as a feedback. We demonstrated that real-time control of the robotic arm only through the subjects’ thought processes was possible using the rtfMRI-based BMI trials.

Translation of brain activity into direct control of mechanical hardware or computer components without the involvement of the peripheral nervous system or muscle is sought after to provide control options for paraplegic patients, to apply to consumer electronics products, and for use in other applications [4,12,18]. The techniques that enable this translation are referred to as brain–machine interface (BMI) or brain–computer interface (BCI) [4]. The traditional measurement of brain activity, in the context of BMI, has relied on the acqusition of electroencephalography (EEG) data via either invasive or non-invasive electrode arrays. The acquired EEG data is analyzed in either the temporal or frequency domains and subsequently translated into appropriate control commands [12,25].

Functional magnetic resonance imaging (fMRI) has provided alternative methods for BMI based on its capability to reveal neuronal activation with superior spatial localization in a non-invasive manner. More specifically, real-time fMRI (rtfMRI) allowed for feedback of region-specific status of brain activations to an individual and, subsequently, empowered individuals to learn to modulate brain function involved in attention, the perception of pain, and the experience of emotions [3,19,23,27]. Recent studies have also shown that fMRI successfully translated brain activity into specific control commands such as computer cursors [26] and the movement of robotic fingers [14]. Goebel et al. [10] used blood oxygenation level dependent (BOLD) signals originating from the regions-of-interest (ROIs) which were best regulated by the subject during the control of the linear cursor movement in a video game setting. Based on invasive electrophysiological recordings [12,21], BMI has been implemented in animals and humans. However, control of a robotic arm using only thought processes has not yet been demonstrated in the context of a non-invasive rtfMRI approach.

Using rtfMRI as a signal detector, we tested the feasibility of linking BOLD activity originating from the motor cortex to the movement of a robotic arm during the performance of a hand motor imagery task. Utilization of the imagery task is important to ensure that only thought processes, without any overt muscle movement, control the robotic arm. Unlike our previous BMI implementation, whereby the interpretation of spatial distribution of the activation pattern resulted in the execution of predetermined sets of cursor commands [26], the current method allows for the real-time control of range of motion and/or direction of the robotic arm. The BOLD signals measured from the hand motor areas in the primary motor cortex (M1; dorsal part of the precentral gyrus and the anterior bank of the central sulcus) of the right and left hemispheres were used to adjust the level of vertical and horizontal movement of a robotic arm, respectively. The movement of the robotic arm was visually fed back to the subject, thus permitting the subject to adjust the level of cortical activation in an attempt to regulate the movement of the robotic arm in two-dimensional space.

Three right-handed volunteers (29-year-old female; 35- and 38-year-old males) participated in the study according to the ethical guidelines set forth by the institutional review board. One subject (S1) had previous experience participating in fMRI studies involving the administered hand motor imagery tasks. The two remaining subjects (S2 and S3) had no prior experience or involvement in fMRI studies. Using a 3-T clinical scanner (Signa VH, GE) and a single-channel head coil, volumetric images were acquired using a standard gradient echo planar imaging (EPI) pulse sequence (TR/TE = 1000/35 ms; FA = 80°; FOV = 24 cm × 24 cm; 13 axial slices; 64 × 64 voxels within a slice; 3.75 mm × 3.75 mm in-plane resolution; 5 mm slice thickness; 1 mm gap). The time delay between EPI data acquisition and reconstruction into a DICOM file format in MR scanner and subsequent transfer and generation of a 3D data matrix to a data processing computer (laptop, Intel Core 2 Duo 2.2 GHz; 2GB of RAM) was negligible (324 ± 153 ms/volume; not accumulative) compared to that of the repetition time (TR = 1 s).

A robotic arm (MR-RA04S, Minirobot; www.minirobot.co.kr) with two-directional degrees of freedom (i.e. horizontal and vertical/azimuthal) was used. The range of horizontal and vertical angular movements was 90° and 80°, respectively. Three targets were located at each of the end points in the horizontal, vertical, and diagonal directions from the virtual origin (Fig. 1a). The interfaces for both the control of the robot and the data processing were developed using the MATLAB (v7.1 R14; Mathworks, Natick, MA) computing environment. A command to control the robot was transmitted via the serial port connection using a RS-232 cable (9600 baud rate). Fig. 2 shows a schematic diagram of the overall study procedure. The experiment consisted of two separate types of trials: (1) fMRI trials for the definition of regions-of-interest (ROIs) and (2) rtfMRI trials for the direct control of the robotic arm. The term ‘trial’ is consistently used in the following texts in order to describe one continuous acquisition of EPI data.

Fig. 1.

(a) Robotic arm along with three pre-defined targets. (b) Block-based design employed during the ROI trial. (c) Illustration of a link between the BOLD signal change and the angular movement parameter of the robotic arm. (d) Schematic representing time lag of feedback during an rtfMRI trial excluding the dummy and baseline period (A: data processing, 1 s; B: horizontal robot move, 2 s; C: vertical move, 2 s).

Fig. 2.

A flow diagram of an overall study design.

Two types of hand motor imagery tasks (right-hand: ‘RH’; left-hand: ‘LH’) were administered based on the block design shown in Fig. 1b (scan duration = 2 min 30 s). For each hand motor imagery task, subjects were instructed to imagine clenching their hands independently at a rate of 2 Hz as realistically as possible. The presence of any overt hand motion was inspected visually.

The fMRI data (DICOM file format) reconstructed in a control console were immediately transferred to the data processing computer via file transfer protocol (FTP). The fMRI data was then formed as a 4D matrix of volume time series (140 volumes excluding the first 10 volumes to equilibrate a T₁-effect). Motion artifacts after the first volume were corrected as implemented in SPM2 (www.fil.ion.ucl.ac.uk). Each of the realigned EPI volumes were spatially smoothed using an 8 mm isotropic full-width-at-half-maximum (FWHM) Gaussian kernel to reduce spatial noise. The resulting 4D fMRI data was then analyzed using a cross-correlation method whereby the voxel-wise correlation coefficients (CC) were calculated between the BOLD time series and the paradigm-related canonical hemodynamic response function (HRF) obtained from SPM2. After applying a statistical threshold (i.e. p-value) to visualize the activation patterns from the CC maps, a single voxel showing a maximum CC value of activation was automatically selected within the corresponding M1 region as guided by the registration [15] of the motor areas defined in the anatomical automatic labeling (AAL) map [22]. The voxels that were mutually activated by both tasks were avoided to reduce the cross-talk between BOLD signals from each task. As a result, the selected voxel, along with the four neighboring voxels (anterior/posterior/left/right) that were exclusively activated, were defined as an ROI. For quality assurance, the activation patterns of the CC maps could be displayed on the laptop for various statistical threshold levels (choices: p < 0.05 and p-values from p < 10⁻² to p < 10⁻⁷ in 10⁻¹ scale).

During the subsequent real-time fMRI trial, the subject was instructed to freely engage both imagery tasks with the intention of hitting the three pre-defined targets by moving the robotic arm. The movement in EPI data was not corrected during the rtfMRI trials; however, potential head motion was passively restrained using a pad molded around the participant’s head. The same imaging parameters and scan duration (150 s) used in the ROI trial were employed. After 20 s, which included a dummy scan period (10 s) and a rest period for baseline intensity detection (10 s), a sound cue signaled the subjects to begin the tasks. Then, the subjects performed each trial within a predetermined time period (130 s). Once they reached the target, the subjects were asked to maintain the robotic arm at the target. Four trials for each of the three targets were administered.

The BOLD signal originating from the ROIs was linked to the movement parameters of the robotic arm. To do so, the averaged BOLD intensities during the 10 s of rest period after the dummy volumes were calculated from two pre-defined ROIs and served as the baseline BOLD intensities. The percent BOLD intensities within the ROIs (i.e. levels of engagement to the RH and LH tasks by the subject) were averaged again for every 8 s interval and were translated into the horizontal and vertical movements of the robotic arm, respectively (Fig. 1c). The temporal averaging of each 8 s interval was used to reduce erratic robotic movement as well as to account for a maximum delay caused by two directional movements of the robotic arm. Since there was only one block of task period conducted during each trial without prior knowledge of the BOLD signal behavior, the potential presence of low-frequency drift (typically less than 0.015 Hz [20]) was not corrected.

The movement parameters for the robot, such as medium (noted as ‘Med’) and maximum (noted as ‘Max’) percent BOLD intensities and the corresponding degree of angular movement (noted as ‘Step’), were adjustable depending on the subject’s level of BOLD contrast. The default values were set as Med = 0.5%, Max = 1.0%, and Step = 20° for both tasks. For example, if the subject modulates the BOLD activity of 0.75%, the robot arm moves 10°. If the subject rests (i.e. BOLD = 0%), the robot arm moves to −20°. The movement of the robotic arm was recorded using a webcam (TrackerPod, Eagletron, NY) and was shown to the subject through MR-compatible visual goggles (Resonance Technology, CA) in real-time. As illustrated in Fig. 1d, the time lag between the acquisition of 8 EPI volumes of each time frame and the data processing in the laptop (including DICOM image transfer, 3D data matrix reconstruction, and calculation of percent BOLD intensities within the ROIs) was about 1 s. The maximum time lag between sending the command to the robot and ending of the robotic movements was on the order of 4 s. At the end of each trial, each subject was asked to report his or her task strategy.

A Monte-Carlo computational method [9] was used to assess the probability of hitting the target by chance. Two standard probability density functions (i.e. Uniform and Gaussian/Normal distributions) of a percent (%) BOLD intensity within each ROI were considered to estimate a chance level in simulated task conditions. More specifically, a potential amount of the percent BOLD intensities of each 8 s time frame (16 total time frames for each trial) were randomly sampled from three different scenarios for each of the two types of probability distributions (see Table 3 for the scenarios). A total of 100,000 “trials” were simulated for each scenario.

Table 3.

The chance levels for hitting a target from the Monte-Carlo method using the percent BOLD intensities (compared to a baseline intensity) simulated from three scenarios of either Uniform or Gaussian distribution (STD: standard deviation).

A scenario of the percent (%) BOLD Intensities	Uniform distribution			Gaussian distribution (mean = 0.5%)
	−0.5 to 1.5%	−1.0 to 2.0%	−1.5 to 2.5%	STD = 1.0%	2.0%	3.0%
Chance Horizontal target	0.2	1.9	3.9	2.8	7.9	10.0
Level Vertical	0.5	3.3	6.3	4.8	11.1	13.9
(%) Diagonal	0.0	0.3	1.3	0.7	4.0	5.7

In order to investigate the short-term learning effect of increased control of motor imagery activations in each individual subject, an additional fMRI trial was administered for each of the RH and LH tasks using the block-based task-paradigm (Fig. 1b). The realigned EPI volumes were further normalized into the Montreal Neurological Institute (MNI) space followed by spatial smoothing using an 8 mm isotropic FWHM Gaussian kernel. The resulting normalized EPI data were analyzed using the general linear model (GLM) method [6]. From the GLM-based activation maps (p < 0.001), the number of active voxels from the primary motor cortex (M1; the precentral gyrus as identified by the automated anatomical registration [15]) was counted from both hemispheres. The corresponding laterality index (LI), which reflects the hemispheric dominance of activation, was then calculated (see Table 4 for the description) to examine the subjects’ ability to regulate regional BOLD signals within the M1 region.

Table 4.

The number of active voxels (p < 0.001) within the primary motor cortex (M1) along with a corresponding laterality index (LI) defined as (L − R)/(L + R), where L and R denote the number of active voxels within the left- and right-hemispheres, respectively. The ‘*’ symbol was used when there were no active voxels.

	Pre-RH		Post-RH		Pre-LH		Post-LH
	L/R	LI	L/R	LI	L/R	LI	L/R	LI
S1	1101/55	0.90	1198/153	0.77	1059/1546	−0.19	309/1139	−0.57
S2	606/281	0.37	422/126	0.54	404/799	−0.33	21/57	−0.46
S3	77/0	1.00	6/11	−0.29	0/0	*	1/283	−0.99

The CC maps that were acquired during the ROI trial from subject 1 (S1) represented highly task-specific activities within the contralateral side of the motor areas. Consequently, the voxel showing the maximum CC value in M1 was successfully selected for each task and was defined as the center of the ROI (CC values of 0.73 for the RH task and 0.90 for the LH task; p < 10⁻¹⁰). The results obtained during the BMI trials are illustrated in Fig. 3. The subject successfully achieved the targeted tasks in all four trials for the vertical target and in three out of four trials for both the horizontal and diagonal targets (the 1st trial missed the target goal). The average numbers of movement steps to reach each target for this subject were 8.7 ± 3.2 (horizontal), 5.5 ± 1.7 (vertical), and 8.3 ± 2.9 (diagonal). Note that during the 2nd and 3rd trials for the diagonal target, the subject intentionally moved the robotic arm after passing the horizontal and vertical targets.

Fig. 3.

The trajectories of robot movements during the rtfMRI trials for subject 1 ((a) horizontal; (b) vertical; (c) diagonal), along with time plots of the percent BOLD signals within the ROIs (sampled from the 4th trial). The horizontal/vertical direction of the 3D arrow indicates the direction of the robotic arm movement. The labels of the unsuccessful 1st trials, where the horizontal and diagonal targets were not reached, are underlined.

From a retrospective analysis of the percent BOLD levels during the 4th trials, the average percent BOLD intensity during each 8 s time-window (thick line) reliably reflected the subject’s intention to move the robotic arm to each target. For example, during the attempt to reach the vertical target in the 4th trial, the neuronal activities within both ROIs increased in the beginning of the trial (moving the arm diagonally). However, after several movements (approximately 70 s later) the subject reduced the RH-related activities (for horizontal movement) while maintaining only the LH-related activities (for vertical movement). This illustrates the subject’s ability to regulate the level of BOLD signal based on visual feedback of the robotic movements.

The defined ROIs from the remaining two subjects are shown in the left column of Fig. 4. The CC values of the voxels in the center of the ROIs were 0.57 (RH; p < 10⁻¹⁰) and 0.40 (LH; p = 9.8 × 10⁻⁷) for subject 2 (S2) and 0.32 (RH; p = 1.2 × 10⁻⁴) and 0.18 (LH; p = 0.034) for subject 3 (S3). During the subsequent rtfMRI BMI trial, the numbers of successful trials for each target from S2 were 1 for the horizontal target (2nd trial; 13 moves), 3 for the vertical target (2nd, 3rd, and 4th; {10 + 6 + 5}/3 = 7 moves), and 2 for the diagonal target (1st and 3rd; {9 + 8}/2 = 8.5 moves). From S3, these numbers were 1 for the horizontal target (3rd; 6 moves), 1 for the vertical target (4th; 4 moves), and 3 for the diagonal target (1st, 3rd, and 4th; {14 + 7 + 5}/3 = 8.7 moves).

Fig. 4.

The defined ROIs within motor areas and illustrations of robotic movements from the successful trials for (a) subject 2 and (b) subject 3. The overlapped activation areas are depicted in green while the activation areas from the RH and LH tasks are depicted in red and blue, respectively. Five voxels of ROI for each task are within a yellow circle. From the registration of the motor areas using the AAL template, the primary motor cortex (M1) and primary somatosensory cortex (S1) were shown as magenta and cyan (light-blue), respectively. (For interpretation of the references to color in this figure legend, please refer to the web version of the article.)

The results pertaining to the retrospective evaluation of the presence of head motion (the translational and rotational movement parameters) are shown in Table 1. The head movements were measured to be less than the size of a voxel (3.75 mm × 3.75 mm × 5 mm) for all of the subjects. The number of successful rtfMRI trials after the retrospective correction on head motion and/or low-frequency drift is summarized in Table 2. For the subject S1, the retrospective motion and/or drift correction of the EPI volumes from the rtfMRI BMI trials did not significantly alter the results. On the other hand, for the subjects S2 and S3, the number of successful trials were maintained or even reduced (except for a case of the vertical target task from S3).

Table 1.

Subject head movements during the rtfMRI trials (mean ± standard deviation; 6-parameter affine rigid-body transformation with 4th degree B-spline interpolation re-slicing/re-sampling in SPM2).

Subject index	Translation (mm)			Rotation (°)
	X	Y	Z	Pitch	Roll	Yaw
S1	−0.21 ± 0.24	0.58 ± 0.17	0.24 ± 0.11	−0.03 ± 0.09	0.68 ± 0.10	−0.12 ± 0.23
S2	0.38 ± 0.98	1.10 ± 0.48	−1.05 ± 0.32	0.18 ± 0.44	0.28 ± 0.27	0.19 ± 0.84
S3	0.48 ± 0.66	0.68 ± 0.89	0.68 ± 1.02	−0.44 ± 0.94	−0.19 ± 0.36	0.87 ± 0.93

Table 2.

Number of successful rtfMRI trials before (denoted as ‘Off’) and after (‘On’) retrospective analyses of motion correction (‘Realign’) and low-frequency drift correction (‘Drift’; 0.01 Hz cut-off).

Subject index	Number of successful trials out of 4 trials for each target (horizontal/vertical/diagonal)
	Realign Off, Drift Off	Realign On, Drift Off	Realign Off, Drift On	Realign On, Drift On
S1	3/4/3	4/4/3	4/3/2	4/3/2
S2	1/3/2	1/1/2	0/3/0	0/3/0
S3	1/1/3	1/4/0	1/3/1	0/3/0

The resulting probabilities of hitting the target by chance, as obtained from the Monte-Carlo simulation, are summarized in Table 3. The maximal chance level of 14% hit rate was anticipated (for vertical movement only; Gaussian distribution with a standard deviation of 3.0%). Considering the typical percent BOLD contrast from a motor imagery at 3 T environment (i.e. < 2%) [28], the hit rate by chance is expected to be less than 5–6% (c.f. max. of 4.8% hit rate for the simulated BOLD intensities following the Gaussian distribution with 0.5% mean and 1.0% standard deviation). Note that the higher probability for the vertical target than horizontal target is due to the reduced number of minimum moves to reach it (i.e. 4 for the vertical target vs. 5 for the horizontal target).

The number of voxels counted from the primary motor cortex is summarized in Table 4, comparing before and after the rtfMRI trials. For each of the subjects, the LI calculated from the LH task indicates that the subjects were able to selectively increase the activations in the right M1 region. The LI values calculated from the RH task did not show any apparent changes associated with the rtfMRI trials. Interestingly, for S3, there were no active voxels in M1 for the LH task (p < 0.001) before the rtfMRI trials; however, the active voxels in M1 within a right-hemisphere were drastically increased (n = 283; LI = −0.99). These results may indicate that rtfMRI trials assisted individuals in regulating regional BOLD activations by the visual feedback of robot movement. In this regard, further questions rose as to the presence of short-term learning and associated changes in neural circuitries after the rtfMRI BMI trials. Group-level analyses, such as random effect analysis (RFX), on a greater number of test subjects are warranted in order to characterize the learning effect for the general population [7].

In this study, we reported the feasibility of voluntary control of two-dimensional movement of a robotic arm regulated by somatomotor activations via motor imagery, as detected by rtfMRI. The levels of activations were translated into horizontal and vertical movements of the robotic arm, which were used as tangible feedback for the subject. One subject (S1) was able to achieve the goal for more than half of the trials. The other two participants, who experienced fMRI for the first time, successfully achieved the goal in at least one of the trials (>25% success rate) with regard to either the horizontal or vertical targets, and in two of the trials (>50% success rate) with regard to the diagonal target.

It was interesting to observe that the fMRI-naive participants (S2 and S3) were able to perform strategies leading to isolated and consistent activations corresponding to either the RH or LH task, especially after showing bilateral or weak activations within the motor areas before the rtfMRI BMI trials. From the examples shown in the arrow plots of Fig. 4a, it can be seen that S2 showed simultaneous activities within both ROIs in the beginning of the 2nd trial for the horizontal target. However, the subject managed to engage the RH task while disengaging from the LH task and successfully hit the horizontal target after 13 movements. Similarly, S3 was able to manipulate the isolated activation within the ROI of the RH task and successfully hit the horizontal target after 6 moves during the 3rd trial. These results suggest that real-time feedback helped the volunteers adopt (and learn) strategies to regulate the activity from spatially distinct brain areas.

The observed degradation of the performance after the retrospective motion/drift correction (i.e. reduced number of successful trials) may be attributed to the following probable circumstances: (1) some of the successful BMI trials were indeed due to the contribution of motion- or drift-related artifactual signals and (2) the subject managed to compensate for these spurious artifacts during the BMI trials, which consequently deteriorated the performance after the removal of these artifacts from the off-line analysis.

Although the head movement and signal drift did not affect our overall results, drift in BOLD intensity due to imperfections of the magnetic gradient field can be problematic since the motion corrected BOLD time series may result in linear signal drifts [5]. One of the possible ways to correct the drift is through high-pass filtering of the entire BOLD time series. However, the high-pass filtering adopted during on-line processing may confound the proper detection of the BOLD signal for our continuous trial design due to an extremely low cut-off frequency (e.g. <0.01 Hz for a period of >100 s). The adoption of correction algorithms for head motion and signal drift, adaptable to on-line fMRI data processing, will be important, especially for rtfMRI applications [24,30], and would warrant further investigation.

One of the limitations of the current study design is the lack of an objective measurement of physical motion during the hand motor imagery tasks. In this investigation, only visual inspection of hand motion was employed. In order to ensure the absence of muscle movement in future studies, MR-compatible electromyography (EMG) recordings will provide unequivocal proof of the absence of muscle engagement during the task. To do so, selective removal of signal artifacts corresponding to the MR gradient switching during the fMRI acquisition [8] would be necessary.

The potential applications of the proposed rtfMRI-based BMI/BCI may include rtfMRI-based neuro-rehabilitation for stroke patients. For example, the robot-assisted locomotor training strategy for motor rehabilitation of chronic stroke patients [13] may provide a more realistic feedback environment via robotic arm motion during neurofeedback-enabled rtfMRI training. We recently reported that the rtfMRI-based feedback of neural activity in the primary motor areas can be learned after several weeks of self-practice [29] with evidence of neural plasticity [16]. Previous fMRI-based BCI studies have also shown the consolidation of a region-specific activation within the somatomotor and auditory areas through the use of neurofeedback training among healthy individuals [2,27]. The role of fMRI-based BMI/BCI and its long-term learning effects and potential neural plasticity constitute subjects of our future investigations.

The information obtained from rtfMRI for BMI/BCI may also be adopted to calibrate and optimize the method of BMI/BCI systems via electroencephalogram (EEG) or electrocorticogram (ECoG) based neuroimaging modalities. An fMRI modality provides a unique opportunity for superior spatial resolution of brain activities among the non-invasive neuroimaging modalities. For an ECoG-based system, the location of the electrodes to be implanted within the brain can be optimized through functional mapping based on fMRI. In regard to the calibration purpose of rtfMRI modality with an EEG-based BMI system, the task-dependent EEG-signatures, such as slow-cortical potential, evoked potential, and spectral rhythms, along with the corresponding target brain ROI can be effectively identified through simultaneous EEG and rtfMRI trials [1,11]. In order to properly execute these approaches, one must address the potential issues that are commonly encountered during simultaneous EEG-fMRI, such as imaging noises, ballistocardiogram, and ocular artifacts [17].