Visuomotor discordance during visually-guided hand movement in Virtual Reality modulates sensorimotor cortical activity in healthy and hemiparetic subjects

Abstract

We investigated neural effects of visuo-motor discordances during visually-guided finger movements. An fMRI-compatible data glove was used to actuate (in real-time) virtual hand models shown on a display in 1^st person perspective. In experiment 1, we manipulated virtual hand motion to simulate either hypometric or unintentional (actuation of a mismatched finger) feedback of sequential finger flexion in healthy subjects. Analysis of finger motion revealed no significant differences in movement behavior across conditions, suggesting that between-condition differences in brain activity could only be attributed to varying modes of visual feedback rather than motor output. Activation in the veridical relative to either altered feedback conditions was localized to the ipsilateral motor cortex. Hypometric feedback and mismatched finger feedback (relative to veridical) were associated with distinct activation. Hypometric feedback was associated with activation in the contralateral motor cortex. Mismatched feedback was associated with activation in bilateral ventral premotor, left dorsal premotor and left occipitotemporal cortex. The time it took the subject to evaluate visuomotor discordance was positively correlated with activation in bilateral supplementary motor area, bilateral insula, right postcentral gyrus, bilateral dorsal premotor areas and bilateral posterior parietal lobe. In Experiment 2, we investigated the effects of hypo- and hypermetric visual feedback in three stroke subjects. We observed increased activation of ipsilesional motor cortex in both hypometric and hypermetric feedback conditions. Our data suggest that manipulation of visual feedback of one’s own hand movement may be used to facilitate activity in select brain networks. We suggest that these effects can be exploited in neurorehabilition to enhance the processes of brain reorganization after injury and, specifically, might be useful in aiding recovery of hand function in patients during virtual reality-based training.

I. Introduction

Feedback (visual and haptic) of one’s motor performance can have a strong influence over the motor system and may serve as a potent signal for remapping sensorimotor circuits during skill acquisition [1–4]. Some have exploited this concept by showing that exaggerated haptic movement error (using robotics) during training can accelerate motor learning [5, 6] which may be potentially applicable for retraining skills in patients with neurological impairment. The visual system too has a strong influence over the motor system and may serve this purpose, for example in virtual reality-based therapeutic interventions [7, 8]. These visuomotor interactions have been largely studied for proximal arm control and are mediated by a cortical and subcortical network including fronto-parietal cortices, cerebellum, and basal ganglia [9]. Our study builds on this knowledge base by investigating the neural networks that subserve visuomotor interactions during fine-grained finger movement.

To study visuomotor interaction, we decoupled visual feedback from movement using a custom-designed Magnetic Resonance Imaging (MRI) compatible virtual reality (VR) system [10, 11]. During event-related fMRI, subjects performed sequential finger movements which were measured by an instrumented glove. Real-time feedback was provided in VR as motion of virtual hand models, displayed in 1^st person perspective [12]. The aim of Experiment 1 was to identify in a healthy subject model, the neural circuits involved in online processing of visuomotor discordance during fine-grained finger movement. For this, we either (A) manipulated the gain between the performed finger motion and the observed finger motion (by applying a moderate (0.65) or strong (0.25) scaling factor to the movement of the virtual finger relative to the subjects’ physical movement making the feedback appear ‘hypometric’), or (B) mapped an incongruent VR hand finger motion with respect to the physically moving finger (causing the feedback of the moving finger to be ‘mismatched’). The purpose of Experiment 2 was to test if neural networks recruited by such VR-based visuomotor discordance are preserved in chronic stroke patients.

The reasons for choosing those specific visuomotor discordances were several-fold. First, although it has been shown that gain discordance in a similar VR environment can bolster M1 excitability [13, 14], it remains unknown if regions outside M1 are also recruited. Second, these feedback discordances mimic specific visual feedback impairments after neurological disorders, such as stroke (in the case of hypometric and/or mismatched feedback) and Parkinson’s disease (in the case of hypometric feedback). As such, by dissociating feedback from movement, we are able to study how altered feedback affects neural circuits in a healthy human model system. Third, these visuomotor manipulations may have therapeutic promise by being integrated into VR-based rehabilitation platforms – making it critical to understand their effects on the brain.

The primary goal of this study was to characterize regions facilitated by altered (relative to veridical) feedback. Our design allowed us to use trial-by-trial kinematic and behavioral response data recorded during the fMRI session to verify compliance with the task, to assure that subjects perceived the different feedback conditions, and to quantify possible between-condition differences in performance that could otherwise confound results. We predicted that altered feedback would require reconciliation of discrepancies between visual and proprioceptive input, and motor commands and therefore may recruit regions involved in sensorimotor processing such as posterior parietal, dorsal premotor, and motor cortices, and possibly the posterior superior temporal sulcus which has been recently shown to be responsive to visuomotor incongruencies [15]. An additional focus was to conduct an exploratory sub-analysis to test whether different types of altered feedback during sequential finger motion (hypometric and mismatched finger) would recruit different regions. Such a finding may provide justification for using different forms of feedback to therapeutically facilitate activation in target brain areas. A particularly interesting finding would be if primary and secondary motor areas could be facilitated by on-line manipulations of feedback. The potential to bolster motor cortex excitability through visuomotor manipulation may have important clinical relevance because these areas are typically under-active in a number of neurological disorders (such as stroke and Parkinson’s disease).

II. Methods

A. Subjects

Exp 1 (Healthy)

Twelve right-handed [17] and healthy subjects (mean age ± 1 standard deviation: 27.3 ± 3.5 years) participated.

Exp 2 (Stroke)

Three right-handed subjects with unilateral hemiplegia due to stroke (see Table I for subjects’ information) participated. Inclusion criteria for stroke subjects included: a history of stroke (at least 6 months prior) and no other neurological pathology, a score of at least a 3 on the Chedoke-McMaster Arm scale [16] and the ability to perform 40° of active finger flexion. Based on the Chedoke-McMaster Arm and Hand scale, Subject 1 was moderately impaired while Subjects 2 and 3 were mildly impaired. All subjects participated in the study providing institutionally-approved informed consent and were deemed safe to participate in fMRI experiments based on fMRI safety questionnaires.

TABLE I.

STROKE SUBJECTS CLINICAL INFORMATION

Pt	Age	Sex	Months since CVA	CVA side L/R	CMA	CMH	Lesion Location
1	58	M	132	R	5	4	R cortical
2	69	F	18	R	7	7	R subcortical
3	41	M	158	L	6	6	L cortical

Pt, patient, CVA, cerebrovascular accident, M, male, F, female, CMA and CMH, Chedoke-McMaster Arm and Hand Impairment scales [16] respectively.

B. Setup and Procedure

Experiment 1

Healthy subjects performed a sequential finger movement (index-middle-ring-pinky) task with the right hand. During the task, hand movement was measured using an MRI-compatible 5DT Data Glove 16 MRI (Fifth Dimension Technologies, http://www.5dt.com, Figure 1A). The glove has 14 fiberoptic sensors that measure the five metacarpophalangeal (MCP), proximal interphalangeal (PIP) joints, and four abduction angles. The 5DT glove measurement of the MCP and PIP joint angles was transmitted in real time to a computer through a 5 meter fiberoptic cable that ran from the glove to the console room through an access port in the wall. The fiber optic signals were digitized by a data acquisition unit that was plugged into the serial port of a personal computer that ran the simulation. The 5DT glove data was used to study movement kinematics and to control the real-time motion of virtual hands in a virtual reality (VR) simulation displayed to the subject in a projected computer screen (Figure 1B).

Figure 1.

Behavioral measurements and experimental setup. (A) A subject wearing the 5DT data glove and flexing the index finger at the start of a sequential movement. B) The display showing the left and right VR hand models. The right virtual hand model was actuated (in real-time) by the subject’s right hand movement. (C) The four feedback manipulations imposed on the virtual finger are: “V”, veridical; “G65” and “G25”, virtual hand amplitude down-scaled to 65% or 25% of movement, respectively; and “MF”, mismatched (non-corresponding) finger. (D) A sample trace showing 3-seconds (separated by hash marks) of MCP joint angle movement (y-axis, degrees) for the index (i), middle (m), ring (r), and pinky (p) fingers in each of the four conditions; the conditions in panel D correspond to the conditions in panel C.

The VR simulation was built using Virtools 4.0 software package (Dassault Systems) and communicates with the data glove through the open source VRPN (VR Peripheral Network) interface [18]. The virtual hand models in the VR simulation match the subject’s hand size. The display showing the simulation also includes a simple instruction text beneath the hand models, which cues subjects to perform the task or rest. Immediately before the experiment, subjects were trained to move their hand with veridical VR feedback to get familiar with the mapping between their motion and the VR hand’s motion. Subjects were instructed to begin each trial with their fingers in a relaxed, extended and adducted, position. For each trial, subjects were required to perform a comfortable sequential finger flexion movement (about a 30° angle from the initial position) starting with the index and ending with the pinkie. Subjects were given 3 seconds to perform the 4 fingers flexion movement, with no restrictions regarding the amplitude of movement; however a comfortable 3 seconds sequential movement can be reasonably achieved by a maximum 45 degrees finger flexion. Subjects were asked to attend to the visual feedback while maintaining consistent movement behavior across all trials despite the difference in feedback. To reduce the likelihood that subjects would alter their motion across the different feedback conditions we purposely did not provide any explicit visual targets/cues toward which the subjects produced their movement.

The correspondence between the subjects’ movement and the motion of the VR hand model viewed in the display was manipulated in one of four ways randomly on a trial-to-trial basis (see Figure 1C). 1) Veridical (V): The VR hands moved one degree for every degree of subjects’ movement; i.e. in perfect correspondence to the actual movement such that subjects were provided with high fidelity feedback of their motion, 2) G65: The VR hands moved 0.65° for every one degree of actual movement produced by the subject. Thus, the amplitude of the VR hands’ motion was 65% that of the subjects’ actual motion. 3) G25: The VR hands moved 0.25° for every one degree of actual movement produced by the subject (i.e. 25% of the subjects’ actual motion), 4) The amplitude between the VR hands’ motion and the actual motion was maintained at a 1:1 ratio but the finger on the VR hand that was actuated was not the same as the actual finger performing the movement (i.e. mismatched finger condition, MF). Conditions 2 and 3 were designed to simulate the feedback of hypometric movements, such as those that may occur due to paresis (i.e. after stroke). By including the veridical, G65, and G25 conditions, we could parametrically investigate effects of varying levels of hypometric feedback. Condition 4 was designed to simulate feedback of unintentional movements such as those that may occur due to spasticity (i.e. after stroke). Each condition lasted 3 seconds and occurred 10 times within a functional imaging run. Thus, each imaging run consisted of 40 movement trials (ordered randomly) with interspersed rest periods (randomly varying in length between 3–7 seconds). After each movement condition (but before the rest period), subjects had 2 seconds to rate on a scale of 1–4 the perceived correspondence of overall quality of feedback. Specifically, subjects were asked to rate a ‘4’ if they perceived perfect, ‘3’ moderate, ‘2’ fair, and ‘1’ poor correspondence between the VR hand motion and their own movement. The ratings were reported by using the left hand to press one of four buttons, and they were the main measure of subjects’ perception and attention to different feedback discordances. These scores were later correlated with the Blood Oxygen Level Dependent (BOLD) signal (see section C). Each run was 5 minutes and 36 seconds long and was repeated four times for each subject (total 160 movement trials per subject).

Experiment 2

The same experimental setup was used in experiment 2. Subjects were instructed to use their paretic hand. Based on the results of experiment 1 (see Results A), we focused on manipulation of visual feedback gain and investigated both hypometric and hypermetric feedback (G25 and G175, respectively) by comparing them to a veridical condition with no visual distortion. In G175, VR hands moved 1.75° for every degree of actual movement produced by the subject. Thus, the amplitude of the VR hands’ motion was 175% that of the subjects’ actual motion. G25 was similar to the experimental condition in experiment 1 with healthy subjects. Experiment 1 showed that subjects did not have difficulty in identifying feedback manipulation, thus we did not require the stroke subjects to rate the visual feedback in order to simplify the experimental design and to perform the experiment fast enough so subjects do not get fatigued while performing paretic hand movement.

C. Functional Magnetic Resonance Imaging (fMRI)

fMRI was performed using a 3T Siemens Allegra head-only scanner with a Siemens standard head coil. Structural (T1-weighted [TR 2500 ms, TE 3.93 ms, FOV 256 mm, flip angle 8°, slice thickness 1mm, voxel size=1×1×1mm, and resolution 256]) and functional images (TR=2500 ms, TE=30 ms, FOV=192 cm, flip angle=90°, bandwidth=4112 Hz/px, 3×3×3mm, 46 slices, slice thickness 3mm) were acquired. fMRI data were preprocessed and analyzed with SPM8. Two dummy images were acquired (but not saved) at the start of each run to account for field inhomogeneity. Each subject’s functional volumes were realigned to the first volume, co-registered, normalized to the Montreal Neurological Institute template, and smoothed (8mm). The onset and duration of each event (V, G65, G25, and MF in exp 1 or V, G175, and G25 in exp 2) were obtained from onset and offset of finger motion on each trial. Each condition was modeled as a separate column in the SPM design matrix. Decision time and rating of the quality of feedback (see section below), were also entered into the design matrix as parametric modulators within each condition. Additional parametric modulators included movement time during each trial and the average of the four fingers peak angles. Correlation of bold activity with movement parameters (movement time and amplitude) was investigated to explore if possible difference in movement kinematics across conditions lead to changes in BOLD activity.

Contrast images were first created for each subject (fixed effect model). For experiment 1, each subject’s contrast image was then passed on to the second-level analysis at the group level (random effects model). Condition-specific differences in the BOLD signal were analyzed with a general linear model (GLM) approach for event-related fMRI using SPM8. For both experiment 1 group level and experiment 2, activations were significant at an uncorrected threshold magnitude of P<0.01 and a cluster extent of 10 voxels. An uncorrected threshold was used because the analysis was performed within a region of interest defined by the task versus rest contrast. The following contrasts were estimated:

Experiment 1

1. Task contrast: V+G65+G25+MF > rest. This movement versus rest contrast (see Figure 2) was used as an inclusive mask to limit the search volume in all subsequent analyses.

2. Main effect of feedback type. For the hypometric type of altered feedback, we identified regions where activation parametrically increased or decreased as a function of the degree of hypometric feedback (G25>G65>V and G25<G65<V). For the mismatched feedback condition, we performed the contrasts MF>V and MF<V.

Figure 2.

Regions showing significant activity during movement. Activation is plotted on a rendering of the fiducial brain (Caret [19]). Colormap represents t-statistics. This map has been used as an inclusive mask for subsequent analyses.

Experiment 2

The main of contrasts of interests are (V>G175, V>G25, G175>V and G25>V). Subject 3 was right hemiplegic so he moved the right hand during the experiment; however, his fMRI scans were flipped before preprocessing so that the right hemisphere is the ipsilesional side in all 3 stroke subjects’ results.

D. Analysis of movement kinematics

The onset of 5DT glove data collection was synchronized with the VR display and the first volume of fMRI. The glove data (movement kinematics) were analyzed offline using custom written Matlab software (MathWorks^®). We defined the onset of each movement as the time at which the mean angular velocity of the MCP) and PIP joints of the first finger in the sequence exceeded 5% of their mean peak angular velocity on the given trial. Movement offset was defined as the time at which the mean angular velocity of the pinky MCP and PIP joints fell below 5% of the peak value. We then identified for each trial: (1) movement duration as the duration of the interval between the offset and the onset; (2) mean (across PIP and MCP joints) peak angular displacement of each finger. The kinematic data were analyzed to confirm that subjects complied with the task, and to verify that finger movements during the execution epochs were consistent across conditions. This was important to rule out the possible confound of movement extent on brain activation. Kinematics were analyzed with a repeated measures analysis of variance (ANOVA) with within factors: FINGER (index (I), middle (M), ring (R), pinky (P)), CONDITION (V, G65, G25, MF in exp 1 or V, G175, G25 in exp 2), and SESSION (run1, run2, run3, run4). Statistical threshold was set at 0.05. In addition to movement kinematics, we also recorded, immediately after each finger movement, subjects’ response ratings for how well the VR hand motion matched their actual motion.

III. Results

A. Experiment 1

Behavior

Figure 1D shows a representative subject’s MCP joint angle trace for each finger and condition recorded during fMRI. Inspection of such traces suggested that subjects complied with instructions to perform sequential index, middle, ring, pinky movements consistently across conditions. Repeated measures ANOVAs for reaction time (RT), movement duration (MT), and movement extent (Peak Flexion Angle) revealed no significant main effects for FINGER, CONDITION, or SESSION (p>0.05, see also Table 1). After each trial, subjects evaluated on a 1–4 scale (4=perfect correspondence) the perceived degree of correspondence between their intended action and the feedback. The group mean ratings in the veridical, G65, G25, and MF conditions were 3.6, 3.3, 1.9, and 1.4. Regression analysis showed a significant linear relationship (r²=0.96; p<0.05) between the perceived correspondence of the VR hand’s motion and the subject’s actual movement, with subjects reporting progressively less correspondence between their actual finger motion and the motion of the VR hand across the V, G65, G25, and MF conditions (see Figure 1B). A Mann-Whitney signed-rank test revealed that perceptual ratings in the veridical condition were significantly higher than those in the G25 (U=144; p<0.001) and MF (U=144; p<0.001) conditions. Further, the ratings in the G25 condition were significantly higher than those in the MF (U=114; p=0.01), suggesting that subjects perceived these forms of feedback as distinct from each other. The above data confirm that subjects attended to the visual feedback during each trial and that the mismatched finger condition was perceived as the strongest visuomotor distortion.

fMRI. Movement versus rest

A distributed sensorimotor circuit typically involved in visually-guided movement was activated in the task versus rest contrast (see Fig. 2). These regions included the bilateral (stronger contralaterally) sensorimotor cortex, bilateral SMA and middle occipital cortex, bilateral insula (stronger ipsilaterally), and bilateral superior temporal gyrus. Additionally, activation was noted in regions within the ipsilateral inferior parietal lobule, contralateral middle frontal gyrus (Brodmann area 9), contralateral supramarginal gyrus, ipsilateral thalamus and ipsilateral fusiform gyrus and contralateral midbrain.

fMRI. Veridical compared to hypometric feedback

While motion of the VR hand model in the veridical (V) condition corresponded to subjects’ movement, the amplitude of the VR hand model was only 65% of actual movement in the hypometric (G65) condition and 25% that of actual movement in the hypometric (G25) condition.

Figure 3 shows that the only region where activation parametrically increased as the magnitude of the gain distortion increased was the contralateral precentral gyrus. Activation in this cluster extended into the fundus of the central sulcus and corresponded to the hand representation (hand knob area) of the primary motor cortex and dorsal premotor cortex. Conversely, an inverse relationship between BOLD and distortion (i.e. parametric increase in activation as a function of a decrease in the gain of distortion) was noted in the contralateral temporal-occipital cortex, corresponding to the extrastriate body area (see also Table III).

Figure 3.

Contrasts between veridical (V) and hypometric (G25) feedback conditions. Regions where activity increased (red) and decreased (blue) with visual discordance gain are shown as SPM T-maps superimposed on an inflated fiducial anatomical rendering (Caret [19]). For plotting, threshold was slightly lowered (p<0.01) to better illustrate activated regions. Beta estimates for activation in each condition (V, G65, G25) are shown in the bar plots. Error bars indicate 90% confidence intervals.

TABLE III.

CLUSTERS OF ACTIVATION IN THE MAIN CONTRASTS (EXP 1)

Region	Side	k	x, y, z	t	z
V > G25
mid occipital g.	L	314	−26 −88 6	5.62	3.78
mid temporal g.	L	645	−46 −60 0	5.27	3.65
sup parietal	L	57	−22 −54 54	3.76	2.95
G25 > V
precentral g.	L	62	−40 −20 68	3.98	3.07
V>G65
middle temporal g.	L	122	−48 −60 −2	5.64	3.79
G65>V
postcentral g.	L	43	−50 −14 50	4.58	3.36
V > MF
mid occipital	L	283	−24 −98 12	7.1	4.26
inf occipital	L	16	−42 −70 −6	3.34	2.72
MF > V
inf frontal opercularis	R	80	54 12 30	9.82	4.92
Inf frontal triangularis	L	42	−52 16 0	6.54	4.1
mid frontal gyrus	R	66	36 2 58	5.6	3.78
inf parietal g.	L	70	−36 −46 44	5.3	3.66
inf parietal g.	R	137	42 −50 54	3.69	2.91
postcentral g.	L	78	−42 −34 42	4.66	3.39
cerebellum	R	51	24 −50 −36	4.35	3.25
supplementary area	L	124	−8 18 44	4.34	3.25
insula	R	59	46 18 −2	4.34	3.25
insula	L	12	−36 18 6	3.81	2.98
mid frontal g.	L	78	−36 4 50	3.77	2.96
mid frontal g.	L	69	−54 16 36	3.74	2.94
precentral g.	L	37	−58 −18 42	3.73	2.94

fMRI. Veridical compared to mismatched feedback

In mismatched feedback (MF) trials, the finger that was actuated never corresponded to the finger that the subjects used but the amplitude of VR hand motion corresponded to actual motion (unlike in the hypometric condition above). The contrast MF>V was associated with activation in the bilateral inferior frontal gyrus (pars opercularis), a caudal region in the contralateral middle frontal gyrus (dorsal premotor cortex, Brodmann area 6), postcentral gyrus, bilateral insula, bilateral supplementary motor area, and anterior portion contralateral of the intraparietal sulcus (see Figure 4 and Table III). Activation in the V>MF contrast was noted only in the contralateral middle and inferior occipital cortex.

Figure 4.

Contrasts between veridical (V) and mismatched feedback (MF) conditions. SPM T-maps are superimposed on an inflated fiducial anatomical rendering using Caret software [19]. For plotting, threshold was slightly lowered (p<0.01) to better illustrate activated regions. Beta estimates for activation in each feedback condition (V, MF) are shown in the bar plots. Error bars indicate 90% confidence intervals.

Correlation between BOLD and decision making time

We measured the time subjects took to rate the perceived amount of correspondence between their actual movement and the VR hand’s movement (i.e. decision-making time). Figure 5A shows that subjects took increasingly longer to evaluate the perceived distortion as the degree of correspondence decreased (i.e., they were quicker to recognize veridical feedback). Interestingly, subjects took the longest to evaluate the feedback of the MF condition, which was arguably the most distinct feedback condition. BOLD activity positively correlated with the decision time in bilateral insula, bilateral superior parietal lobules, contralateral caudal middle frontal gyrus (dorsal premotor area, Brodmann area 6), bilateral supplementary motor area and bilateral inferior occipital lobe.

Figure 5.

(A) Scatter plot showing strong correlation between subject’s perception of feedback congruence and strength of visuomotor discordance. (B) Decision time (to estimate visuomotor discordance) per condition.

Correlation between BOLD and movement kinematics

Analysis of correlation between BOLD signal and movement amplitude revealed no significant correlation in the sensorimotor cortex. This was possibly attributed to the high homogeneity of movement kinematics across the trials (see Behavior Results section). Correlation of BOLD signal and movement time showed small activity in ipsilateral thalamus but no activity in the sensorimotor cortex

B. Experiment 2

Behavior

In contrast to Experiment 1, average velocity of finger motion and peak flexion angle varied across feedback conditions in each of the three stroke subjects (see Table IV). Movement amplitude was larger in V and G25 conditions than in the G175 condition (by less than 0.1 radian on average for each subject). Movement time varied across conditions only for subject 1.

TABLE IV.

Behavioral Data Across Conditions (EXP 2)

Pt.	Condition	Peak Flexion Angle	RT (ms)	MT (ms)	Movement Mean Velocity
1	G175	0.503 (0.061)	983 (256)	931 (171)	1.08 (0.225)
	V	0.547 (0.015)	970 (181)	1011 (178)	1.0(0.156)
	G25	0.639 (0.082)	946 (269)	1201 )129)	0.872 (0.037)
	P	<0.0001	0.756	<0.001	0.0082
	F	21.41	0.238	15.98	5.662
2	G175	0.581 (0.044)	861 (164)	1916 (158)	0.368 (0.051)
	V	0.651 (0.048)	864 (132)	1905 (134)	0.432 (0.046)
	G25	0.674 (0.043)	786 (103)	1787 (200)	0.462 (0.053)
	P	<0.001	0.365	0.183	0.0005
	F	20.96	1.06	1.852	11.45
3	G175	0.487 (0.056)	433 (60)	1828 (158)	0.724 (0.097)
	V	0.597 (0.068)	434 (61)	1816 (88)	0.821 (0.107)
	G25	0.59 (0.068)	421 (47)	1842 (147)	0.769 (0.062)
	P	<0.001	0.823	0.75	0.03
	F	16.216	0.196	0.29	3.98

Mean (±1 standard deviation) of kinematic data extracted from the instrumented glove and button responses. Peak Flexion Angle is in radians and Movement Mean Velocity is in radians/sec.

fMRI. Veridical Compared to Hypermetric and Hypometric conditions

Figure 6 (see also Table V) shows the main effect of the G25>V contrast on the activity in the ipsilesional motor cortex. Beta estimates show that both hypometric and hypermetric visual feedback conditions had a similar facilitatory effect on ipsilesional motor cortex. Activity in the motor cortex in the G175>V contrast overlapped with the activity in the G25>V contrast. This area in turn anatomically overlaps with the motor cortex activation area in the G25>V contrast in experiment 1.

Figure 6.

Results of stroke subjects, experiment 2. Regions showing significantly more activation in the veridical (V) than the G25 condition shown over a fiducial brain (Caret [19]). Beta estimates for activation in the ipsilesional motor cortex in each condition are shown in the bar plots to the right (G175, V, G25). Error bars indicate 90% confidence intervals.

TABLE V.

CLUSTERS OF ACTIVATION IN THE MAIN CONTRAST G25>V (EXP 2)

Region	Side	k	x, y, z	t	z
Subject 1 G25>V
precentral g.	L	41	−39 −10 67	3.43	3.42
inf frontal opercularis	L	10	−60 11 7	3.35	3.34
SMA	R	55	6 8 73	3.22	3.21
precentral g.	L	15	−27 −16 73	2.98	2.97
sup parietal	R	10	36 −73 49	2.81	2.8
precentral g.	R	17	36 −13 64	2.81	2.8
inf parietal	L	10	−39 −61 55	2.76	2.76
Subject 2 G25> V
precentral g.	R	222	48 −13 55	3.77	3.76
postcentral g.	L	34	−48 −25 28	3.33	3.32
SMA	R	11	9 5 52	2.94	2.93
inf parietal	R	29	42 −55 52	2.85	2.84
Subject 3 G25>V
precentral g.	R	14	30 −19 55	2.45	2.45
SMA	L	21	−12 −4 61	2.36	2.36

IV. DISCUSSION

We dissociated visual feedback from movement to study effects of visual feedback on neural circuits. Despite altered feedback, kinematic data indicated that subjects maintained consistent movements across different visual feedback conditions, as instructed. Moreover, the strong linear relationship between the feedback congruence and subjects’ estimate of feedback quality confirms that subjects attended to the visual feedback throughout each trial. The task was associated with activation in a typical distributed network of sensorimotor regions sub-serving visually guided sequential action. Our main finding was that, within this task-related mask, different forms of altered feedback had unique effects on brain activity. These differences were driven entirely by visual feedback rather than by potential discrepancies in motor output since movement kinematics were similar across conditions and movement-based activation was subtracted out in each contrast. Unlike healthy subjects, stroke patients (Experiment 2) had difficulty maintaining consistent movement kinematics across condition, a finding previously reported in persons with stroke [20]. Despite this, however, cortical activation in stroke patients was evident across both the high- and low-gain feedback conditions (in which movements were slower and faster relative to veridical, respectively), suggesting that movement kinematics could not be a confounding factor in the pattern of activation.

Contralateral M1 is facilitated by discordance in gain

Our most notable finding was that quantitative discordance in gain between executed movement and observed feedback was associated with a parametric increase in activation in contralateral M1. Analysis of movement kinematics confirmed that performance could not confound this result. This finding is consistent with a model in which M1 is involved in on-line processing of error-based information for visual guidance of movement. These data fits previous imaging work integrating gain manipulations into isometric force production tasks [21] and sinusoidal line tracing with finger flexion-extension movements [22]. In these studies, activity in contralateral M1 was found to be increased as accuracy demands, which required subjects to modify their motor output in order to reduce error, also increased. Our experiment adds to this knowledgebase by (1) ruling out the possibility that feedback-based modulation of M1 is affected by performance changes (since performance in our study was clamped), (2) showing that M1 can be modulated even in the absence of an explicit target or goal, and (3) demonstrating that this modulation can occur at a single trial level rather than after adaptation that occurs over the course of a block of training, as in the above mentioned studies.

Another interesting finding was the absence of contralateral M1 facilitation in the mismatched finger (MF) condition, suggesting that the modulation may be feedback-error specific. A parsimonious explanation is that low-gain feedback in the G25 and G65 conditions (relative to veridical) upregulated neural activity in the motor system as if M1 was acting to reduce the discrepancy between the intended action and the sluggishly moving virtual finger. Such upregulation would not be necessary in the MF condition (since the observed amplitude of the incorrect finger motion matched the actual movement) and may be the reason why no M1 modulation was noted in the MF condition. In a recent imaging study in which subjects lifted objects whose weight was unpredictably lighter or heavier than expected [23], the authors noted that M1 activity increased only when the object weighed more than predicted, but not in the opposite condition, and concluded that this M1 modulation reflected the gradual increase in lift force (above predicted levels) after the object was grasped. Collectively, these data suggest that low-gain feedback manipulations may serve as a useful therapeutic tool during training by having a facilitatory effect on the motor system. Like the haptic feedback manipulation used by [23], we demonstrate that visuomotor discordance may also bolster the motor system. However, the data in Experiment 2 show that both hypermetric and hypometric feedback may bolster activity in the ipsilesional motor cortex of stroke patients, suggesting that both high-and low-gain discordance may have excitatory effects on the lesioned motor system. Since visual manipulations can be easily implemented into virtual reality-based systems, VR may be an ideal platform for delivering interventions with visuomotor discordances [7, 10, 24]. These conclusions are of course to be taken with caution given the limited number of patients in our study and lack of other published data in this regard. However, it nevertheless beckons the need to further investigate the interesting potential that the application of visuomotor gain discordance may have on recovery and brain reorganization after stroke, and in a broader spectrum of patients. In this study, we required subjects to be able to produce a minimum of 40° finger motion in order to have a perceivable discrepancy between the different feedback gain conditions. This limited our stroke sample to those with relatively mild impairment. Given our findings, however, it would be interesting to explore in future work whether this effect would be preserved in those with larger lesions or deficits. Another important and related future direction is to understand if repeated training with discordance would produce similar or perhaps more amplified effects in the motor system. Last, another interesting question is whether different types of discordance (eg., hypermetric versus hypometric) may be uniquely more suited for patients with different deficits. For example, perhaps individuals with highly limited finger motion would benefit from hypermetric visual feedback discordance since hypometric feedback may not be feasible.

Processing of observed movement amplitude and the Extrastriate Body Area

There is an ongoing interest in the role that higher-order visuomotor processing areas, such as the extrastriate body area (EBA), play in action observation. A recent meta-analysis elegantly demonstrates that EBA is overwhelmingly recruited for tasks involving task-level control and focal attention [26]. More specific to motor control, the EBA has been repeatedly identified for its role in higher order visual processing of observed biological movements [15, 27] [28]. An interesting proposition by Downing and co-workers [15, 29] is that activity in EBA reflects observed actions independent of efferent motor signals. If true, EBA may not be involved in reconciling discordance between intended and observed motor outcomes. Our experiment supports this view by showing a positive but equal increase in BOLD in EBA for both veridical and mismatched finger conditions, in which the amplitude of physical and observed movement was clamped in spite of the incongruence. Importantly, EBA activity decreased as the amplitude of the observed movement decreased from the V to the G25, despite the physical movement remaining constant. Urgesi et al. [30] found that functional disruption of EBA using rTMS impaired the ability to visually distinguish between subtle differences in human body posture configurations of the same body part. In the context of this work, our findings suggest, therefore, that activity in EBA may be modulated by the amount of observed body motion of the same body part, whether it is congruent with the executed movement or not.

Mismatched feedback activates a frontoparietal network

Virtual hand motion in the mismatched feedback condition was both amplitude- and phase-locked to the subject’s movement. Only the mapping between fingers was altered, creating a discrepancy between the intended action and the visual feedback of that action. Mismatched feedback was perceived by the subjects as more discordant than the gain feedback manipulation and arguably was the only condition in which the body schema was violated. The contrast between the mismatched and veridical feedback revealed activation in the bilateral insula, inferior frontal gyrus (pars opercularis), postcentral gyrus, supplementary motor area, contralateral anterior intraparietal sulcus, and dorsal premotor cortex. No significant motor cortex activation was noted in this contrast, as was the case in the G25 and G65 relative to the veridical condition. Our recent work demonstrated that observation of actions with the intention to imitate the observed movements results in activation of similar parietal and insular networks [10]. Given these regions’ involvement in processing percepts of agency / ownership of actions [31], intentional action observation of action [32] [33], it’s capacity to remap by incorporating tools as part of the body image [34], and its connectivity with the premotor cortex [35], it is likely that observing motion of a mismatched finger condition elicited a salient discordance in the self-other representation, bolstering activity in the parietal and insular areas as it reconciled this discordance.

An interesting finding was the significant activation of bilateral premotor areas in the mismatched relative to veridical feedback conditions. Previous work has identified premotor areas to be recruited during action observation, particularly when sensorimotor transformations between executed and observed movement were necessary [36, 37]. For example, in one study, subjects observed either correct or incorrect pairings between hand postures and objects, having to analyze whether the hand posture was appropriate for functionally grasping the object. Similar to our study, the authors noted bilateral ventral premotor activation in the “incorrect pairing” versus “correct pairing” contrast. In another study, however, [36] in which subjects observed intentional and unintentional actions, the authors noted stronger activation in the lateral premotor areas for the intentional (relative to the unintentional) condition. Thus, although the jury is still out with regard to the particular responses of lateral premotor cortex to discordance, it seems compelling that it is processing visuomotor transformations as it is forced to move between veridical and discordant feedback states.

Neural Activity Correlation with Perceptual Judgment of Feedback

We were also interested in understanding how subjects’ perception of the feedback in VR affected BOLD activation. Subjective ratings of the quality of feedback (the degree to which the observed motion of the VR hand matched the subjects’ action) were significantly correlated with the altered feedback, with mismatched feedback being reported to correspond the least with the performed action. In other words, subjects perceived mismatched feedback, though similar in amplitude to their physical motion, to be more disruptive than the hypometric (G65 and G25) feedback. This implies that the correspondence of an effector was a more important feedback attribute than movement amplitude.

Correlation between BOLD activity and the time taken by subjects to evaluate the feedback (decision time) was significant in bilateral insula, bilateral superior parietal lobules, contralateral caudal middle frontal gyrus (dorsal premotor area, Brodmann area 6), bilateral supplementary motor area and bilateral inferior occipital lobe. The insula activity is somewhat unsurprising given its role in self agency distinction. However, it is interesting that this rather extensive sensorimotor network was correlated with this phase in which subjects were evaluating the degree of correspondence. Although the significance of these activations continues to be a focus of our investigation, we interpret these preliminary findings to suggest that presenting complex feedback in VR may be used as a way to target activation in higher-order cortical regions in the frontal and parietal lobes.

V. Conclusion

We used different forms of virtual reality-augmented feedback in real-time to recruit select regions of the cortex. Given the existence of rich intra-hemispheric cortico-cortical projections between occipital, parietal, and frontal cortices [38–44], our data in humans supports the primate literature [45–49], by showing that vision can be a potent signal to sensorimotor centers. The implications of this may be far-reaching, particularly in the field of technology-assisted neurorehabilitation [50]. We are currently using similar VR interfaces to provide patients who have had a stroke with targeted visual feedback while they train on a motor task [51]. In some of this work, we demonstrate that patients training their arm-hand-finger for several weeks in a similar (but richer) VR environment show 20%–25% functional improvements [18, 51]. In our other work, we demonstrate that a similar VR interface providing mirrored visual feedback (i.e. right hand movement actuates left VR hand movement) can selectively recruit the ipsilateral sensorimotor cortex [52]. It may be that such neural modulation can be exploited to facilitate reorganization though Hebbian mechanisms. Thus, our previous and current work suggests visuomotor manipulations in VR may offer a tool to clinicians to facilitate functional recovery in patients.

TABLE II.

Behavioral Data Across Conditions (EXP 1)

Condition	Finger	Peak Flexion Angle	RT (ms)	MT (ms)	DT (ms)
V	I	0.53 (0.21)
	M	0.56 (0.33)	434	2498	833
	R	0.66 (0.21)	(95)	(218)	(227)
	P	0.55 (0.21)
G65	I	0.54 (0.19)
	M	0.57 (0.34)	438	2499	860
	R	0.67 (0.21)	(102)	(221)	(176)
	P	0.50 (0.16)
G25	I	0.55 (0.22)
	M	0.56 (0.31)	448	2465	871
	R	0.66 (0.21)	(116)	(234)	(288)
	P	0.52 (0.16)
MF	I	0.54 (0.21)
	M	0.52 (0.29)	453	2445	921
	R	0.64 (0.18)	(89)	(181)	(213)
	P	0.51 (0.17)
P		0.841	0.235	0.122	0.814
F		0.31	1.49	2.08	0.32

Mean (±1 standard deviation) of kinematic data extracted from the instrumented glove and button responses. Peak Flexion angle is in radians. DT (Decision Time) is the interval between the time of the cue to judge feedback and the press of the response button. Statistics are shown in the last two rows.

Biographies

Eugene Tunik, is an assistant professor in the department of Rehabilitation and Movement Sciences. He received his Baccalaureate in Science degree in Physical Therapy from Northeastern University (1997) and his Doctor of Philosophy degree in Neuroscience from Rutgers University Center for Molecular and Behavioral Neuroscience (2003). Following this, he completed a three-year postdoctoral fellowship at the Department of Psychology and Brain Science at Dartmouth College. Dr. Tunik’s academic and research interests include understanding brain mechanisms involved in motor control and learning in health and disease and how this information can shape the delivery of therapeutic interventions to patients. Eugene Tunik is a member of Society for Neuroscience.

Soha Saleh (SM’05) is a postdoctoral fellow in the lab of Movement Neuoroscience at the University of Medicine and Dentistry of New Jersey (UMDNJ). She received her BS degree (2006) in computer and communication engineering and minor in Biomedical Engineering from the American University of Science and Technology in Beirut, and her MS degree (2008) in Biomedical Engineering from the New Jersey Institute of Technology (NJIT). She recently received the Ph.D. degree in Biomedical Engineering from NJIT and UMDNJ graduate school of Biomedical Sciences (2012). Her research interests include stroke rehabilitation, brain reorganization after motor recovery, virtual reality and its effect on brain activity and brain functional connectivity. Soha Saleh is a member of IEEE, EMBS and Society for Neuroscience.

Sergei V. Adamovich (M’06) received the Ph.D. degree in physics and mathematics from the Moscow Institute of Physics and Technology. He is an Associate Professor in the Biomedical Engineering Department of the New Jersey Institute of Technology and Adjunct Professor in the Department of Rehabilitation and Movement Science at the University of Medicine and Dentistry of New Jersey. Dr. Adamovich’s research interests include basic mechanisms of neuromuscular control of human movement and sensorimotor learning, both in healthy populations and in people with neurological disorders like Parkinson’s disease, stroke and cerebral palsy. He is also involved in the research on technology-assisted rehabilitation. The current focus is on the use of robotics and virtual reality in the rehabilitation of hemiparetic arm and hand. He uses functional MRI and transcranial magnetic stimulation to study neural mechanisms of action and observation, and to evaluate brain reorganization after injury and during recovery. Dr. Adamovich is a member of IEEE and Society for Neuroscience.