Following and intercepting scribbles: interactions between eye and hand control

Abstract

The smooth pursuit eye movement system appears to be importantly engaged during the planning and execution of interceptive hand movements. The present study sought to probe the interaction between eye and hand control systems by examining their responses during an interception task that included target speed perturbations. On 2/3 of trials the target increased or decreased speed at various times, ranging from about 300 ms before to 150 ms after the onset of a finger movement directed to intercept the target and was triggered by a GO signal. Additionally the same 2D sum-of-sines target trajectories were followed with the eyes without interception. The smooth pursuit system responded more quickly if the target speed perturbation occurred earlier during the reaction time (i.e., near the time of the GO signal). Similarly, the finger movement began more quickly if target speed was increased earlier during the reaction time. For early perturbation conditions, the initial direction of the finger movement matched the predicted target intercept using the new target speed. For perturbations occurring after finger movement onset initial direction of finger movement did not match target interception such that, the finger path began to curve toward the perturbed target after about 150–200 ms. The results support the idea of an active process of visual target path extrapolation simultaneously used to guide both the eye and hand.

Introduction

Many daily tasks require the use of visual information to guide body movements. An activity such as catching a ball requires the use of visual sensory information and cognitive predictions to guide movements of the arm to put the hand in the correct location at the correct time to intercept the ball. However, this task is relatively easy because the path of the ball is only controlled by gravity and air resistance. A more difficult task is to swat a flying insect. In this case the insect can move erratically; thus the rapid change in target motion makes the task more challenging. In cases like this, because the movements of the target path are less predictable, the sensorimotor system must work quickly and efficiently to complete the task.

In any predictive task such as interception, it takes time to engage in sensorimotor processing thus we cannot use current information to plan behaviors, or movements would be delayed and inaccurate. Previous studies have shown that the visuomotor system partially makes up for processing delays by extrapolating the position of a target to guide the initial direction of interception movements. In two dimensions, both at the onset and during the course of an interceptive movement, the hand typically points to a location about 150 ms ahead of the target (Mrotek & Soechting, 2007; Soechting et al., 2010). Subjects use the speed and angular velocity of the target motion to make such predictions. In these experiments, subjects on average needed 350 ms to successfully complete the finger movement. Thus 200 ms of motion would be unaccounted for in the initial prediction, and consequently the system must be able to smoothly incorporate changes in sensory feedback into the interception task.

Similarly, during tracking tasks the eye movement system uses prediction to maintain accuracy if the target motion is consistent. When an unanticipated perturbation occurs the system responds after a delay. Smooth pursuit is particularly responsive to changes in the target speed (perturbations) during tracking in one dimension (Schwartz & Lisberger, 1994). Following a small increase in target speed, gaze speed increases and gaze speed decreases following a small decrease in target speed. However in these experiments perturbations away from the general tracking direction did not invoke a faithful response. In two dimensional tracking the impact of perturbations in one of the tracking dimensions also impacted the smooth pursuit response in the other dimension (Leung & Kettner, 1997; Mrotek et al., 2006). This could indicate the vertical and horizontal dimensions of tracking depend on each other and that one-dimensional tasks do not fully explain the complexities of the response.

There is also an impact of adding hand tracking to the eye tracking task. In one dimension the impact of adding a hand movement to an eye tracking task improves performance (Gauthier et al., 1988; Koken & Erkelens, 1992). In two dimensions, Engel and colleagues (2003) showed that when subjects added hand tracking to a typical eye tracking task the eye movements slowed. These researchers examined the behaviors made when tracking a target that made one abrupt direction change. They asked subjects to track the target with the eye alone, the hand alone, or both the eye and hand together. When the eye tracked the target alone they showed that, gaze was capable of changing direction nearly instantaneously as when saccades interrupt smooth pursuit and travel in an entirely different direction (Engel et al., 1999, 2000). The hand is not capable of changing direction instantaneously, instead in order to change direction it follows the 2/3’s power law (Lacquaniti et al., 1983). Therefore, to change direction the hand must slow down. These studies showed that in conditions when the eyes and hand track a target together, the eye moved in a fashion more like the hand moves.

Overall, target interception movements utilize predictions in target motion to guide the task, but the prediction does not cover all the time required for the performance. Further, eye-hand movements have been shown to influence one another differently during different tracking tasks. It is not clear how smooth pursuit might respond to speed perturbations during an unpredictable two-dimensional trajectory when also paired with an impending hand movement. In this paper we examine the impact of a change in target speed on the behaviors of the hand interception and eye movements during the interception task.

Methods

Subjects

A total of eight subjects completed four sessions requiring them to track or intercept a target moving on a computer screen. Two sessions required subjects to intercept a target moving in two-dimensions (2D) after a GO signal and the other two sessions required subjects to track the target with their eyes only. Each subject had normal or corrected to normal vision and signed the Institutional Review Board approved consent form. Subjects used the extended index finger of the preferred hand to intercept the target (one left-handed subject, one ambidextrous using the right hand, and six right-handed subjects). The finger maintained contact with a vertically oriented touchscreen (described below) as the hand moved upward from a start location at the bottom of the screen to intercept the moving target.

Equipment

Subjects were seated in front of a vertical touchscreen monitor (Elo Touchsystems; Mitsubishi Diamond Scan 20 M; 38.4 cm by 26.0 cm; 60 Hz refresh rate). The touchscreen was calibrated with a 5x5 set of targets to a spatial resolution of 0.01 cm. Room lights were dimmed in order to improve contrast on the monitor. Each subject placed the chin in a chin rest to help stabilize the head. Subjects’ eyes were approximately 40 cm from the subjective center of the screen (vertically and horizontally), which put the entire monitor screen within comfortable reaching distance.

Eye movements were monitored via a head mounted eye movement recording system (SMI Eye Link). The system utilizes three cameras to record eye position; two cameras record the right and left eye positions relative to the head, and a third camera records the location of the head relative to the environment. These signals are used to compute gaze position. Camera data were sampled at 250 Hz. The system was calibrated for each subject with a 3x3 grid of targets and prior to each trial a drift correction was performed. Eye movements were analyzed using customized off-line software (Matlab). Initially x- and y- position data for each eye within each trial were averaged and then saccades were detected and removed. Saccades were found using an algorithm described previously (Mrotek et al, 2006). Briefly, peaks in eye velocity were found and magnitude of eye acceleration was used to determine onset and end of the saccades. Each detected saccade was visually inspected to ensure correct location of onset and offset. To describe smooth pursuit, we interpolated x- and y- gaze velocity signal after the saccades were removed (Matlab interp1.m) and filtered with a zero lag 40 Hz low-pass filter (Matlab filtfilt.m).

Procedures for Interception Sessions

Subjects began each trial by touching the screen at a specified location (within a box of 1cm by 1cm) at the bottom center of the screen. Subjects watched a cyan target (~0.5°; 0.55 cm in diameter) move along one of three paths (Figure 1A). Target motion was determined using a sum-of-sines method using the following equations:

$$\text{X}(t)=\text{A}1cos(\omega t)+\text{A}2cos(\text{hx}\omega t+\varphi \text{x})$$ (Equation 1)

$$\text{Y}(t)=\text{B}1cos(\omega t)+\text{B}2sin(\text{hy}\omega t+\varphi \text{y})$$ (Equation 2)

Figure 1.

Experimental design. A) The three target paths used for both interception and control sessions in the 100, 200 and 400 ms conditions are shown. These three paths were rotated (90° counterclockwise) and reversed (along the vertical axis) in 0 ms condition to prevent familiarity with the trajectories. B) The timeline indicates possible speed perturbation events for all experimental conditions. Fast or slow perturbations could occur at the same time as the GO signal (which is 0 ms throughout the text) or 100, 200 or 400 ms after. Subjects began the interception movement approximately 300 ms after the GO signal and if successful intercepted the target approximately 400 ms later.

Such that X(t) and Y(t) are the horizontal and vertical positions of the target at time t also, h represents the harmonics (ranging from 2–5) and ϕ represents the phase (ranging from −250–230°). The speed changed in a natural way as the target moved through the path (it was faster along the straighter portions and slower along the more curved portions). On average the speed profiles of these trajectories generated with a sum-of-sines method correlated with the same trajectories generated with the two-thirds power law with an r=0.85. The period required for the target to complete the entire shape was 5.0 seconds; however the GO signal was always provided before the completion of shape trace, and interception ended the trial (Figure 2A). A change in target color served as the GO signal; it signaled the subject to quickly and accurately move the index finger to intercept the target. Subjects were instructed to use one smooth movement of the fingertip along the screen. The onset of the finger movements was determined by a speed threshold (2 cm/s) and then the algorithm searched backwards for the first minimum or when the finger speed value fell to just below an earlier (non-moving) mean+2SD. Each trial was also visually inspected for correct onset and whether or not the subject was successful on the first attempt at catching the target. The trial ended when the subject successfully placed the fingertip within 20 pixels (~0.90°; 1.0 cm) of the center of the target.

Figure 2.

Examples of trial events. A) The 200 and 400 ms perturbations were intermixed in a single data collection for half the subjects (with 0 and 200 ms intermixed for the other half of the subjects). Note that within the conditions in each panel of the figure the perturbation location remained consistent. Also, for a given subject the conditions in a single data collection remained consistent for interception and control sessions. Initial target location is represented by the asterisk and moved in the direction of the arrows along the trajectory without leaving a trace. When it reached the location of the GO signal (filled black circle) it changed from cyan to yellow to indicate to the subject to begin the interception movement. Abrupt speed changes could occur when the target reached the positions marked with the open circle. The target continued along the trajectory until it was successfully intercepted or it completed two full rotations of the trajectory (10 seconds). B) Target, eye and finger data from three individual trials (Subject 2). The target started at the asterisk and moved along the thin, dashed, black line in the direction indicated by the arrows. The GO signal was located at the filled black dot and if there was a perturbation it occurred at the location of the open circle. Smooth pursuit (thin solid colored lines near the target trajectory) from 200 ms before the GO signal to the time of interception follows along the target path, occasionally interrupted by saccades (solid thicker magenta lines). The path of the finger is shown with the thickest solid line, and is color-coded for the slower perturbation (−0.5, blue, left panel), the faster perturbation (+1.5, red, right panel) and the control situation (0, green, center panel). Since there was no requirement to stop at the target, this subject chose a strategy where her finger continued past the interception location, which is why in all three trials the finger movement path extends past the target trajectory. In each case the straight dashed lines that begin at the finger start location point ~150 ms ahead of where the target was when the finger started moving. In all panels the green dashed-dotted line represents the prediction if the perturbation was not included and the colored dotted line (blue, red) represents if the prediction took the perturbation into account. All panels in B) are from interception sessions.

At certain points along the target trajectory the target speed could be altered. This occurred on 2/3 of the trials. On 1/3 of the trials the target underwent an abrupt increase in speed, on 1/3 of the trials the target underwent an abrupt decrease in speed, and on 1/3 of the trials the speed adhered to its original trajectory. The timeline for the various conditions is displayed in Figure 1B. Throughout this text the GO signal will be referenced as time=0 ms. Speed perturbations occurred at 0 ms, 100 ms, 200 ms and 400 ms (perturbation time conditions). At all these times one of three perturbation magnitude conditions occurred; the target could remain travelling in the same speed pattern (no perturbation) a perturbation could be an increase in speed (either to 150% or 200% of the unperturbed speed; fast perturbation) or a decrease in speed (to 50% of the unperturbed speed; slow perturbation). Speed perturbations occurring at 200 ms or 400 ms were to 200% or 50%. And speed perturbations occurring at 0 ms or 100 ms were to 150% or 50%. For half the subjects the 200 ms and the 400 ms perturbations were paired in one data collection session (thus the 0 ms and 100 ms perturbations were in another collection session); for the other half of the subjects the 100 ms and 400 ms perturbations were paired in one data collection session (thus the 0 ms and 200 ms perturbations were also paired) in order to counterbalance the conditions.

All eight subjects each completed a total of 720 trials. There were ten trials in each of the four perturbation times (0 ms, 100 ms, 200 ms and 400 ms) and at each of the three perturbation magnitudes (slow, no and fast) and at each of the three trajectories for a total of 360 trials in two data collection sessions. All of these conditions were completed for the interceptions sessions (360 trials) and for the control sessions (360 trials).

In Figure 2A we show the event details for one target trajectory. The asterisk and arrow indicate the target’s start location and initial movement direction. The onset of finger motion was approximately 300 ms after the GO signal (black filled circle), and therefore the target speed perturbation was prior to onset of finger motion in the 0, 100 and 200 ms conditions, and after the onset in the 400 ms condition (see Figure 1B).

Since the GO signal location varied, we checked to make sure that this did not introduce consistent differences in target speed across conditions. We sought to avoid a situation where the GO was located on straighter or more curved sections of the path for a specific set of conditions, e.g., always on straight sections for 200 ms perturbations, but on curved sections for the 400 ms perturbations. Thus, as part of the experimental design, target speed characteristics were examined across all conditions and in all experiments. An ANOVA showed that across all trajectories and all perturbation times, the target speed in the no perturbation condition was statistically similar from 200 ms before to 300 ms after the GO signal. In contrast, in the perturbation conditions, the target speed began to significantly differ across conditions beginning 50 ms after the GO signal (due to the perturbations).

Procedures for Control Sessions

Another set of two data collection sessions served as eye-only control experiments. In these sessions the subjects made no finger movements; they were asked to merely track the moving target with their eyes. The target motion was the same as for the Interception Sessions, except that the length of the trials was shortened. The period for the shape remained consistent, but the subjects did not track the entire path. Time was reduced by eliminating the target path at the beginning and at the end of each trial to decrease the time required for the subjects to complete the experiment. The GO color change and speed perturbations were included, with at least 500 ms of target motion before the GO and after the perturbation. Half the subjects completed these control sessions prior to the interception sessions and the other half of the subjects completed the interception sessions first.

Results

In this study we performed experiments to examine how a change in target speed is incorporated into smooth pursuit behavior and a finger interception movement. In two sessions we asked subjects to intercept a target moving in two dimensions after a GO signal. We abruptly adjusted the speed of the target at three times during the planning of the interception movement and one time after the interception movement likely began. In the control sessions we asked the same subjects to visually track the target moving under the same conditions without an interception movement.

General observations

Subjects began each trial with the finger in the same location and watched a target move in a pseudorandom path until the target changed color (the GO signal), and then they tried to make one swift movement to intercept the target. Subjects required on average 305±51 ms (SD) to initiate the finger interception movement and another 402±105 ms (SD) to successfully intercept the target (i.e. with no secondary corrections on the first attempt). These results are similar to previous findings (Port et al., 1997; Brouwer et al., 2002; Merchant et al., 2003; Eggert et al., 2005).

In general, initial finger movements were along a direction that was where the target would be predicted to be in about 150 ms (Mrotek & Soechting, 2007). Figure 2B shows three individual trials in three different conditions for one subject. The smooth dashed black lines show the trajectories of the target and the arrows show the target movement direction. The middle panel shows the control trial where the target motion was consistent. The thick green line from the bottom of the figure shows the path that the finger took to successfully intercept the target. The dashed-dotted green line also emanating from the lower part of the figure shows the direction the finger should go to intercept the target approximately 150 ms ahead of where the target was located at the onset of the finger movement. Previously we showed that the prediction used target speed and angular velocity from 50 ms before the onset of movement to compute the predicted target position. Note that the initial finger movement matches the predicted finger direction and then the actual movement curves away from the prediction to correctly intercept the target in 408 ms.

Trials with target speed perturbations are shown on the left and right panels of Figure 2B. The slow perturbation is on the left and the fast perturbation is on the right. It appears that this subject took the perturbation into account when planning the direction of the finger movement. One can see that the thick line representing finger movement on the left figure initially follows the dotted blue line. Thus the line that shows a direction to a predicted target location 150 ms after the onset of finger movement (taking the slow perturbation into account) matches the actual finger direction in the trial. Additionally, the dashed-dotted green line in the panel on the left side shows the direction from the initial finger location to a predicted target location without taking the perturbation into account. Near the beginning of the movement the finger direction matches the prediction that included the impact of the perturbation. Similarly, in the panel on the right, the early movement follows the predicted path including the perturbation (follows the red dotted line) rather than the predicted path without including the perturbation (does not follow the green dashed-dotted line). This also indicates that the subject moved towards the future target location including the effect of the perturbation.

These results were consistent across subjects accordingly, similar to previous reports, the finger was initially directed to a location that the target would likely be in 150 ms. Since the finger interception movements required more than 150 ms, the initial movement direction would not successfully intercept the target. Thus in each one of the trials shown the finger movement had to deviate from the prediction to ultimately intercept the target successfully.

In addition to movement direction it is also often useful to characterize the speed of movement. We did so and found variations in finger movement speed during the interception were related to target position. In fact, the best way to represent the change in maximum finger speed was to ignore target motion direction and perturbation time conditions. To test this we performed a forward linear regression and found that distance travelled (standardized β=0.588, p<0.001) and target speed at the onset of finger movement (standardized β =0.107, p=0.026) were significant predictors. Together these variables produced an r=0.598 and a shared variance of 36%. As expected the farther the finger travelled and the faster the target was moving at the onset of movement the faster the finger ultimately moved (Brenner et al., 1998).

Smooth pursuit eye reaction to target speed change

In response to the target speed change, smooth pursuit speed also changed. We calculated smooth pursuit speed from the x- and y- pursuit velocity. Figure 3 demonstrates the response. The thin colored and stylized lines show the target speed and the thick lines with hatching show the smooth pursuit speed (±1 SEM). The two top panels show conditions where the perturbation occurred closer to the time of the GO signal, thus earlier in movement planning. The two bottom panels show conditions where the perturbation was just before or after the onset of the finger movement. The gray vertical lines represent the average time when we detected the smooth pursuit response to the speed change. It appears that when the perturbation was closer to the onset of finger movement it took longer for smooth pursuit to respond to the target speed perturbation because Figure 3A and B show shorter smooth pursuit responses times, compared to Figure 3C and D (as measured from the time of the perturbation to the vertical gray line).

Figure 3.

Smooth pursuit speed for all subjects, perturbation magnitude conditions and one path (inset between A and B) during interception sessions. Speed (of the target and gaze) is on the y-axis and time is on the x-axis. Target speed is shown with the thin lines and the various perturbation magnitudes are delineated by different line colors and styles. The target speed for the slow perturbation condition is shown with the thin blue dotted line. The no perturbation condition target speed is shown with the thin green dashed lines and the fast perturbation condition target speed is shown with the thin solid red lines. Average smooth pursuit speed (with ±1SEM hatching) is shown with the thicker colored lines. Data from the slow perturbation condition are shown with the blue lines with the most hatching. Data from the no perturbation condition are shown with the green lines with the intermediate amount of hatching and data from the fast perturbation conditions are shown with the red lines and the least hatching. Each panel features a different perturbation time condition: A=0 ms, B=100 ms, C=200 ms, D=400 ms. The trials were aligned on the time of the GO signal and this is depicted as time zero for each panel. The thick vertical gray line shows the average time when smooth pursuit responded to the change in target speed for that particular perturbation time condition for that trajectory. The thick vertical black lines show the time of finger movement onset for that particular perturbation time condition for that trajectory.

In order to examine this observation more thoroughly we calculated the average smooth pursuit speed (±1 SEM), for each subject and each condition. At the beginning of each of the condition average traces, the smooth pursuit speed for each of the three perturbation types (fast, slow and no) overlapped (Figure 3 at the left of each plot). After the target speed perturbation and a short delay, smooth pursuit speed followed the new speed in each condition (to the right of each gray line in each panel in Figure 3). We examined how long it took smooth pursuit to respond to the change in target speed by finding the time when the pursuit speed in the fast condition no longer overlapped with the smooth pursuit speed in the slow condition. Thus, for each subject and condition we computed average pursuit speed (±1 SEM) and found the first time after the perturbation that the values no longer overlapped. Average time of separation is shown for all subjects and target trajectories and for each perturbation time condition in Figure 4A. The gray bars represent the values from the experiments that included the interception task. For these (gray) bars the response to the new target speed after the 0 ms perturbation time condition occurred sooner than the response after a perturbation that occurred at 400 ms (F(3,92)=3.635, p=0.016).

Figure 4.

A) Quantification of reaction times (y-axis) for the response of smooth pursuit speed to a change in target speed for each perturbation time condition (x-axis). The shaded bars depict data from sessions with both eye and finger movement (interception sessions); the white bars refer to sessions which served as the eye-only control experiments (no finger movement). The white bars are not significantly different from each other. The shaded bar in the 400 ms perturbation condition is significantly different from that in the 0 ms perturbation condition. B) Reaction time (y-axis) for the finger interception movement for each perturbation time condition (x-axis) and for each perturbation magnitude (bar color coded). White bars represent data from the slow perturbation condition. Light gray bars represent the data from the no perturbation condition. Dark gray bars represent the data from fast perturbation conditions. The asterisked conditions are significantly different from each other (i.e. the 0 ms condition is different from the 200ms and 400ms conditions).

The white bars (in Figure 4A) represent the time it took for smooth pursuit to respond to the perturbations in the control experiments (where there was no interception task). There was no difference in the smooth pursuit response time across all perturbation times when there was no interception task (F(3,92)=2.717, p=0.049, with no post hoc differences). Using this measure it appears that the occurrence of a finger movement slows the gaze speed response to the target speed perturbation especially for later perturbations. The main question for these analyses was to determine if the planning of a hand movement impacted smooth pursuit response time, however, we also tested for a Experiment Type (control, interception) by Perturbation Time (0, 100, 200, 400) interaction and found none (two-way ANOVA, F(3,184)=1.600, p=0.191; Main Effects: Experiment Type F(1,184)=3.714, p=0.055; Perturbation Time F(3,184)=1295.357, p<0.001, all times differ from each other).

The magnitude of pursuit gain was also examined during the planning of the finger interception movement. We measure average gaze speed gain from 150 ms to 250 ms after the GO signal (approximately before the finger moved). After the different perturbation magnitudes the pursuit gain will be different, therefore we examined for an interaction between the perturbation magnitude and experiment type (control and interception) conditions and none was found (two-way ANOVA, F(2,570)=2.468, p=0.086). In the slow perturbation conditions the gain was larger during interception sessions (1.252±0.514) than during control sessions (1.053±0.432; F(1,190)=8.522, p=0.004). The same was true for conditions without perturbations (interception gaze speed gain=0.949±0.211 and control gaze speed gain=0.894±0.167; F(1,190)=3.929, p=0.049) and fast perturbation conditions (interception gaze speed gain=0.781±0.265 and control gaze speed gain=0.684±0.245; F(1,190)=6.877, p=0.009). For gaze tracking it appears that the addition of an impending finger movement improves tracking gain.

Interception finger movement onset

We also examined if the onset of the finger movement depended on the time of the perturbation. First we tested for an interaction between Perturbation Magnitude (fast, no, slow) and Perturbation Time (0, 100, 200, 400) and found none (two-way ANOVA, F(6,276)=1.826, p=0.094; Main Effects: Perturbation Magnitude F(2,276)=0.096, p=0.908; Perturbation Time F(3,276)=5.841, p=0.001, reaction time in the 0 ms condition is less than the reaction time in the 200 ms and 400 ms perturbation conditions). We found that the perturbation time impacted the length of the reaction time for the finger movement. Including all the conditions were there was a fast perturbation, we found that when the perturbation occurred at the same time as the GO signal the finger the reaction time was shorter than when the perturbation occurred near the time of finger movement onset (200 ms and 400 ms perturbations; F(3,92)=4.344, p=0.007) (Figure 4B). In the 0 ms fast perturbation condition reaction time was 269±43 ms (values in this paragraph are mean ± SD), in the 100 ms perturbation condition the reaction time was 292±39 ms, then in the 200 ms perturbation condition the reaction time was 313±48 ms, and finally in the 400 ms perturbation condition the reaction time was 313±65 ms. This was not true in the conditions without perturbations. When there was no perturbation the finger reaction time was the same across all perturbation times (F(3,92)=1.076, p=0.363). These results show that when a target abruptly increases speed while a subject is planning an interception task, it likely impacts the speed of processing and changes the time of movement onset.

Finger movement reaction to target speed change

It is possible that the impact of the perturbation is not just shown in the reaction time for smooth pursuit and finger movements, but also in the outcome of the movement plan. Figures 5 and 6 qualitatively show the reaction to the target speed change after the finger began to move. Figure 5 shows the finger path and finger movement direction for two target trajectories (A and B). The panels on the left show data from a perturbation that occurred early in movement planning and the panels on the right show data from the latest perturbation time. Within each part of the figure the top plot shows target and finger position and the lower plot shows finger movement direction over time. All green-dashed lines represent the no perturbation condition, the red-solid lines represent the fast perturbation, condition and the blue-dotted lines represent the slow perturbation condition.

Figure 5.

Changes in the path and direction of finger movement, in response to changes in target speed during interception sessions. Parts A and B show examples from two different paths for one subject (Subject 8). The left column depicts the 100 ms perturbation condition and the right column depicts the 400 ms perturbation condition. In the top panel of each part of the figure the target trajectory is shown for this condition (black thin lines). The target followed the trajectory along the direction shown by the arrows. The GO signal is represented by the solid black circle and the perturbation location (if there was one) is shown with the open circle. The finger average movement paths for this one subject in the one trajectory are shown emanating from the bottom of this aspect of the figure. Average finger movement paths are shown for all three perturbation magnitude conditions. In the bottom panel in each part of the figure the finger direction over time for each perturbation magnitude condition is shown over the time of the trial for the current target trajectory. Time zero is the time of the GO signal (not shown). The finger movement directions are color-coded and have varying line style; for the slower perturbation (−0.5, blue dotted lines for finger direction, blue line with the most hatching for finger position), the faster perturbation (+1.5, red solid line for finger direction and red line with the least hatching for finger position) and the control situation (0, green dashed line for finger direction and green line with intermediate hatching for finger position). Each trace is an average (±1SEM) of all successful first movement attempts in each condition. Vertical lines in each panel represent the time of finger movement onset (black) and the time finger direction separates (gray) across all subjects and all trials in the particular target trajectory and perturbation time condition.

Figure 6.

For the 400 ms perturbation, three more examples of finger direction and finger position patterns are shown for one target trajectory. Condition average examples are shown for A) Subject 1, B) Subject 2, C) Subject 4 in interception sessions. In each panel finger movement direction (±1SEM) over the time of the trial is on the left. Time zero (not shown) is the time of the GO signal. The all subject average time of finger movement onset (black vertical lines) and the all subject average time finger direction separated (gray vertical lines) is shown for this trajectory and this perturbation time condition in each plot. On the right side, target and finger position are shown for the condition. Finger position is the condition average for each subject ±1SEM in both the horizontal and vertical dimensions. The target trajectory is represented with a thin black line, with the location of the GO signal as the black filled circle and the perturbation location with a black open circle. Arrows show the direction of target motion along the trajectory. Data from slow perturbation conditions are shown with the blue dotted lines (finger direction plots) and blue lines with the most hatching (finger position plots); data from the no perturbation conditions are shown with the green dashed lines (finger direction plots) and green lines with intermediate hatching (finger position plots); and data from the fast perturbation conditions are shown with the red solid lines (finger direction plots) and red lines with the least hatching (finger position plots).

Figure 5A shows a target trajectory where the target was moving up during movement planning. Even though it is clear that the finger positions are different during the majority of the movement (left panel), the finger direction plot shows some overlap. Figure 5B shows a target trajectory where the target moved diagonally during movement planning. In the left part of Figure 5B the direction and position of the finger movement are clearly different in the three conditions and the subject took the perturbation into account for nearly the entire movement. If one compares this to the plots on the right side of Figure 5B (with a perturbation that occurred after the finger movement began), one can see that for the later perturbation the finger movement trajectories overlap nearly in their entirely in both position and direction. Changes were only made near the end of the path (more obviously in the finger direction plot because the finger was slowing at the end of the trial and thus position was not changing much).

From Figure 5 we estimated that the time of the perturbation had an impact on the finger movement path during the interception task. Thus we examined the path more closely for an idea of when this impact began. Figure 6 depicts finger position (right panels) and finger movement direction over time (left panels) for three subjects for the same perturbation time and target trajectory condition as in Figure 5. The vertical gray line represents the time (averaged across all subjects) that the finger direction is different in the fast perturbation and slow perturbation conditions. This figure shows that after the 400 ms perturbation it seems to take less than one visual reaction time to alter the finger movement path. In all subjects one can see a split in finger direction between 550–600ms after the GO signal (which is approximately 150–200 ms after the perturbation).

We quantitatively examined the impact of the perturbation time condition on the time that finger direction in slow and fast perturbations separate. We computed finger direction (±1SEM) for each subject and each condition and found the time that the finger movement direction in the slow and fast perturbation conditions stopped overlapping. We examined the split by comparing when the direction (as measured by ±1 SEM from the mean) differed and remained different through the end of the condition. We computed the time of this difference relative to the reaction time for the onset of finger movement. We found that when the perturbation was at 400 ms the time the direction split occurred later compared to all other perturbation times (F(3,92)=11.367, p<0.001). When the perturbation time was at 0 ms finger direction split on average at 123±114 ms after movement onset, at 100 ms: 102±82 ms, at 200 ms: 94±35ms and at 400 ms: 214±70 ms. At the latest perturbation time, on average this occurred after half the finger interception movement was completed (Figure 6A=67%, 6B=56% and 6C=62%).

We also examined when the finger path separated via a position measure. For this analysis we inspected the average finger position (±1 SEM horizontally and vertically) for every subject and condition. We compared the position in the slow perturbation condition to the fast perturbation condition and determined when the finger position no longer overlapped during each condition. We found that the finger position differed earlier in the 0–200 ms perturbation conditions as compared to the 400 ms perturbation time condition (F(3,91)=16.625, p<0.001; 0 ms condition: 52±84 ms, 100 ms condition: 47±41 ms, 200 ms condition: 99±56 ms, 400 ms condition: 172±108 ms). Thus, for the perturbations before the finger began to move, the finger direction and position generally seemed to show the difference relatively near the onset of movement (Figure 5 left panels). For the 400 ms perturbation, finger position and direction diverged at a time less than one reaction time. Thus as measured by both finger direction and position, it appears that a change within an ongoing movement may be produced more quickly than the onset of a response to a stimulus.

Discussion

In this set of experiments we examined how alterations in the movement plan are incorporated during the planning and execution of a quick interception movement. We did this by perturbing the speed of target motion in between the time of the GO signal and the interception of the target. We found that subjects were able to easily adapt to a change in target speed when that change occurred before the onset of the interception movement. We demonstrated this by showing that the initial path of the finger movement tended to match an interception trajectory aimed to the future target location only if the perturbation was taken into account (Figure 2B). For example if the target was moving from right to left after the GO signal a subject would have to initially move along a path farther to the left than the target location at the onset of finger motion in order to intercept it (Figure 2B; green-dashed-dotted trial). Then if there was an increasing speed perturbation the subject would have to aim even farther to the left at the onset of finger motion (Figure 2B red-solid trial); however if there was a slowing speed perturbation the subject would not have to move as far to the left (Figure 2B blue-dotted trial). Subjects were able to use information from very near the onset of finger motion to guide the path of the finger. This is consistent with previous findings from our laboratory (Mrotek & Soechting, 2007; Soechting et al., 2009). In fact, it has been shown that we can incorporate changes in target speed (using unusual speed profiles) into a movement plan if the speed change occurs more than 50ms before the onset of a finger movement (Soechting et al., 2009).

Finger Movement Response in Conditions with the Perturbation before Movement Onset

When the perturbation was close to the time of the GO signal the finger path (between fast and slow perturbation conditions) differed very early in the movement (Figure 5, left panel). Even so, the time required to alter the path tended to grow slightly with perturbations that occurred closer to the onset of the finger movement (as measured by the time finger position separated; in the 0 ms perturbation condition it was 52 ms and in the 200 ms perturbation condition it was 99 ms). Subjects were able to adjust to the perturbations when the perturbation occurred close to the onset of finger movement, but they adjusted marginally sooner if they had more time to utilize information about the perturbation (i.e. when the perturbation occurred closer to the GO signal). These results show that the movement plan is malleable during planning and execution. It does take time to incorporate the influence of environmental changes into the behavior, but the amount of time appears to be less than one reaction time. Updates to the behavioral plan used information after the perturbation and were added very quickly prior to or just at the beginning of the finger movement. It has been suggested that cell populations in posterior parietal cortex (specifically LIP, PRR, Area 5) are involved in carrying the signals related to decisions about hand movements utilizing visual information as a guide especially in situations where errors are detected (Batista et al., 1999; Shadlen & Newsome, 1996; Tanaka et al., 2009). It is likely that these plans are initially encoded in eye-centered coordinates and they are malleable in this coordiante frame before converstion to body-centered coordinates at another stage of processing (which could still be located in parietal cortex; Buneo et al., 2002). There also may be a relationship between the rate of neuronal firing in these parietal areas and in closely linked areas (i.e. dorsolateral prefrontal cortex, premotor cortex) where firing representing decisions to make a movement (or movement change in this experiment) are detected (Kim & Shadlen, 1999; Shadlen & Newsome, 1996). It is possible that the stimulus used in this experiment allowed for fairly rapid changes in the firing in these types of populations during hand movement planning because of the quick alterations in the hand movement behavior, but it is still likely that the processing was not instantaneous.

This is different than the model proposed by Gauthier and colleagues (1988). They proposed that there are two independent processing mechanisms for eye and hand movements. These mechanisms could, however, exchange information. In their report the subjects had to be trained to use the mechanisms for tracking without sharing signals. In our experiment the subjects received no such training. It is possible, however, to interpret the current results as emanating from two independent controllers that could share information during sensorimotor processing.

Finger Response to Perturbations after Finger Movement Onset

Similarly we showed that if the perturbation occurred after the onset of finger movement the gaze response to the perturbation was delayed less than one visual reaction time. Alternatively in a two-dimensional reaching movement where the target was perturbed, the response required approximately one reaction time (Georgopoulos et al., 1981; van Sonderen et al., 1988; van Sonderen et al., 1989). In those reports, the fact that a movement had already been planned did not seem to change how much time it took to incorporate the impact of the perturbation into the movement. However, those previous experiments altered the position of a stationary target location during the planning or execution of a reach movement. This information may indicate that it takes a similar amount of time to process new information while a person is already moving (and monitoring their movements for success), as it does to incorporate a change in the motor plan during initial processing to a typically non-moving target.

The results in the current report indicate that the alteration in the finger movement may require less than a typical reaction time and this amount of time is similar to previous one-dimensional reports with changes in target speed (Brenner et al., 1998). The response to a target direction perturbation during an ongoing hand movement takes even less time than the response to the speed change did (Brenner & Smeets, 1997; Prablanc & Martin, 1992).

The response may be in part due to the impact of smooth pursuit tracking during the tasks. Research has shown that the gain to a speed perturbation during a one-dimensional pursuit tracking task is quite strong (Schwartz & Lisberger, 1994; Churchland & Lisberger, 2001). Two-dimensional pursuit responses to speed/velocity changes are also strong and they are more complex (Engel et al., 1999; Leung & Kettner, 1997; Mrotek et al., 2006). Even so, all the responses had an approximate reaction time of 100 ms. It is possible that the processing utilized to change the smooth pursuit response during tracking can just be modified slightly (see above) to adjust the ongoing hand movement.

Correspondingly, the use of the visual cues for ocular pursuit requires sensory information to be utilized and motor decisions made in the same association cortices (and cerebellum) for eye movements as where hand movement decisions and plans might be made (Fu et al., 1997; Duhamel et al., 1997; Kim & Shadlen, 1999; Leung et al., 2000). It is also possible that since similar processing occurs for eye and hand movements, this can be interpreted as allowing subjects to begin moving the hand before hand movement processing is completed and in the meantime signals from the eye movement processing are used. Alternatively in this experiment, the hand movement had already engaged this same system thus the system may be primed for further decisions and plans for the eye (i.e. sensory feedback with a feedforward plan may engage both systems; Gauthier & Hofferer, 1976). Overall, the processing needed to guide ongoing eye movements is partially the same processing needed for guiding hand movements and vice versa so information can be shared quickly.

Finger Reaction Time after Early/Fast Perturbations

If a fast perturbation occurred close to the GO signal, it impacted the time needed to process the information and begin the finger movement. When the perturbation was at the same time as the GO signal, the reaction time decreased by approximately 44 ms relative to when the perturbation was near the time of the onset of finger movement (Figure 4B). Similarly, the reaction time for a response in smooth pursuit also decreased (28 ms) when the perturbation occurred at the same time as the GO signal relative to the two perturbations near the onset of finger movement (gray bars, Figure 4A). This change was much less if subject were asked to track the target with the eyes only and not make an interception movement (14 ms; white bars, Figure 4A). It is possible that the fast target speed paired with the GO stimulus resulted in an intense stimulus (relative to the others) and thereby increased the speed of the processing. Others have shown that more intense stimuli results in shorter reaction times (Bell et al., 2006; Carreiro et al., 2011; Jaskowski & Sobieralska, 2004; Lakhani et al., 2012; Smeets & Brenner, 1995; van Donkelaar et al., 1992; Veerman et al., 2008). Additional support for this interpretation is that target speed was a predictor for maximal finger speed during the interception task (although to a lesser extent than distance travelled) and other have shown this to be true in one dimension as well (Carnahan & McFadyan, 1996; Savelsbergh et al., 1992; Smeets & Brenner, 1995; van Donkelaar et al., 1992; Veerman et al., 2008).

Eye Interactions with Hand Movement

There are strong interactions between hand and eye behaviors during anticipation, tracking and pointing tasks that do and do not include perturbations (Prablanc & Martin, 1992; van Donkelaar et al., 1994; Engel et al., 2000; Barnes & Marsden, 2002; Henriques & Crawford, 2002; Gielen et al., 2009). Using smooth pursuit response time as a measure in the current experiment, it appears that the occurrence of an impending hand movement slows the onset of the pursuit response to the target speed perturbation. It is possible that this is a similar mechanism as shown by Engel and colleagues (2003) in that the eye slows to match the movement capabilities of the hand during eye/hand tracking in two dimensions. However the results of the current experiment also show that smooth pursuit gain is aided by the impending performance of a hand movement. Others have demonstrated that during smooth pursuit tracking in one dimension the addition of hand tracking improved the pursuit gain, but not lag, and can be improved with practice (Gauthier et al., 1988; Koken & Erkelens, 1992). It is possible that the sensorimotor system controls both effectors together and thus the eye is delayed so it can use the predicted finger-target interception location as a target. Meanwhile during the planning of the hand movement itself (efference copy), this could ready the system and enhance pursuit. Alternatively, in Gauthier and colleague’s model (1988) the processing of the hand movement and kinesthetic feedback of the hand directed through the Coordination Control Center could spill into the eye movement controller to increase output from the system plant. To avoid this, mediation of the signals could occur with maturation and/or training.