Eye–Hand Coordination during Learning of a Novel Visuomotor Task

Abstract

We investigated how gaze behavior and eye–hand coordination change when subjects learned a challenging visuomotor task that required acquisition of a novel mapping between bimanual actions and their visual sensory consequences. By applying isometric forces and torques to a rigid tool held freely between the two hands, subjects learned to control a cursor on a computer screen to hit successively displayed targets as quickly as possible. The learning occurred in stages that could be distinguished by changes in performance (target–hit rate) as well as by gaze behavior and eye–hand coordination. In a first exploratory stage, the hit rate was consistently low, the cursor position varied widely, and gaze typically pursued the cursor. In a second skill acquisition stage, the hit rate improved rapidly, and gaze fixations began to mark predictively desired cursor positions, indicating that subjects started to program spatially congruent eye and hand motor commands. In a third skill refinement stage, performance continued to improve gradually, and gaze shifted directly toward the target. We suggest that during the exploratory stage, the learner attempts to establish basic mapping rules between manual actions and eye-movement commands. In this process, subjects may establish correlations between hand motor commands and their visual sensory consequences, primarily in fovea-anchored, gaze-centered coordinates, and correlations between recent hand motor commands and eye motor commands. The established mapping rules are then implemented and refined in the skill acquisition and refinement stages.

Introduction

Skilled motor behaviors involve the ability to predict the consequences of one's actions, and motor learning involves the acquisition of new maps relating motor commands and desirable sensory outcomes (Wolpert and Miall, 1996; Kawato, 1999). During the early stage of motor skill learning, performance of an unfamiliar task is usually uncoordinated and requires considerable effort in making decisions and executing movements (Fitts and Posner, 1967; Brooks et al., 1995; Willingham, 1998). In this early stage, during which actions may not be able to be controlled as to achieve sensory goals or improve performance, the learner attempts to discover the basic mapping rules relating motor and sensory signals before they can be implemented so as to improve performance. For example, when a child learns to ride a bike, she must first discover the basic coordination needed for stability before she can even perform the task and begin to improve. Most studies of learning of visually guided manual skills have examined goal-directed reaching under novel load or visuomotor conditions (Newell and Rosenbloom, 1981; Cunningham, 1989; Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994; Krakauer et al., 1999, 2000; Flanagan et al., 2003). These reaching tasks do not require an exploratory stage in that performance improves from the first trial onward. This suggests that the learning in these tasks may involve adaptation of previously acquired basic sensorimotor mapping rules rather than the discovery of novel rules.

Although there is a growing body of work on eye–hand coordination (for review, see Crawford et al., 2004), little is known about the role of gaze in learning of visually guided manual tasks. Here, we examined eye–hand coordination during acquisition of a task that in most subjects required a period of exploratory behavior before improvement in performance occurred. Thus, we asked how humans exploit gaze fixations during learning, starting at the stage when the manual performance is consistently poor, and how the coordination of eye and hand actions may change when performance subsequently improves. The task required controlling a cursor to hit successively appearing visual targets (presented on a screen) by applying forces and torques to a tool held between the hands. We expected that once the task was well learned, gaze would shift directly to the new target and remain there until the cursor arrived, as is observed when reaching to targets under natural conditions (Prablanc et al., 1979; Prablanc and Martin, 1992; Neggers and Bekkering, 2000). One possibility is that the same gaze behavior will be observed throughout learning with gaze shifting directly to the target and the visual consequences of hand actions being monitored in peripheral vision. An alternative hypothesis is that gaze fixations will be directed to the poorly controlled cursor in early stages of learning. Confirmation of the latter hypothesis would suggest an important role of foveal vision and gaze shifts in learning novel mappings between hand motor commands and desirable sensory outcomes.

Materials and Methods

After providing written informed consent, 10 students (4 women and 6 men; age, 21–34 years) with normal vision participated in the study. Nine were right-handed, and one was left-handed (Oldfield, 1971). The ethics committee of the University of Umeå approved the study. With their elbows supported on a table, subjects sat in front of a vertical computer screen (24.5 × 18.5 cm, 60 frames per second, 800 × 600 pixels) located ∼37 cm from the eyes. The task was to move a cursor to hit successively displayed targets by applying forces to a rigid tool held between the two hands freely in the air (Fig. 1 A).

Figure 1.

Apparatus and target distribution on screen. A, Tool used by the subject to control a cursor on the screen by bimanually applying forces and torques about the long axis of the tool. B, The corresponding arrows in A and B indicate the two different mapping rules relating hand forces and torques to cursor movements. C, Distribution of target positions (filled circles) in which consecutively appearing targets are connected with a line; the target shape (square) is shown to scale in the bottom right corner.

Apparatus. Gaze position of the subject's right eye was recorded at 120 Hz using an infrared video-based system (RK-726PCI; ISCAN, Burlington, MA). A bite-bar was used to stabilize the head. A detailed description of the apparatus, calibration procedures, and the accuracy of gaze measurements is provided in a previous report (Johansson et al., 2001). The subjects held the tool horizontally by grasping cylindrically shaped polyvinyl chloride handles (diameter, 3.4 cm; length, 3 cm) attached to each side of a rectangular box (12 × 8 × 3.5 cm; weight, 255 g) (Fig. 1 A). The handles were grasped between the thumb and two fingers (index and long finger) of each hand. Because the rigid tool was held free and almost stationary in the air, the two hands generated forces of practically equal magnitude, but in opposite directions. A force/torque transducer attached to one of the handles (F/T-Nano 17; Assurance Technologies, Garner, NC) and fixed inside the box measured the applied forces at 200 samples per second. A 20-cm-wide vertical wooden stand supporting the eye tracker blocked the subjects' view of the tool during the experiment.

Compressing and stretching forces applied between the handles along the longitudinal axis of the tool moved the cursor horizontally, and torques applied around this axis moved the cursor vertically. Thus, moving the cursor diagonally required a combination of longitudinal forces and torques. The forces controlling the cursor were low-pass filtered at 5 Hz. With no forces applied, the cursor was in the center of the screen. The position of the cursor, relative to the screen center, scaled linearly with force. Applying a force of 1N tangential to the surface of the handles resulted in a cursor movement of 4.8° visual angle; 1N twist force corresponded to a torque of 1.75 Ncm.

Task. The task was to move the cursor (Fig. 1, filled circle) (3 mm diameter or 0.5° visual angle) to hit a target (Fig. 1, open square) (14 × 14 mm or 2.2 × 2.2°) displayed on the screen. The target was hit when the centers of the cursor and target were <5 mm (0.8°) apart for at least 100 ms. When the target was hit, subjects heard a click sound (1 kHz, 5 ms duration), and the target reappeared at a new unpredictable position.

Each subject experienced two sessions for which the mapping rule relating manual actions and cursor movements differed (Fig. 1 B). With one rule, the cursor moved leftward when the tool was subjected to compression forces and downward when the hands produced torques in the clockwise direction (Fig. 1 B, left). In the other session, the mapping was reversed. That is, the cursor moved rightward with longitudinal compression forces and upward with clockwise torque (Fig. 1 B, right). Five subjects started with the first mapping rule, and five subjects started with the second mapping rule. There was a pause of ∼5 min between the sessions.

The target was located at 44 different positions. These were distributed with equal probability across the four quadrants of the screen under the constraints that the distance between two successive targets was 18° and that the direction from the previous target was uniformly distributed between 0 and 360° (Fig. 1C). In a given session, the same sequence of 44 target locations was repeated until 500 targets were hit. Pilot experiments indicated that the increase in performance, measured as the target hit rate, markedly attenuates after some 300 targets were hit. After each 100 s of the task, there was a rest period lasting for 30 s.

Before the experiment, subjects were instructed to hit the targets as fast as possible. They were informed that they could move the cursor horizontally by applying forces longitudinally between the handles and vertically by applying twist forces, but they received no additional instructions about the prevailing mapping rule. To encourage fast learning, the subject received 0.20 Swedish krona for every hit and an additional 2 Swedish krona for every 100 s period during which they broke their own previous record in terms of the number of hits. During each 30 s rest period, the screen provided feedback about the number of hits in the previous 100 s period, the maximal number of hits in any period, and the amount of money earned.

Data analysis. For each target transition, we assessed the instantaneous hit rate as the inverse of the time between the previous and current target hits and a path index computed as the ratio between the distance the cursor traveled between hits and the actual intertarget distance. We also assessed the number of saccades and the number of cursor submovements that occurred during each target transition. Although the values of these variables (obtained at each hit) generally changed with practice, they could fluctuate from hit to hit. To obtain a measure of the overall change of these variables as a function of time for plotting, for each session we first resampled them at 5 Hz. For example, if the first target transition took 10 s, the hit rate was considered to be 0.1 hits per second for this period, and we would have obtained 50 samples with a hit rate of 0.1 for the period. We then filtered the resampled variables with a moving median over ±100 s. We also applied a moving median over ±100 s when plotting variables pertaining to discrete gaze and hand events (saccades and hand submovements; see below).

Saccades and gaze fixations were defined as described previously (Johansson et al., 2001). Hand (i.e., cursor) movements were classified into submovements. To determine these submovements, we first low-pass filtered the force and torque signals (5 Hz cutoff frequency) and then calculated the tangential velocity (i.e., speed) of the cursor as the vectorial sum of the first time derivatives of cursor position in x and y coordinates using ±9-point numerical differentiation. The cursor movements generated by the subjects had approximately bell-shaped speed profiles, and a submovement was considered to have occurred when the negative peak of the second time derivative of cursor speed dropped below –24,000°/s³ based on ±24-point numerical differentiation. After testing various procedures for identification of submovements, we found that this approach captured most cursor submovements that could be distinctly discerned during visual inspection of the speed signal. Start and end times of each detected submovement were defined as the times of the first and second positive peaks of the second time derivative of cursor speed obtained using ±2-point differentiation for adequate temporal resolution. Saccades and submovements of amplitudes <1° were not scored. The instantaneous saccade rate and cursor submovement rate were computed as the inverse of the interval between the onset of the previous and current saccades and submovements, respectively. Saccadic gaze shifts that started and ended within 3° of the center position of a target were considered as target refixations and were not analyzed further. We chose 3° because this is considered to be the size of the functional fovea in manipulation tasks (Johansson et al., 2001; Terao et al., 2002) (but see Coren, 1986). Of 30,701 saccades recorded in total, 7682 (∼25%) were target refixations. Blink saccades were identified but not analyzed.

Saccadic displacement was measured as the straight-line distance between the start and end position of the saccade. Saccade direction was defined both in screen coordinates (0° = right, 90° = up) and with reference to the current target position (0 and ±180° indicate a saccade directly toward and in the direction opposite to that of the target, respectively). We obtained corresponding measures for each detected cursor submovement.

We used two approaches to examine whether gaze fixations, directed to recent, current, or future cursor positions, were leading or lagging the cursor. First, for each gaze fixation, with reference to fixation onset, we looked for the time when the distance between gaze and cursor positions was minimal within a time window from 400 ms before fixation onset to the end of the current target transition interval. Again guided by the estimated size of the functional fovea, we only included fixations with distances <3° between cursor and gaze, which captured 86% of all saccadic gaze fixations recorded. Positive and negative times indicated that the cursor arrived at this position after and before gaze did, respectively. Second, we calculated the change in distance between cursor position and gaze that occurred during each saccade. Negative and positive values characterized saccades that shifted gaze toward or away from the cursor, respectively.

Repeated-measures ANOVAs were used to assess experimental effects; when appropriate, the dependent variable was logarithmically transformed to obtain approximately normal distributions. Unless stated otherwise, in these ANOVAs we used subject medians. Post hoc analyses were performed with the Tukey's honestly significantly different test. The nonparametric Spearman rank correlation test was used to assess correlations (r_s). A p value of 0.05 was considered to be statistically significant.

As described below, we selected only one of the two sessions (and hence mapping rules) that each subject performed for analysis. Importantly, we observed that hit-rate performance was not affected by the mapping rule or the order (first or second) in which the two rules were experienced. A repeated-measures ANOVA with epoch (the first hit versus last five hits) and mapping rule (map 1 vs map 2) as within-subjects factors and mapping rule order (first versus second) as a between-subject factor did not reveal the main effects of map or map order, and there were no interactions between any of the factors. As expected, there was a main effect for epoch (F_(1,8) = 121.0; p < 0.0001) verifying that learning occurred. Similar results were obtained for the path index with a main effect for epoch (F_(1,8) = 45.80; p < 0.0001) but no effects pertaining to mapping rule and mapping rule order.

Visual inspection of the hit-rate data suggested that learning occurred in three stages with an initial exploratory stage with no improvement in performance, a skill acquisition stage during which the hit-rate performance increased strongly, and a final skill refinement stage in which performance continued to increase gradually. To quantitatively delineate these stages, we fit a five-parameter piecewise linear function to the filtered hit-rate curve. The function consisted of three adjoining lines with the first constrained to be horizontal. Using linear regression, we tested this five-parameter model against three submodels: a four-parameter model with two lines, a three-parameter model with two lines (the first of which was constrained to be horizontal), and a two-parameter model with a single line. We based our analysis on sessions that were best fit by the three-stage, five-parameter model (see Results). To combine data from different subjects for plotting while preserving stage information, we normalized the time base of each learning stage to the median duration of that stage measured across all sessions with three learning stages.

Results

Learning reflected in task performance

All subjects gradually learned the novel visuomotor task as reflected by an overall increase in the hit rate during the practice sessions. Although the time course of learning varied across subjects, visual inspection of data from all 20 sessions (10 subjects by two visuomotor mapping rules) suggested that the learning typically occurred in three stages (Fig. 2A). At the beginning of each session, performance was poor and could remain poor for a considerable period. We refer to this first stage as the exploratory stage of learning because subjects were very active with the cursor and yet performance did not improve. In the next stage, termed the skill acquisition stage, the hit rate increased quite distinctly. This stage was followed by a third stage, termed the skill refinement stage, characterized by a more gentle improvement in performance.

Figure 2.

Learning of the novel visuomotor task as reflected by an increasing hit rate and decreasing cursor path index. A, The hit rate as a function of time for one practice session by each of the eight subjects that displayed three learning stages. Data points indicate, for each target hit, the instantaneous hit rate. The thick gray curve shows the hit rate filtered with a moving median of ±100 s. The superimposed thin black lines shows the piecewise linear function fitted to the hit rate curves to distinguish transitions between learning stages. The shaded area represents the skill acquisition stage. B, Examples of sessions with only two learning stages (same format as in A). C, Superimposed thin lines show the filtered hit-rate curves on a normalized time base for the same eight subjects shown in A. The thick curve represents the mean values of these subject curves, and the shaded zone provides the range. Ex, Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage. D, Filtered cursor path index for individual subjects shown as a function of normalized time using the same format as in C. Note the logarithmic scale of the ordinate.

A regression model including these three stages provided best fits in 13 of the 20 sessions recorded (see Materials and Methods) (Fig. 2A, thin black lines). For these 13 sessions, the median durations of the exploratory, skill acquisition, and skill refinement stages were 360 s (range, 92–1269 s), 410 s (range, 140–611 s), and 509 s (range, 313–678 s), respectively. The other seven sessions only showed two clear stages, with no obvious exploratory stage in five sessions (Fig. 2B, left) (cf. Brooks et al., 1995) and no obvious skill refinement stage in two sessions (performed by one subject) (Fig. 2B, right).

To obtain a balanced experimental design, we focused our analysis on data obtained in sessions that showed all three learning stages and included data from only one session per subject. If the first session showed all three stages, we used data from this session. Otherwise, we used data from the second session. Eight of the 10 subjects exhibited at least one session with all three stages. Figure 2A shows the hit rate as a function of time for the selected sessions, and the superimposed curves in Figure 2C (thin lines) show, for the same sessions, the filtered hit-rate curves (moving median of ±100 s) on a normalized time base. Analysis of the sessions with only two learning stages indicated that the gaze behavior and eye–hand coordination was similar to that observed for the corresponding stages recorded in sessions with all three stages.

In summary, although variable across subjects and practice sessions, the learning typically occurred in three successive stages: the exploratory stage in which the hit rate was consistently low, the skill acquisition stage in which the hit rate increased quite distinctly, and the skill refinement stage in which the hit rate more gently increased.

Changes in gaze and manual behavior during learning

As shown in Figure 3, substantial changes occurred in both gaze and manual behavior during learning. In the exploratory stage (Fig. 3A), subjects generated numerous manual submovements that frequently resulted in cursor movements that were vertical and horizontal. Likewise, subjects generated numerous saccadic gaze shifts. Gaze intermittently fixated the target but most often appeared to pursue the cursor, often fixating recent cursor positions (Fig. 3B). Later, in the skill acquisition stage (Fig. 3C), the cursor path between the successively displayed targets gradually became straighter, the number of saccades and cursor submovements per target transition decreased, and gaze more often fixated the actual cursor position or a forthcoming cursor position. With additional practice, gaze fixations mainly marked forthcoming cursor positions. In the skill refinement stage (Fig. 3D), the cursor path became even straighter, and the saccades were principally directed to the target. Thus, when a new target came up after a target hit, both a saccade and a cursor movement were launched, at about the same time, in the direction of the new target position (18° away). Gaze was shifted to the target either directly or by two saccades, the first of which brought gaze to a position en route to the target. In either case, gaze remained at the target until the cursor arrived. Once the cursor was near the target, it often took subjects a considerable amount of time to actually hit the target. This homing in behavior is illustrated in Figure 3E, which shows time-varying gaze and cursor positions and velocities using the same data shown in Figure 3C. Such homing in was observed even in the skill refinement stage in which 48 ± 20% (mean ± 1 SD) of the total target transition time occurred after the cursor first arrived within 3° of the center of the target. The variable duration of this homing in period contributed strongly to fluctuations in the hit rate.

Figure 3.

Gaze and cursor movements for examples of target transitions during a practice session by one subject. A–D, Dashed and solid lines represent the positions of the cursor and gaze on the screen, respectively, and squares indicate target positions. The numbers in bold indicate successive gaze fixations, and the numbered circles indicate the cursor position at the start of the corresponding fixation. A, Exploratory stage. Starting at the center of the screen, the subject moves the cursor to the very first target (square). For this target transition, the path index was 63.4 (ratio between the distance traveled by the cursor and the intertarget distance). B, A 2 s epoch of the movements shown in A; the most right point of the cursor trajectory is marked by an asterisk, which corresponds to the asterisk inA. C, D, Example of target transitions during the skill acquisition and skill refinement stages, respectively, with path indices of 2.4 (C) and ∼1.2 (D). E, Positions of the target, cursor, and gaze together with cursor and gaze tangential velocity as a function of time during the movements shown in C. Thick-line segments of the cursor velocity trace mark scored hand submovements, the start and end of which are indicated by vertical lines.

The increase in the hit rate observed during skill acquisition and refinement (Fig. 2A,C) was accompanied by a straightening of the cursor path. The path index (ratio between the distance traveled by the cursor between hits and the intertarget distance) decreased (Fig. 2D). One-way, repeated-measures ANOVAs showed an effect of learning stage on both the hit rate (F_(3,21) = 176.6; p < 0.0001) and path index (F_(3,21) = 55.7; p < 0.0001). In these and subsequent ANOVAs, we represented the stage of learning at four levels: the exploratory stage (Fig. 2C, Ex), the first and second halves of the skill acquisition stage (SAa and SAb), and the skill refinement stage (SR). The split of the skill acquisition stage allowed us to quantify large behavioral changes that occurred during this stage. Post hoc analysis indicated significant differences between the successive learning stages both for the hit rate (p < 0.001 for all three pairs) and the path index (p < 0.05 for all pairs). Likewise, for each subject, during the 500 target transitions, there was a significant negative correlation between the hit rate and path index (–0.75 < r_s <–0.91; p < 0.0001 in all 8 subjects).

Frequency and amplitudes of saccades and manual submovements

Both the number of saccades and the number of manual (cursor) submovements per target transition decreased substantially with learning (Fig. 4A,B)(F_(3,21) = 56.3, p < 0.0001 and F_(3,21) = 97.7, p < 0.0001, respectively). The drop in the number of saccades per target transition primarily occurred in the first half of the skill acquisition stage, whereas the number of submovements decreased throughout this stage. Post hoc analysis indicated no difference between the second half of the skill acquisition stage and the skill refinement stage for the number of saccades (p > 0.999), but differences between all successive phases were observed for the number of manual submovements (p < 0.05 for all three pairs). The gradual decrease in the number of submovements was primarily because of the shortening of target transition intervals, because the submovement rate (submovements per second) did not change appreciably with learning (Fig. 4D)(F_(3,21) = 2.3; p = 0.1). In contrast, the saccadic rate decreased somewhat (Fig. 4C) (F_(3,21) = 23.4; p < 0.0001) and thus contributed to the decrease in the number of saccades during the skill acquisition stage. The number of target fixations (i.e., gaze within the 3° radius target zone) decreased with learning (Fig. 4E) (F_(3,21) = 26.9; p < 0.0001), and in the second half of the skill acquisition stage and the skill refinement stage, subjects typically entered the target zone once per target transition. The higher number of target fixations during early learning indicated that gaze exited and reentered the target zone during the target transition interval.

Figure 4.

Occurrence of saccades and manual submovements. A, B, The number of saccades and number of manual (cursor) submovements per target transition. Note the logarithmic scale of the ordinate. C, D, The rate of saccades (saccades per second) and rate of submovement (submovements per second). E, The number of saccades that brought the gaze into the 3°radius target zone per target transition. In all panels, the superimposed thin lines plot, on the normalized time base, data from the eight subjects shown in Figure 2 A after filtering with a moving median of ±100s.The thick line gives the mean values for those subject curves, and the shaded zone provides the range. Learning stages are indicated as in Figure 2C.

In summary, although the rate of manual submovements did not change appreciably with learning, their number per target transition decreased substantially when the transition times rapidly decreased in the skill acquisition stage. The decrease in the number of saccades per target transition mainly occurred in the first half of the skill acquisition stage because the saccadic rate also decreased during this stage.

Temporal coordination between gaze fixations and hand actions

In addition to the target, subjects often directed gaze to recent, current, and forthcoming cursor locations (Fig. 3A–C). To assess whether gaze fixations were leading or lagging cursor positions, we computed, for each gaze fixation, the time of the minimal distance between gaze and cursor position relative to the onset of the fixation. With gaze pursuing the cursor, this time is negative, and with gaze fixations leading forthcoming cursor positions, it is positive. This lead/lag time changed with learning (Fig. 5) (F_(3,21) = 48.1; p < 0.0001) and, in all subjects, was positively correlated with the instantaneous hit rate at the current gaze fixation (0.23 < r_s < 0.57; p < 0.0001 in all cases) (Fig. 5A). In the exploratory stage, the lead/lag time was typically negative, indicting that gaze tended to pursue the cursor (Fig. 3A,B). At around the time of the transition between the exploratory and the skill acquisition stages, the lead/lag time began to increase markedly (Fig. 5A,B). Early in the skill acquisition stage, it changed from generally negative to generally positive values, and gaze, on average, led the cursor for the rest of the session. Post hoc analysis indicated differences for the lead/lag time between the exploratory and early skill acquisition stages and between the early and late skill acquisition stages (p < 0.002 in both cases) but not between the late skill acquisition and the skill refinement stage (p = 0.43).

Figure 5.

Postsaccadic fixations leading versus lagging the cursor position. A, Data points show, for a single subject (Fig. 2 A, asterisk), the difference between the time at which the distance between gaze and the cursor reached a minimum and the fixation onset (negative time implies cursor lead) for each relevant gaze fixation (see Materials and Methods) as a function of practice time. The curve shows the same data filtered with a moving median of ±100 s. B, The corresponding filtered curves obtained in all eight subjects shown in Figure 2 A (same format as in Fig. 2C). C, Distribution of this time difference including data from all relevant gaze fixations obtained from all subjects. Each of the four columns represents data from one of the four learning stages (Fig.2C). Negative times (cursor leading gaze) are indicated by the black areas of the histograms. Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

During the exploratory stage (Ex), when gaze pursued the cursor most frequently, the mode of the lead/lag time was approximately –200 ms (Fig. 5C, left), suggesting that saccades were reactively programmed for fixating the current cursor position. However, because of the reaction time of the saccadic system (typically ∼200 ms during reaching to a single visual target) (Biguer et al., 1982; Gribble et al., 2002) and the lively movements of the cursor, the cursor could be located at a different position when the postsaccadic fixation started (Fig. 3B). For gaze fixations lagging cursor positions (negative times), the cursor had moved 2.5 ± 2.2° (mean ± 1 SD) from its position at the time of the minimum distance to the time of the start of the fixation.

We also assessed the temporal coordination between gaze shifts and hand actions based on the time it took for the gaze and cursor to leave a zone of 3° around the last target after presentation of the next target. The average saccadic latencies (based on subject means) across the four task phases (exploratory stage, first and second halves of the skill acquisition stage, and skill refinement stage) were 344, 305, 282, and 262 ms, and the corresponding hand latencies were 345, 309, 273, and 251 ms, respectively. A two-way, repeated-measures ANOVA revealed an effect of phase (F_(3,21) = 17.31; p < 0.0001) but no effect of effector (p = 0.77) and no interaction between effector and phase (p = 0.57). Thus, the hand neither lagged nor led the eye, suggesting that initial gaze and hand movements were specified in parallel based on information about the new target position in peripheral vision as indicated previously for reaching movements (Biguer et al., 1982; Gribble et al., 2002). Note that our measurements of hand actions involved isometric forces and were therefore not subject to delays attributable to the inertia of the arm (Biguer et al., 1982; Gribble et al., 2002).

In summary, most gaze fixations tended to pursue the cursor in the exploratory stage. In the early skill acquisition stage, the gaze behavior changed from generally lagging to leading the cursor position.

Amplitudes of saccades and cursor submovements

The overall distribution of saccade displacements exhibited two main modes corresponding to short (∼3°) and long (∼15°) saccades. The balance between these modes changed with learning (Fig. 6A–C). In the exploratory stage, the majority of saccades were short. As performance improved in the skill acquisition stage, more and more longer saccades were observed. In the skill refinement stage, subjects typically made one long (12–18°) saccade from the previously hit target to the new target or the vicinity of the new target (18° away). The short saccades in this stage typically represented a second saccade, bringing gaze to the target when the long saccade undershot the target (Fig. 3D). Changes in saccade displacements with learning were verified by a main effect of learning stage on the subject medians (F_(3,21) = 42.3; p < 0.0001). Furthermore, despite the bimodal distribution of saccade displacement, in each subject, saccade displacement was positively correlated with the instantaneous hit rate (0.31 < r_s < 0.52; p < 0.0001 in all subjects). In Figure 6C, the black areas of the histograms represent the subpopulation of saccades that resulted in fixations that lagged the position of the cursor (Fig. 5, negative lead/lag times). The vast majority of the (short) saccades in the exploratory stage lagged the cursor position, but in the second half of the skill acquisition stage and especially in the refinement stage, such saccades were rather rare. The displacement of cursor submovements also increased with learning (Fig. 6D)(F_(3,21) = 18.7; p < 0.0001), but the size of this increase was less pronounced than for saccades, and the distribution of submovement amplitudes was unimodal rather than bimodal in all learning stages (Fig. 6D).

Figure 6.

Amplitudes of saccadic gaze displacement and cursor submovements. A, Data points indicate saccade displacements as a function of practice time for a single subject (Fig. 2 A, asterisk). The curve shows the same data filtered with a moving median of ±100 s. B, Corresponding filtered curves obtained for the eight subjects shown in Figure 2 A (same format as in Fig. 2C). C, Distributions of saccadic displacements including data obtained from all subjects. The black areas of the histograms indicate saccadic displacements for which the onset of the postsaccadic fixation lagged the position of the cursor (Fig. 5, negative times). D, Frequency distribution of cursor submovement displacements including data obtained from all subjects. C, D, Learning stages represented as in Figure 5C. Ex, Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

In summary, most saccades in the exploratory stage were short. Gradually longer saccades were observed in the skill acquisition stage. In the skill refinement stage, long saccades typically brought the gaze from the previously hit target to the vicinity of the new target.

Spatial distribution of gaze fixations with reference to cursor and target positions

During learning, there was a shift in the relative salience of the cursor and the target as gaze-fixation goals. In the exploratory stage, saccades generally brought gaze toward the poorly controlled cursor, whereas in later stages, a larger and larger proportion of saccades moved gaze toward forthcoming cursor positions and thus also toward the target. This change in gaze behavior was demonstrated by the effects of the learning stage on the change in distance between gaze and cursor positions that occurred during saccades (Fig. 7A–C) (F_(3,21) = 34.5; p < 0.0001). During the exploratory stage, the majority of saccades moved the gaze toward the cursor (Fig. 7A–C, negative values), and most of the postsaccadic gaze fixations were within a few degrees of the position the cursor was at when the saccade was initiated (Fig. 7D). With increasing performance, an increasing proportion of saccades moved gaze away from the cursor (Fig. 7C, positive values), compatible with postsaccadic fixations being located at future cursor positions. Likewise, the distance between the cursor position and the site of the postsaccadic gaze fixations gradually became longer with the progression of learning (Fig. 7D)(F_(3,21) = 32.0; p < 0.0001). These changes in gaze behavior occurred mainly during the skill acquisition stage. That is, for both the gaze–cursor distance change and the postsaccadic gaze–cursor distance, post hoc analysis showed a difference between the early and late skill acquisition stages (p < 0.001 in both cases) but neither between the exploratory and the early skill acquisition stage (p > 0.57) nor between the late skill acquisition and the skill refinement stage (p > 0.30). In early learning, a substantial fraction of the saccades directed toward the cursor lagged the actual cursor position (Fig. 7C, black areas of the histograms). This is consistent with the observation that gaze tended to pursue the cursor during this stage (Fig. 3A,B).

Figure 7.

Distance between gaze fixations and cursor and target positions and changes in these distances during a saccade. A, Data points indicate, for a single subject (Fig. 2 A, asterisk), the change in distance between cursor and gaze during each saccade plotted as a function of practice time (negative distance implies that gaze shifted toward the cursor). The curve shows the same data filtered with a moving median of ±100 s. B, Corresponding filtered curves obtained for the same eight subjects shown in Figure 2 A (same format as in Fig. 2C). C–F, Frequency distributions including data from all saccades or submovements obtained from all subjects showing gaze–cursor distance changes during saccades (C), distance between gaze and cursor at the onset of gaze fixations (D), gaze–target distance changes during saccades (negative distance implies that gaze shifted toward the target) (E), and distance between gaze and target at the onset of gaze fixations (F). Learning stages are represented as in Figure 5C. The black areas of the histograms in C and E refer to postsaccadic fixations that lagged the position of the cursor (Fig. 5, negative times). Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

The effect of learning on saccadic behavior was also reflected by the change in distance between the target and gaze position that occurred during saccades (Fig. 7E) (F_(3,21) = 49.6; p < 0.0001). During the exploratory stage, when most saccades were rather short and pursued the cursor, gaze shifts did not, on average, change the gaze-to-target distance (mean, –0.3°) (Fig. 7E, left). In contrast, in the skill acquisition stage, an increasing proportion of saccades reduced the distance between gaze and target (Fig. 7E, negative values), and in the skill refinement stage, virtually all saccades brought the gaze in the direction of the target. As a result, the distance between the position of postsaccadic gaze fixation and the target decreased while the learning progressed (Fig. 7F)(F_(3,21) = 59.3; p < 0.0001). As noted above, in the late skill acquisition and refinement stages, subjects tended to generate a long (12–18°) initial saccade that was often followed by additional shorter saccades (Fig. 3C,D). This resulted in the bimodal distribution of the cursor-to-target distance change seen in Figure 7E for these stages. For both the gaze–target distance change and the postsaccadic gaze–target distance, post hoc analyses showed a difference between the early and late skill acquisition stages (p < 0.01 in both cases) but not between the second half of the skill acquisition stage and the skill refinement stage (p > 0.19 in both cases).

In summary, saccades generally brought gaze toward the poorly controlled cursor during the exploratory stage. A gradually increasing proportion of saccades moved gaze toward forthcoming cursor positions during the skill acquisition stage. Virtually all saccades in the late skill acquisition stage and in the skill refinement stage reduced the distance between gaze and target.

Coevolution of target-directed gaze and cursor submovements

Regardless of the learning stage, there was a tendency to direct gaze shifts toward the target. Typically, a first saccade was directed to the target when it appeared at a new location. Likewise, the target was always fixated when hit. In the exploratory stage and in the first half of the skill acquisition stage, although gaze primarily pursued the cursor, subject nevertheless fixated the target intermittently (Fig. 4E), presumably to update information about its position. Overall, the direction of saccades with reference to the target changed with learning (Fig. 8A–C). During the exploration stage and during the first half of the skill acquisition stage, saccades became gradually more target directed (Fig. 8A,B). In the second half of the skill acquisition stage and in the skill refinement stage, saccades were almost exclusively directed to the target. A similar change in directional distribution occurred in the cursor submovements (Fig. 8D,E). However, this change appeared less pronounced and more gradual than for the saccades. A two-way, repeated-measures ANOVA (effector by learning phase) revealed main effects of the learning stage (F_(3,21) = 176.8; p < 0.0001) and effector (F_(1,7) = 13.0; p < 0.01) on the angular dispersion (Zar, 1996) of the distribution of movement directions (gaze and cursor) relative to the target. Overall, the dispersion was greater for submovements than for saccades (Fig. 8, compare C, D). Furthermore, an interaction between effector and learning stage (F_(3,21) = 36.4; p < 0.0001) verified a different pattern of directional changes in gaze and cursor submovements. Notably, there was no significant difference for gaze shifts between the second half of the skill acquisition stage and the skill refinement stage (p = 0.13; post hoc test), but there was a difference for cursor submovements (p < 0.05). Figure 8F illustrates the nonlinear coevolution of directional changes in gaze and cursor submovements by plotting the absolute direction of saccades with reference to the target against the corresponding data for cursor submovements. In the exploratory stage, the largest directional change occurred in saccades. Although most saccades were short (Fig. 6) and resulted in gaze fixations located near the cursor (Fig. 7D), they gradually became more biased toward the direction of the target. In the skill acquisition stage, saccades continued to become increasingly directed to the target and, toward the end of the early skill acquisition stage, cursor submovements started to become more target directed. This pattern is consistent with the observation that gazepursuing behavior diminished substantially in the early skill acquisition stage while the subjects still had a relatively poor control over the cursor.

Figure 8.

Direction relative to target and absolute direction (in screen coordinates) of saccades and cursor submovements. A, Data points indicate, as a function of practice time, direction relative to the target for the saccades generated by a single subject (Fig. 2 A, asterisk). Saccade direction (±180°) is provided in absolute values, where 0° indicates gaze shifts toward the target. The curve shows the corresponding data filtered with a moving median of ±100 s. B, Corresponding filtered curves obtained for the eight subjects shown in Figure 2 A (same format as in Fig. 2C). C, D, Distribution of the direction of saccades and cursor submovements relative to the target for all saccades and submovements obtained from all eight subjects. E, Superimposed thin lines plot absolute direction of cursor submovement relative to the target on the normalized time base for the same subjects after filtering with a moving median of ±100 s using the same format as in B. F, Absolute direction of saccades with reference to the target plotted against the corresponding direction of cursor submovements for the duration of the practice session. The curve represents the mean values obtained from the eight pairs of filtered curves representing saccade direction (B) and submovement direction (E) for each of the eight subjects. The dark and light gray zones represent the variability across subjects expressed as ±1 SEM and ±1 SD in the y-coordinate. G, H, Distribution of absolute directions (in screen coordinates) for all saccades and submovements obtained from all eight subjects. C, D, G, and H, Learning stages represented as in Figure 5C. Ex, Exploratory stage; SAa, first half of the skill acquisition stage; SAb, second half of the skill acquisition stage; SR, skill refinement stage.

The absolute directions of saccades and manual submovements (computed in screen coordinates) also changed with learning (Fig. 8G,H). In the exploratory stage, the distributions of both saccade and submovement directions exhibited four peaks at 0, 90, 180, and –90°. That is, there was a tendency to make saccades and submovements vertically and horizontally (Fig. 3A). Thus, subjects tended to produce submovements either by applying linear forces along the longitudinal axis of the tool or torques around this axis. In collateral experiments with 45 and –75° rotations of visual feedback on the screen, we found corresponding peaks at angular values that still matched longitudinal and twist forces. Thus, the tendency to generate movements vertically and horizontally appears to be related to motor output rather than to visual perception (cf. Flanagan and Rao, 1995). The multipeak topography of the distribution of directions of saccades and, to a lesser extent, cursor submovements attenuated with learning. To quantify this topography, we divided the directions of saccades and submovements (in screen coordinates) into 45° bins starting at –22.5° and determined the percentage of the total number of movements that fell within the four bins centered on 0, 90, ±180, and –90°. A two-way, repeated-measures ANOVA (effector by learning phase) revealed main effects of learning (F_(3,21) = 30.2; p < 0.0001) and effector (F_(1,7) = 10.9; p < 0.02). Overall, the topography was more pronounced for the submovements than for the gaze shifts. An interaction between effector and the learning stage was also observed (F_(3,21) = 6.0; p < 0.005); the percentage of movements vertically and horizontally declined substantially during the skill acquisition stage for gaze shifts but more gently for cursor submovements. The directional distribution of saccades in the second half of the skill acquisition stage and in the skill refinement stage did not differ significantly from a uniform distribution of saccades in all directions (p > 0.12 for both stages), whereas the direction of submovements did (p < 0.0001 for both stages; Watson's test) (Zar, 1999). Again, this difference between the saccade and cursor submovement directions may relate to the diminishing of gaze pursuing behavior in the early skill acquisition stage.

In summary, although the subject fixated the target now and then in the exploratory and early skill acquisition stages, saccades became more target directed, especially in the latter stage. They were almost exclusively directed to the target in the late skill acquisition and refinement stages. A similar change in directional distribution occurred for cursor submovements, which, however, was less pronounced and lagged that of the saccades.

Discussion

Eye–hand coordination has been examined previously in various types of tasks under conditions in which subjects have mastered the sensorimotor mappings required for skilled performance (Ballard et al., 1992; Inhoff and Wang, 1992; Vickers, 1992; Land and Lee, 1994; Kinsler and Carpenter, 1995; Epelboim et al., 1997; Land et al., 1999; Land and McLeod, 2000; Johansson et al., 2001; Flanagan and Johansson, 2003; Crawford et al., 2004). Here, we examined, for the first time, patterns of eye–hand coordination during learning of a complex visuomotor task in which subjects must acquire novel sensorimotor maps. We have demonstrated that markedly different patterns of gaze behavior are associated with different stages of learning. In the initial exploratory stage, when subjects generated frequent and poorly controlled cursor movements, saccadic gaze fixations typically pursued the moving cursor. That is, subjects did not anchor gaze at the spatial target of the current action phase as observed during natural and well learned manual tasks (Ballard et al., 1995; Epelboim et al., 1997; Land et al., 1999; Johansson et al., 2001). Although performance did not appreciably improve during this stage, subjects eventually gained a basic knowledge of the sensorimotor map signaling the onset of the skill acquisition stage. During this stage, manual performance improved rapidly, and subjects began to program spatially congruent eye and hand actions with gaze fixations marking future cursor goals. In the final skill refinement stage, performance continued to improve gradually, and gaze consistently shifted toward the current target, indicating an effective implementation of the new mapping relating motor commands and sensory outcomes. Although the learning generally occurred in these three stages, its time course varied across subjects, and it happened that subjects during one or both practice sessions failed to show an apparent exploratory stage and directly entered the skill acquisition stage. The reason for this is not clear. Transfer from previous experiences with certain visuomotor tasks and/or differences among subjects in learning capacity may have contributed.

At the start of each experimental session, subjects encountered a novel rule relating hand actions to cursor movements and therefore could not estimate the motor commands required to achieve the task. We presume that during the exploratory stage, subjects attempt to learn this novel mapping by actively exploring the degrees of freedom of the task in an attempt to discriminate between effective and ineffective strategies (Fitts and Posner, 1967; Brooks et al., 1995). Pursuing the cursor with gaze may assist in learning the mapping between manual actions and cursor movement in several possible ways. First, this would allow subjects to take advantage of foveal vision to monitor the movement of the cursor. Note that although the cursor typically had moved by the time gaze arrived, saccades during the exploratory stage generally served to bring gaze closer to the cursor. Although the location of targets for both eye and hand movements can be extracted effectively from peripheral vision (Biguer et al., 1982; Jeannerod, 1988; Land et al., 1999), it may be that foveal or parafoveal vision may be better for detecting changes in local movement direction (Paillard, 1996). If so, then subjects may direct gaze fixations near the cursor to accurately monitor cursor movements.

Second, by pursuing the cursor with gaze, subjects may attempt to bring gaze close to the cursor so that subsequent cursor and eye movements will have a common origin in gaze-centered coordinates. This may facilitate learning the mapping between eye and hand movements, a map that, once established, would effectively enable subjects to control the cursor. With this gaze strategy, most cursor movements will be directed away from the fovea. Populations of posterior parietal neurons sensitive to visual stimuli moving radially with respect to the fovea (Motter and Mountcastle, 1981; Caminiti et al., 1999) may be involved in coding these cursor motions.

Third, gaze shifts in pursuit of the cursor may facilitate learning basic correlations between eye motor commands and recent hand motor commands. Indeed, it has been proposed that gaze shifts in pursuit of the moving hand represent an important stage in learning visuomotor transformations in the context of visually guided reaching (Burnod et al., 1999). Although human adults rarely fixate the hand or hand-held objects in natural visually guided manual tasks (Land et al., 1999; Johansson et al., 2001), there is evidence that newborn human infants make an effort to view their hands (van der Meer et al., 1995; van der Meer, 1997). von Hofsten (2004) has suggested that by closing the visual–manual loop, the infant can begin to explore the relationship between commands and movements and between vision and proprioception and discover the possibilities and constraints of manual movements. Coactivation of neural processes that represent gaze and hand movements may establish associations that ultimately enable the brain to specify congruent gaze and cursor movements in parallel, which is characteristic of well learned visuomotor tasks (Prablanc et al., 1979; Biguer et al., 1982; Prablanc and Martin, 1992; Gribble et al., 2002). Likewise, establishing correlations between hand and eye motor commands would allow the specification of hand motor commands in gaze-centered coordinates (Andersen and Buneo, 2002; Engel et al., 2002; Crawford et al., 2004). In visually guided reaching, there is evidence that both targets and the hand are represented in gaze-centered frames of reference in various brain regions (Mushiake et al., 1997; Batista et al., 1999; Stuphorn et al., 2000; Buneo et al., 2002; Medendorp et al., 2003; Khan et al., 2005).

In the skill acquisition stage, performance suddenly began to improve rapidly, and gaze behavior gradually shifted from primarily pursuing the cursor to fixating future cursor goals. Thus, the parallel programming of spatially congruent eye and hand motor commands developed. We observed that as the skill acquisition stage progressed, gaze could be anchored at a position between the previous target (or cursor start point) and the target. This may represent a compromise between having parafoveal/foveal vision near the cursor for the majority of the movement and fixating a useful goal so that retinal and extra-retinal cues can be used to guide the cursor in the direction of the target (Prablanc et al., 1979; Bock, 1993). Alternatively, the location of these intermediate fixations may represent a more manageable preliminary goal for the cursor, which often also went to these locations. This eye–hand behavior may reflect a process in which subjects attempt to extend what they are learning about the sensorimotor map to longer and longer movements. Notably, most changes in gaze strategy occurred in the first half of the skill acquisition stage.

In the late skill acquisition and skill refinement stage, subjects showed a rather stable gaze strategy, making one or two saccades that brought the gaze to the target. Nevertheless, substantial improvements in manual skill occurred in these stages, especially in the late skill acquisition stage. This suggests that peripheral vision provides sufficient information to refine hand-movement control in these late stages of skill learning. Mappings that can support useful on-line corrections of hand trajectory based on peripheral vision (Desmurget and Grafton, 2000) might develop and help to eliminate errors gradually (inappropriate hand coordination patterns) and thereby contribute to the learning. In the skill refinement stage, performance continued to improve, but at a slower rate than in the skill acquisition stage. In collateral experiments, in which subjects practiced for 20 min per day for 5 d, we found that performance continued to improve throughout, although the rate of improvement slowed down with practice (R. S. Johansson, unpublished observations).

In summary, we have shown that gaze behavior in relation to hand movements changes dramatically across different stages of learning of a complex visuomotor task. It has been argued that visually guided manual tasks are governed by action schema that not only incorporate sensorimotor routines for manual action but also accompanying routines specifying task-specific eye movements that support the planning and control of manual action (Land and Furneaux, 1997; Flanagan and Johansson, 2003). Within this framework, our results indicate that learning a novel and complex manual task requires a very different action schema than does performing the task after it has been mastered. In particular, the patterns of eye movements supporting manual action can change markedly depending on the stage of learning.