Eye-Hand Coordination During Target Selection in a Pop-Out Visual Search

Abstract

We examined the coordination of saccades and reaches in a visual search task in which monkeys were rewarded for reaching to an odd-colored target among distractors. Eye movements were unconstrained, and monkeys typically made one or more saccades before initiating a reach. Target selection for reaching and saccades was highly correlated with the hand and eyes landing near the same final stimulus both for correct reaches to the target and for incorrect reaches to a distractor. Incorrect reaches showed a bias in target selection: they were directed to the distractor in the same hemifield as the target more often than to other distractors. A similar bias was seen in target selection for the initial saccade in correct reaching trials with multiple saccades. We also examined the temporal coupling of saccades and reaches. In trials with a single saccade, a reaching movement was made after a fairly stereotyped delay. In multiple-saccade trials, a reach to the target could be initiated near or even before the onset of the final target-directed saccade. In these trials, the initial trajectory of the reach was often directed toward the fixated distractor before veering toward the target around the time of the final saccade. In virtually all cases, the eyes arrived at the target before the hand, and remained fixated until reach completion. Overall, these results are consistent with flexible temporal coupling of saccade and reach initiation, but fairly tight coupling of target selection for the two types of action.

INTRODUCTION

In real-world situations, the target of a reaching movement must often be selected from distractors. Furthermore, reaching movements are seldom executed in isolation; instead, they are usually coordinated with saccades. We used a visual search task to examine the extent to which target selection is coupled for saccades and reaches and to investigate temporal eye-hand coordination.

In humans, eye-hand coordination has been examined extensively in tasks in which movements are made to single or sequentially presented targets. These studies have typically found that subjects look where they are going to reach and that saccades to irrelevant areas of the scene are rarely observed (e.g., Fisk and Goodale 1985; Frens and Erkelens 1991; Gielen et al. 1984; Helsen et al. 2000; Prablanc et al. 1979; Sailer et al. 2000). Such behavior is also observed in more natural tasks. For instance, Land, Mennie, and Rusted (Land et al. 1999) demonstrated that while making a cup of tea, participants consistently make saccades toward relevant objects in the scene (e.g., kettle and lid) to guide reach movements. There are very few fixations to nonessential loci. Similar close coupling between eye and hand movements has been also reported in other routine daily activities such as making sandwiches, playing cricket, walking, hand-washing, and driving (Hayhoe et al. 2003; Land and Hayhoe 2001; Land and Lee 1994; Land and McLeod 2000; Patla and Vickers 2003).

Task-specific strategic eye and hand coordination has been also observed in a task in which participants are required to copy a colored pattern of building blocks as fast as possible (Ballard et al. 1992, 1995; Mennie et al. 2007; Pelz et al. 2001). While most fixations are related to the immediate action, a small number of fixations, so-called look-ahead fixations, are made to objects relevant only to future actions. Also participants sometimes maintained eye-hand coordination by delaying the hand movements until the eye is available for guiding the movement. These various patterns of eye-hand coordination suggest a synergistic linkage between hand and eye movements.

Studies of eye-hand coordination have found that often, after a saccade, the eyes remain at the target location until the hand arrives when there is no subsequent reach target (e.g., Johansson et al. 2001; Neggers and Bekkering 2000, 2002; Pelz and Canosa 2001). For instance, Neggers and Bekkering (2000, 2002) demonstrated that when participants were required to reach to the initial target and make a saccade to a newly appearing target during reaching, gaze was locked onto the initial target until the hand arrived.

The extent to which saccade and reach target selection is spatially coupled has been studied to a lesser extent. Some studies have demonstrated highly correlated pointing and gaze errors, suggesting a shared target selection mechanism (Gielen et al. 1984; Soechting et al. 2001). Other studies have shown that directional and variable errors for eye and hand movements are poorly correlated in favor of independent selection mechanisms (Prablanc et al. 1979; Sailer et al. 2000, 2002b). For instance, Sailer and colleagues (2002a,b) demonstrated that the end positions of eye and hand movements are commonly drawn toward a distractor presented near the target, i.e., spatial averaging, suggesting a coupled target selection process for both effectors. Yet they also found that under some conditions, the amplitude and direction of the spatial averaging effect are different for the eye and hand, suggesting that the eye and hand use separate target representations that exchange information.

Eye-hand coordination has been less well studied in monkeys. In many reaching studies, eye movements are either not measured or are not a focus of the study (e.g., Caminiti et al. 1991; Cisek et al. 2003; Crammond and Kalaska 2000; Crutcher and Alexander 1990; Fu et al. 1993; Georgopoulos 1991; Kalaska et al. 1989, 1997; Moran and Schwartz 1999). Scherberger et al. (2003) studied target selection for reaches and saccades using a choice paradigm in which monkeys either reached while remaining fixated or made a saccade without reaching. They found that target selection for reaches and saccades shares a similar reference frame, suggesting that the two effectors rely on a common representation for target selection.

Our study differs from theirs in that we investigated the characteristics of coordinated eye and hand movements rather than testing each effector in isolation. Specifically, we used a pop-out visual search task similar to those that have been used extensively to study saccade target selection (e.g., Bichot and Schall 1999, 2002; Kim and Basso 2008; McPeek and Keller 2001, 2002, 2004; McPeek et al. 1999; Schall and Hanes 1993; Thomas and Pare 2007; Thompson et al. 1996), and more recently, reach target selection (Song and Nakayama 2006, 2007a,b, 2008; Song et al. 2008). Monkeys were rewarded for reaching to an odd-colored target presented with distractors. Eye movements were unconstrained and reward was not contingent on any particular eye-movement behavior.

METHODS

Two male rhesus monkeys (Macaca mulatta ) weighing 6 and 11 kg were used in this study. All experimental protocols were approved by the Institutional Animal Care and Use Committee at the Smith-Kettlewell Eye Research Institute and complied with the guidelines of the Public Health Service policy on Humane Care and Use of Laboratory Animals.

Apparatus

Testing was performed in a dimly illuminated room. Data collection and storage were controlled by Power Mac G4, which also generated the visual displays using software constructed with the PsychToolbox library (Brainard 1997; Pelli 1997). Visual stimuli were presented on a 17-in color CRT touch-sensitive monitor (ELO touch systems), positioned 25.5 cm in front of the monkeys. The monitor had a spatial resolution of 800 × 600 pixels and a refresh rate of 75 Hz. Eye position was sampled at 1 kHz using an EyeLink 1000 high-speed video tracker (SR Research).

Stimuli and behavioral procedure

Each animal was seated in a primate chair with its head restrained and its left arm loosely restrained during testing. Monkeys were trained to reach to a target for liquid reward and were allowed to work to satiation. At the beginning of each trial, a yellow square appeared in the center of the monitor. The square subtended 1–2° and had a luminance of 1.6 cd /m² against a black homogeneous background of 0.2 cd /m². The monkeys touched the square with their right hand and were required to maintain contact with it for 500 ms. After this interval, the square was extinguished and an odd-colored target was presented with three distractors.

The target and distractor stimuli consisted of red or green discs subtending 2.5° of visual angle (1.1 cm in diameter on the screen) with a luminance of 1.2 cd /m². The color of the target was randomly selected in each trial to be either red or green, and the distractors were of the opposite color. The stimuli were presented at an eccentricity of 12° (5.4 cm from the center), separated by angles of 90°. The position of the target was randomly chosen from the four oblique directions (45, 135, 225, and 315°). The display remained onscreen until a response was made, and monkeys were rewarded if the first point at which the hand touched the screen (after lifting off from the center point) was within 5° of the correct target location and remained at this position for 300–500 ms. Monkey J used the tip of the right index finger (D2), and monkey H used the tip of the right middle finger (D3) to touch the screen. Eye movements were not constrained during either the training or testing sessions and did not have any effect on the monkeys' reward for reaching to the target. Following several weeks of training, we collected 7,908 trials (monkey H: 5,236 trials; monkey J: 2,672 trials). In a later set of sessions, a small reflective hemisphere was taped to the right hand, ∼1 cm from the tip of the index finger of monkey J and the middle finger of monkey H to measure reaching movement trajectories. The reflective hemisphere was illuminated by infrared light and was optically tracked at a rate of 60 Hz using a Northern Digital Polaris tracker. In these sessions, monkey H performed an additional 1,997 trials and monkey J performed an additional 1.155 trials. Because these data were collected several weeks after the main dataset, they are analyzed separately in the last section of results.

Data analysis

Off-line data analysis was conducted on both eye and hand movements. An algorithm using velocity criteria detected the beginning and end of saccades, and the algorithm's identification of saccades was inspected to verify its accuracy. Small corrective saccades (<3°) in the same direction of the previous saccade were excluded for the purposes of the analysis. Saccades that ended within 5° of a stimulus (either target or distractor) were classified as being directed to that stimulus. Saccadic target acquisition time was defined as the interval between target onset and fixation of the target.

In the analysis of reaching movements, movement onset was measured as the time at which the hand was lifted from its initial position in the center of the touchscreen, and reach initial latency was defined as the interval between the onsets of the target and the movement. Reach endpoint was measured at the first location contacted on the touchscreen after movement onset. A trial was classified as a target selection error when the reach endpoint was >5° away from the target. Reach target acquisition time was defined as the interval between target onset and hand contact of the touchscreen within 5° of the target.

When reaching trajectories were tracked, movement velocity was calculated from the three-dimensional (3D) position traces after filtering with a low-pass filter (cutoff frequency of 25 Hz). The beginning and end of reaching movements were detected using a velocity criterion (8–10 cm /s). The algorithm's identification of movements was inspected to verify its accuracy.

The curvature of movement trajectories was quantified by calculating the maximum curvature of the movement in the two-dimensional (2D) plane of the video monitor. Maximum curvature was defined as the maximum perpendicular distance from a straight-line path between the start and endpoints of a movement trajectory divided by the length of the straight-line path. We also calculated the radial direction of movements at each point during the movement duration. We defined radial direction as the direction of the vector from the initial starting position to the current hand position in the plane of the monitor (Song et al. 2008). Statistical comparisons were conducted by Wilcoxon tests unless otherwise stated. Error bars in the figures and numerical error ranges given in the text indicate SE.

RESULTS

Overall, both monkeys performed the task well. Figure 1 shows 2D plots of reach endpoints for each monkey, with the four target locations superimposed. Monkey H reached to the correct target in 90.6% of trials (4,744/5,236 trials) and monkey J in 93.7% (2,504/2,672 trials). The other 9.4 and 6.3% of trials were classified as reach target selection errors (see methods). Overall, there was no difference in error rate for the four different target locations (χ² test: P = 0.16 for monkey H and P = 0.12 for monkey J). In the following sections, we will confine our analyses to correct reaching trials unless explicitly indicated otherwise.

Fig. 1.

Spatial distribution of reach endpoints in the visual search task for the 2 monkeys. The 4 target locations are marked by white (left) or black (right) circles. The 30 × 30° space has been divided into 0.5 × 0.5° bins, with the darkness of each bin indicating the number of reach endpoints in that bin.

Saccade endpoints in the reach target selection task

In the vast majority of correct trials, monkeys fixated the target location even though they had never been trained to do so. In roughly half of correct reach trials (54% in monkey H and 40% in monkey J), a single saccade was made from fixation to within 5° of the target location. In >90% of correct reach trials, the target was foveated within two saccades, and it was virtually always foveated within three saccades. Figure 2A shows the cumulative probability of target foveation as the number of saccades increases, which was greater than would be predicted by chance. Across all correct reach trials, monkey H executed an average of 1.6 ± 0.009 saccades, and monkey J executed 1.9 ± 0.014 saccades per trial. Once monkeys made a saccade to the target, fixation was maintained until the reach to the target was complete with the exception of ∼2–3% of trials in which target fixation was broken while the hand was en route to the target.

Fig. 2.

A: cumulative probability that the odd-colored target was foveated is plotted for each consecutive saccade during the trial. B: diagram illustrating the relative positions of the 4 stimulus locations. C: spatial distribution of 1st saccade endpoints in the visual search task for the 2 monkeys. Endpoints have been normalized by a rotation and/or a reflection such that the target location is always to the right of fixation, the ipsi location is above fixation, the contra location is below fixation, and the opposite location is to the left of fixation.

Ipsilateral distractor bias for saccades in correct reach trials

Saccades that were not directed to the target typically landed near a distractor. Overall, saccades that landed >5° away from either the target or a distractor occurred in only 2% of trials for monkey H and 1% of trials for monkey J. Such trials were discarded, due to their scarcity, and for the remaining trials, the intended goal of each saccade was taken to be the nearest stimulus (target or distractor) within a 5° radius.

Figure 2C shows plots of the endpoints of the first saccade made in correct reaching trials for the two monkeys. Here the endpoints have been normalized by a rotation and/or reflection, such that saccades directed to the target are plotted as 0° in direction (rightward from fixation), whereas saccades to the distractors are plotted as 90, 180, or −90° in direction. As shown in Fig. 2B, the 90° location corresponds to the distractor in the same hemifield as the target (“ipsi” location), the 180° location corresponds to the distractor diametrically opposite the target (“opposite” location), and the −90° location corresponds to the distractor that is 90° away from the target in the other hemifield (“contra” location). As is evident from the figure, many initial saccades are made to the target location. Interestingly, when the first saccade is not directed to the target, it appears to go more frequently to the ipsilateral distractor than to the other distractors. This indicates that even when the correct stimulus is ultimately chosen as the goal for the reach, initial saccades which are not directed to the target tend to go to the ipsilateral distractor rather than other distractors.

To examine this ipsilateral saccade bias in more detail, in Fig. 3 (left), we plotted, for each monkey, a histogram of the normalized direction of initial saccades that were directed toward distractors for trials in which a reach was made to the target. From these plots, it is clear that initial saccades to distractors are not randomly distributed. Instead they are directed significantly more often to the ipsilateral distractor (90° in direction) than to any other distractor (χ² test: P < 0.0001 for both monkeys). The ipsilateral saccade bias is consistently observed for each target location as shown in Fig. 3, right (P < 0.0005 for each target location in each monkey). This result indicates that distractors in the same hemifield as the target are more attractive for saccades than opposite-hemifield distractors (e.g., McPeek 2006; McPeek et al. 2003).

Fig. 3.

Histograms of initial saccade endpoint direction in 2-saccade trials with a correct reach to the target collapsed across all target locations (left) and for individual target locations (right). Left: endpoints have been normalized by a rotation and a reflection such that saccades directed to the target location are in a direction of 0°.

We also found that both monkeys tended to make fewer initial error saccades to the opposite distractor furthest from the target (180° direction) than to the distractor adjacent to the target in the contralateral hemifield (−90° direction), as confirmed with binomial tests (P < 0.0001 for both monkeys). This is consistent with previous studies suggesting that reach and saccade target selection develops over time in a spatially coarse-to-fine progression (Bichot and Schall 1999; Findlay 1997; Gilchrist et al. 1999; McPeek and Keller 2001; Song et al. 2008).

Coupling of eye and hand endpoints in reach target selection error trials

To further examine the extent to which target selection for saccades and reaches is spatially coordinated, we also analyzed trials in which reach target selection errors were made. We found that in error trials in which a single saccade was executed (34 and 65% of reach error trials in monkeys H and J, respectively), the saccade was indeed always directed to the same distractor as the reach. Even when multiple saccades were executed in a trial, overall, in both monkeys, in 95% of reach error trials the hand was directed to the fixated distractor by the last saccade. In the remaining error trials, the eyes fixated the distractor, but after the onset of the reaching movement to that distractor, a saccade was made to another stimulus location. Thus overall, the error trials are consistent with a tight linkage between saccade and reach target selection.

Ipsilateral bias for reach target selection errors

Figure 4 shows histograms of reach endpoint direction in error trials. The endpoints have been normalized by a rotation and/or reflection similar to the saccades in Fig. 3 (left), such that reaches directed to the target location would be plotted in a direction of 0°. Clearly, more reach target selection errors are directed to the ipsilateral distractor (90° in direction) than to the contralateral (−90°) or opposite (180°) distractors in both monkeys (χ² test: Ps < 0.0001 for both). The ipsilateral bias for reach target selection errors is consistently observed for each target location as shown in Fig. 4, right (P < 0.002 for all target locations in both monkeys).

Fig. 4.

Histograms of normalized reach endpoint direction in reach target selection error trials collapsed across all target locations (left) and for individual target locations (right). Reach endpoints have been normalized in the same manner as the saccade endpoints in Fig. 3.

Thus when an incorrect reach is made to a distractor, there is a clear bias toward selecting the ipsilateral distractor. As we showed earlier, when a correct reach to the target is accompanied by more than one saccade, the initial saccade is biased toward the ipsilateral distractor (Fig. 3). Thus comparing across these two different situations, we find similar target selection biases for both reaching and saccades. This common bias suggests that the two effectors share common target selection mechanisms.

Temporal eye-hand coordination

In addition to investigating the spatial aspects of eye-hand coordination, we also examined the temporal sequencing of saccades and reaches in visual search. Specifically, we compared correct reach trials in which a single saccade to the target is followed by a reach to the target (1-saccade trials) with trials in which an initial saccade to a distractor is followed by a second saccade and a reach to the target (2-saccade trials). The comparison of these two different trial types allows us to determine if there are regularities in the timing of the final target-directed saccade and reach regardless of whether an initial saccade to a distractor was made. Together, one- and two-saccade trials comprised 96 and 91% of correct reach trials in monkeys H and J, respectively. A comparison of the latencies of the initial saccades in one- and two-saccade trials (Fig. 5) shows that on average, initial saccades in two-saccade trials have slightly shorter latencies (P < 0.0001 for both monkeys). This suggests that initial saccades in two-saccade trials are directed to a distractor rather than to the target because they have been executed prematurely before the target selection process has been fully completed.

Fig. 5.

For both monkeys, the mean latency of the 1st saccade is shorter when the saccade is directed to a distractor than when it is directed to the target. Error bars in this and subsequent figures indicate ±1 SE.

Relationship between target-directed saccade latency and reach latency

First, we examined the relationship between the initiation of a saccade to the target and the initiation of the reach to determine if the two actions show rigid temporal coupling in this task. A fixed delay between saccade and reach initiation would suggest that the two actions share a common trigger signal or that the initiation of the saccade triggers the reach. In one-saccade trials, a saccade is made directly to the target, whereas in two-saccade trials, an initial saccade to a distractor is followed by a second saccade to the target. Note that in all of these trials, reach target selection was correct. As might be expected, it takes significantly longer for the eyes to foveate the target in two-saccade trials than in one-saccade trials (P < 0.0001 for both monkeys). Reaching latencies were also significantly longer in 2-saccade than 1-saccade trials (P < 0.002 for both monkeys). However, the increase in reach latency was much smaller than the increase in saccade latency. Figure 6A compares the mean latencies of target-directed saccades and reaches in one- and two-saccade trials for both monkeys. Overall, saccades to the target in two-saccade trials occurred on average 117 ms later in monkey H and 153 ms later in monkey J, while reaches were initiated only 18 ms later in monkey H and 5 ms later in monkey J. This disparity suggests that planning of the reach movement begins even before the onset of the second saccade to the target in two-saccade trials.

Fig. 6.

A: comparison of the mean latency for each monkey of saccades and reaches in 1- and 2-saccade trials. B: histograms show, for each monkey, the distribution of the interval between reach and saccade onset toward the target in 1-saccade (black) and 2-saccade (gray) trials. Positive numbers on the abscissa indicate trials in which the saccade to the target is initiated before the reach, and negative numbers indicate the opposite order.

To examine the extent to which saccades to the target preceded reaches on a trial-by-trial basis, for each trial, we calculated the interval between the onset of the target-directed saccade and of the reach to the target. In Fig. 6B, positive numbers on the abscissa indicate saccades that were initiated earlier than reaches, while negative numbers indicate reaches that were initiated before the target-directed saccade. The histograms show clear differences in the relative timing of saccade and reach initiation in one- and two-saccade trials.

In one-saccade trials (black), monkeys always initiated a saccade to the target before the reach, on average by 235 ± 1.1 ms in monkey H and 139 ± 1.2 ms in monkey J (P < 0.0001 for both). In contrast, in two-saccade trials (gray), both monkeys sometimes initiated a reach before or near the onset of the saccade to the target. Therefore the differences between saccade and reach onsets were reduced or even eliminated in two-saccade trials compared with one-saccade trials: in monkey H, saccades were executed 139 ± 1.5 ms earlier than reaches (P < 0.0001), whereas in monkey J, in many trials, the reach was initiated before the saccade to the target, while the eyes were still fixated on a distractor. Indeed, in monkey J, the mean difference between reach and saccade initiation in two-saccade trials was 2.22 ± 1.7 ms, which was not a statistically significant difference (P = 0.317).

The results from the one-saccade trials suggest a relatively fixed relationship between saccade and reach onset: a saccade to the target was always initiated well before the reach. On the other hand, two-saccade trials show that a reach can begin near or even before the onset of a saccade to the target, suggesting a more flexible timing relationship between initiation of the targeting saccade and the reach (e.g., Abrams et al. 1990). In a later section, we will show that when reaches are executed earlier than the target-directed saccades, they exhibit systematic trajectory deviations toward the initial saccade endpoint, consistent with a close linkage between saccade and reach target selection.

Temporal relationship between saccadic and reaching acquisition of the target

In one-saccade trials, saccades were initiated well before reaches, and subsequently, the eyes virtually always remained fixated at the target location until reach completion. In contrast, in two-saccade trials, reaches were often initiated near the onset time of target-directed saccades, or even earlier. However, saccades have much shorter movement durations than reaches, and so even in two-saccade trials, the eyes always reached the target before the hand.

To visualize the temporal relationship between saccadic and reaching acquisition of the target, we plotted the mean times of target fixation and hand contact of the target in one- and two-saccade trials in Fig. 7A. Overall, both saccades and reaches landed on the target later in two-saccade trials than in one-saccade trials (P < 0.0001 for both monkeys). On average, the eyes waited for the hand in one-saccade trials for 344 ± 1.6 ms in monkey H, and 254 ± 2.0 ms in monkey J. In two-saccade trials, this waiting time was significantly shorter than in one-saccade trials (both monkeys P < 0.0001): 269 ± 1.8 ms in monkey H and 177 ± 1.78 ms in monkey J. Multivariate analysis confirmed this interaction effect, indicating that the additional corrective saccade in two-saccade trials leads to a longer delay in saccade target acquisition than in reach target acquisition (P < 0.0001 for both monkeys).

Fig. 7.

A: comparison of the time required for the eyes to foveate the target or the hand to contact the target in 1- and 2-saccade trials for each monkey. B: histograms show, for each monkey, the difference between saccade and reach target acquisition times in 1-saccade (black) and 2-saccade (gray) trials.

To examine the temporal relationship between saccadic and reaching acquisition of the target in more detail, we plotted trial-by-trial histograms of the time interval between target fixation and hand contact of the target in Fig. 7B. The shortened delay between saccadic and reaching acquisition of the target for two-saccade trials compared with one-saccade trials is clearly visible. However, we also noted that in two-saccade trials, the eyes always fixated the target before the hand arrived even when reaches to the target were initiated before saccade onset. The eyes then remained at the location of the reach goal until the completion of the reaching movement, as previously reported in humans (Bekkering and Neggers 2002; Johansson et al. 2001; Neggers and Bekkering 2000).

Reach trajectory deviations and spatiotemporal coupling of target selection

Previous studies have demonstrated that reach trajectories are sensitive to competition between a target and distractors (Song and Nakayama 2006, 2007b, 2008; Song et al. 2008; Tipper et al. 1998; Welsh and Elliott 2004). For example, Song and colleagues found that reach trajectories often swerve toward a competing distractor, and the frequency and magnitude of curved trajectories increases as competition between the target and distractors increases (Song and Nakayama 2006, 2007b, 2008; Song et al. 2008).

Earlier, we showed that the final saccade in each trial is directed to the reach target location, suggesting that target selection for the two effectors is coupled. If this is true, then we would expect to see differences in the kinematics of reaches in two-saccade trials, where an initial saccade is made to a distractor, when compared with reaches in one-saccade trials, where a saccade is made directly to the target. In particular, we would predict that the trajectory of the reaching movement will deviate toward the initial saccade goal when the reach is initiated before the second saccade to the target in two-saccade trials.

However, in the current dataset, we did not record reach trajectories. Therefore we collected an additional dataset, under identical stimulus conditions, which included recordings of reach movement trajectories (see methods), a few weeks after the first dataset was collected. In this second dataset, we again found that in two-saccade trials, the initial saccade was more frequently directed to the ipsilateral distractor than to any other distractor (χ² tests: P < 0.0001, for both monkeys). Because these “ipsi 2-saccade” trials formed the majority of our two-saccade trials, we focused on them for our subsequent analyses.

Figure 8, A and B, shows reach trajectories in one-saccade trials and ipsi two-saccade trials for monkey H. Each trajectory is color-coded according to the location of the target. In one-saccade trials (Fig. 8A), trajectories toward each of the four target locations are relatively straight. In contrast, many curved trajectories are observed in the ipsi 2-saccade trials (Fig. 8B). In these trials, the early portion of the curved trajectory was often directed toward the ipsilateral distractor, where the initial saccade was directed, and was corrected in mid-flight toward the target, where the final saccade landed. Monkey J showed similar trajectory patterns. To quantify the curvature of the movement trajectories, we computed the maximum curvature (see methods) toward the ipsilateral distractor and found that reach trajectories in ipsi 2-saccade trials were significantly curved toward the ipsilateral distractor (P < 0.0001 for both monkeys).

Fig. 8.

A and B: examples of reach trajectories in 1-saccade (A) and ipsi 2-saccade trials (B) for monkey H. These trajectories are tracked in 3 dimensions, but, for clarity, only the components in the plane of the touchscreen monitor are shown. Trajectories associated with each target location are plotted in a different color: green (45°), red (135°), blue (225°), and black (315°). The scale bar indicates 1 cm. C and D: plots of radial direction vs. time during the movement for monkey J in ipsi 2-saccade trials. Trials were categorized according to whether the saccade to the target began before reach onset (C) or after reach onset (D). In all cases, the eyes arrived at the target before completion of the reaching movement.

To more closely investigate the influence of the initial saccade on reach target selection, we separated ipsi 2-saccade trials based on whether the saccade from the distractor to the target occurred before or after reach onset. If target selection for saccades and reaches is linked, then we would expect that when the reach is executed after the onset of the second saccade, the reach trajectory to the target should be relatively normal. On the other hand, when the reach is executed before the second saccade, while the eyes are still fixated on a distractor, the reach trajectory should be deviated toward the fixated distractor.

To test this possibility, we computed radial direction throughout the course of each reaching movement (see methods). Figure 8, C and D, shows plots of radial direction versus time during reaching movements made by monkey J in ipsi 2-saccade trials. Here, radial direction has been normalized across target positions, such that a direction of 0° corresponds to the target direction, 90° corresponds to the ipsilateral distractor, −90° to the contralateral distractor, and 180° to the opposite distractor as in Fig. 2. Figure 8C shows a temporal plot of radial direction for reaches when the target-directed saccade was executed earlier than the reach, whereas Fig. 8D shows trials when the final target-directed saccade was executed later than the reach.

We found that when the saccade was initiated before the reach (Fig. 8C), radial direction converges near the onset of the movement, such that the reach is directed toward the target location (0°). In contrast, when the reach was executed earlier than the target-directed saccade, the radial directions of many movements show much more extensive curvature toward the ipsilateral distractor location (90°), indicated by changes in radial direction occurring later in the movements (Fig. 8D).

To summarize and quantify these results, we calculated the radial direction 150 ms after reach movement initiation in both monkeys in ipsi 2-saccade trials (Jax and Rosenbaum 2007). In Fig. 9, we plotted these values as a function of the interval between the onset of the target-directed saccade and the reach. Figure 9 shows that when the target-directed saccade was executed before the reach (open squares), radial direction during this time was toward the target, whereas when the reach was executed earlier than the saccade (filled circles), radial direction tended to shift toward the ipsilateral distractor (90°), where the eyes were fixated at the moment of reach onset. The difference in radial direction between these groups was confirmed by Kruskal-Walllis tests (P < 0.0001 for both monkeys). These results demonstrate that even though the exact temporal relationship between saccade and reach onset is flexible, target selection for reaches and saccades is fairly tightly linked both temporally and spatially.

Fig. 9.

Plots of normalized radial reach direction 150 ms after movement initiation in the ipsi 2-saccade trials as a function of the interval between the onset of the target-directed saccade and the reach, shown for each monkey. Negative numbers on the abscissa refer to target-directed saccades that were initiated before the reach (open square), while positive numbers refer to saccades that were initiated after reach initiation (filled circle).

DISCUSSION

To the best of our knowledge, this study is the first to investigate spatiotemporal eye-hand coordination during a reaction-time visual search task. The aim of the study was to examine the extent to which target selection is coupled for saccades and reaches and to investigate temporal eye-hand coordination in a visual search task similar to those used to study saccade target selection.

Coupling of target selection for saccades and reaches

We found that target selection for saccades and reaches in this task was tightly coupled. Although eye movements were unconstrained and the animal's reward was not contingent on eye-movement behavior, the final saccade in each trial was virtually always directed to the goal of the reaching movement. Even in error trials, the saccade and reach were directed to the same stimulus.

Saccades and reaches also showed similar target selection biases. In some correct reach trials, an initial saccade was made to a distractor, followed by a second saccade and reach to the target. In these trials, the first saccade was directed more often to the distractor located in the same hemifield as the target than to any of the other distractors, indicating an ipsilateral bias in target selection for saccades. When we examined trials in which a reach was made to a distractor, we found that most of the reaching errors were also directed to the ipsilateral distractor.

Overall, these results point to a close linkage between reach and saccade target selection, which is in accord with previous studies in humans (e.g., Frens and Erkelens 1991; Gielen et al. 1984; Horstmann and Hoffmann 2005; Johansson et al. 2001; Land and Hayhoe 2001; Land and McLeod 2000; Neggers and Bekkering 2000, 2002; Scherberger et al. 2003; Soechting et al. 2001). However, this does not necessarily imply that saccades and reaches share a common target selection mechanism; two separate but parallel target selection processes could exist (Sailer et al. 2002b). Indeed, it is difficult to distinguish shared from separate target selection mechanisms using behavioral methods.

Temporal relationship between saccades and reaches

We examined the temporal relationship between saccades and reaches by comparing the timing of these actions in one-saccade trials in which a single saccade to the target was followed by a reach and two-saccade trials in which a saccade to a distractor was followed by a second saccade and a reach to the target. In one-saccade trials, reaches consistently began after initiation of the saccade to the target. However, in two-saccade trials, the time between reach onset and the onset of the target-directed saccade was often much shorter, and in some cases, the reach began even before the saccade, indicating flexible temporal coupling of saccade and reach initiation.

When we examined the time of target acquisition for saccades and reaches, we found that for both one- and two-saccade trials, the eyes always arrived at the target before the hand. Furthermore, in both trial types, the eyes virtually always remained fixated at the target location until the reaching movement was completed, consistent with studies in humans (Bekkering et al. 1995; Johansson et al. 2001; Neggers and Bekkering 2000, 2002).

On average, the amount of time that the eyes were required to wait for the hand was significantly shorter in the two-saccade trials than in the one-saccade trials due to the fact that reaches in the two-saccade trials were initiated closer to saccade onset. This suggests that in these trials, some portion of the reach planning began even before the second saccade to the target. As a result, reaches to the target were only slightly delayed when multiple saccades were made compared with trials with a single saccade. This demonstrates that flexible temporal coordination between the eyes and hand can increase the efficiency of goal-directed actions.

Spatiotemporal relationship between saccade initiation and reach trajectory

In two-saccade trials, reaches to the target were sometimes initiated near or even before the onset of the final saccade to the target. When the reach was initiated after the final saccade, the trajectory of the reaching movement was fairly straight toward the target. On the other hand, when the reach was initiated before the saccade, we often observed deviations in the reach trajectory toward the fixated distractor, further supporting the idea that target selection for reaches and saccades is linked. We hypothesize that a change in the selected target occurring during the reach causes the reach trajectory to change course toward the newly selected target, while also triggering a saccade to the same stimulus.

Within hemifield distractor-selection bias

The bias toward selecting a distractor in the same hemifield as the target is consistent with studies of saccade target selection that have shown greater competition between within-hemifield stimuli than across-hemifield stimuli (e.g., McPeek 2006, 2008; McPeek et al. 2003). It is worth noting that this bias is not simply a proximity effect: the contralateral distractor is located the same distance from the target in the opposite hemifield. Furthermore, it is not particularly related to the responding hand. For example, previous studies have reported increased reach latencies and errors to targets in the presence of distractors ipsilateral to the reaching hand versus distractors at other locations (Meegan and Tipper 1998; Tipper et al. 1998). Yet our result differs from theirs since we found a consistent ipsilateral bias regardless of the target's location, suggesting an object-based distractor frame of reference.

We speculate that the ipsilateral bias may be due to slower propagation of lateral interactions when information must cross the corpus callosum (Innocenti et al. 1986; Newsome and Allman 1980; Wilson et al. 2001). Indeed a recent study has shown that grouping of stimuli is more efficient within a hemifield than across hemifields (Butcher and Cavanagh 2008; Pillow and Rubin 2002). Thus we conjecture that the ipsilateral selection bias observed here occurred because the two homogeneous distractors in the contralateral hemifield might be perceptually grouped quickly, and thus can be excluded earlier in the target selection competition. As a result, if a movement is initiated before target selection is fully resolved, it is less likely to be directed to a contralateral distractor.

Neural substrates of target selection

The neural substrates of saccade target selection in visual search have been studied extensively, and include the superior colliculus (SC), frontal eye field (FEF), and lateral intraparietal area (LIP) (e.g., Basso and Wurtz 1998; Bichot and Schall 1999, 2002; Ipata et al. 2006; Kim and Basso 2008; McPeek and Keller 2002, 2004; Schall and Hanes 1993; Schiller and Tehovnik 2005; Thomas and Pare 2007; Thompson et al. 1996). Less is known about the neural mechanisms of target selection for visually guided reaching. Recent studies have suggested that higher-order movement-related areas in the frontal lobe including the dorsal premotor area (PMd) and in the posterior parietal cortex (PPC) including the parietal reach region (PRR) and area 5 are likely to be involved (Cisek 2006; Cisek and Kalaska 2002, 2005; Hoshi et al. 2000; Pesaran et al. 2008; Scherberger and Andersen 2007). For example, single-unit recording studies have demonstrated that the PMd can simultaneously encode the two competing movement goals when two potential targets are present, suggesting a role in target selection (Cisek and Kalaska 2002, 2005). Furthermore, PMd neurons encoded the relative position of the target, hand, and eye using the differences in locations among all three. This implies that PMd could be involved in coordinating hand and eye movements (Pesaran et al. 2006b).

In PRR, Scherberger and Andersen (2007) have shown that neural activity is linked to target choice when monkeys choose one of two sequentially presented targets. In addition to an immediate movement goal, the PRR also encodes subsequent movement goals, suggesting its role in more complex sequential target selection (Baldauf et al. 2008). Neural activity in PPC also reflects the choice of specific effector (eye or hand): neurons in the lateral intraparietal area (LIP) are selective for saccades, whereas those in the PRR are selective for reaches. This effector specificity suggests that the PPC is actively involved in action selection and movement preparation (Cui and Andersen 2007). Finally, Pesaran et al. (2008) examined how PMd and PRR are involved in the decision circuit to select a single reach target from multiple alternatives. They found evidence for information transfer between PMd and PRR, which may underlie the selection of a common reach goal by the two areas.

If target selection for reaches and saccades is based on common mechanisms, the neural underpinnings of this eye-hand interaction are still unknown. However, behavioral and neural studies have suggested that the saccadic and reaching systems interact extensively. For example, the peak velocity of saccades is typically considered an invariant function of the saccade amplitude. However, saccades show significantly higher peak velocities when they are accompanied by reaching movements to the same goal (Epelboim et al. 1997; Snyder et al. 2002), and the trajectories of saccades can be influenced by reaching movements (Tipper et al. 2001).

There is also considerable evidence for interactions between the saccadic and reaching systems at the neural level. In the SC, a structure traditionally viewed as oculomotor, a class of neurons has been identified that are selectively active during visually guided reaches (Werner et al. 1997). Some of these SC cells code reach goals in retinotopic coordinates (Stuphorn et al. 2000) as do reach-related neurons in PRR (Batista et al. 1999). In frontal cortex, the activity of reach-related PMd cells is modulated by eye position (e.g., Boussaoud et al. 1998; Pesaran et al. 2006a), and the activity of saccade-related neurons in FEF is modulated by hand position (Thura et al. 2008). Many supplementary eye field (SEF) neurons also show modulations in activity depending on whether a saccade is made alone or with a coordinated reach (e.g., Mushiake et al. 1996). Similarly, the activity of some neurons in parietal reaching areas is modulated by eye movements (e.g., Merchant et al. 2001; Snyder et al. 2000; Wurtz et al. 2001). Clearly, there are many possible neural substrates for cross-talk between the saccadic and reaching systems, and future work will clarify their potential roles in target selection and eye-hand coordination.

GRANTS

This study was supported by National Eye Institute Grant R01-EY-014885 to R. M. McPeek and Core Grant P30-EY-006883 and by a R. C. Atkinson Fellowship Award to J.-H. Song.