Evidence of common and separate eye and hand accumulators underlying flexible eye-hand coordination

Previous studies on eye-hand coordination have considered mainly the means of eye and hand reaction time (RT) distributions. Here, we leverage the approximately linear relationship between the mean and standard deviation of RT distributions, as predicted by the drift-diffusion model, to propose the existence of two distinct computational architectures underlying coordinated eye-hand movements. These architectures, for the first time, provide a computational basis for the flexible coupling between eye and hand movements.

Abstract

Eye and hand movements are initiated by anatomically separate regions in the brain, and yet these movements can be flexibly coupled and decoupled, depending on the need. The computational architecture that enables this flexible coupling of independent effectors is not understood. Here, we studied the computational architecture that enables flexible eye-hand coordination using a drift diffusion framework, which predicts that the variability of the reaction time (RT) distribution scales with its mean. We show that a common stochastic accumulator to threshold, followed by a noisy effector-dependent delay, explains eye-hand RT distributions and their correlation in a visual search task that required decision-making, while an interactive eye and hand accumulator model did not. In contrast, in an eye-hand dual task, an interactive model better predicted the observed correlations and RT distributions than a common accumulator model. Notably, these two models could only be distinguished on the basis of the variability and not the means of the predicted RT distributions. Additionally, signatures of separate initiation signals were also observed in a small fraction of trials in the visual search task, implying that these distinct computational architectures were not a manifestation of the task design per se. Taken together, our results suggest two unique computational architectures for eye-hand coordination, with task context biasing the brain toward instantiating one of the two architectures.

NEW & NOTEWORTHY Previous studies on eye-hand coordination have considered mainly the means of eye and hand reaction time (RT) distributions. Here, we leverage the approximately linear relationship between the mean and standard deviation of RT distributions, as predicted by the drift-diffusion model, to propose the existence of two distinct computational architectures underlying coordinated eye-hand movements. These architectures, for the first time, provide a computational basis for the flexible coupling between eye and hand movements.

although eye and hand movements are routinely made in isolation, for reaching and grasping behaviors they are typically coupled together. However, studies investigating the temporal coupling between eye-hand movements have reported heterogeneous results. Studies by Prablanc et al. (1979), Biguer et al. (1984), Sailer et al. (2000), and Dean et al. (2011) have reported low reaction time (RT) correlations (0.1–0.4), suggesting that the two movements are initiated by separate motor commands sharing common perceptual inputs (Dean et al. 2011; Gielen et al. 1984; Sailer et al. 2000). Other studies have observed moderate (∼0.6) to high (∼0.95) correlations (Fischer and Rogal 1986; Frens and Erkelens 1991; Gopal et al. 2015; Herman et al. 1981), suggesting that a common motor command is responsible for initiating the two effectors (Bizzi et al. 1971). While there is an implicit understanding that task context could drive the range of RT correlation observed (Sailer et al. 2000), neither has the context dependency of eye-hand coordination been explicitly tested, nor has the computational architecture that generates the observed flexibility been studied.

While most studies on eye-hand coordination have been phenomenological, two recent studies have used the drift diffusion framework, which explains movement initiation as an accumulation to a fixed threshold (Ratcliff 1978; Ratcliff et al. 2016; Ratcliff and Van Dongen 2011), to suggest two contrasting computational architectures underlying eye-hand coordination. Using a dual-task paradigm where RT correlation was low, Dean et al. (2011) proposed that the initiation of eye and hand movements are best modeled as two separate but interacting accumulators (separate accumulator model) for eye and hand effectors. Conversely, Gopal et al. (2015) used a saccade and reach task to propose that a common accumulator with a hand-specific peripheral delay (common accumulator model) was responsible for generating coordinated eye-hand movements. This model was based on the observations that the RT correlation was high, and that the standard deviation (SD) of eye and hand RT distributions were comparable, although their means were not. Such comparable SDs of eye and hand RT distributions violates the predictions of the drift diffusion model, where the SD scales approximately linearly with the mean (Wagenmakers et al. 2005; Wagenmakers and Brown 2007). Thus there are two contrasting predictions of the two models: the common accumulator model predicts high eye and hand RT correlations and similar SDs of eye and hand RT distributions, while the separate accumulator model predicts low eye and hand RT correlations and dissimilar SDs of eye and hand RT distributions.

In this study, we provide a computational framework that can explain the flexibility reported by studies on coordinated eye-hand movements. In addition, we attempted to provide insights into the nature of the accumulation process, which has lacked in the two studies mentioned before (Dean et al. 2011; Gopal et al. 2015). Despite using the diffusion model, neither of the studies clarified what the accumulation indicates, i.e., whether it represents a decision process where the sensory evidence is accumulated (Gold and Shadlen 2007; Reddi 2001) as there was no explicit decision-making involved in either of the two studies. As evidence accumulation during decision-making is considered to be a stochastic process, if some part of the accumulation process represents a decision process, it is expected that the SD of the RT distribution will scale along with the mean RT, depending on the task difficulty. Thus, in this study, we addressed two questions: 1) to evaluate whether the common accumulator model can be extended to tasks involving explicit decision-making, and elucidate if some aspect of the decision is accumulated in the diffusion process; and 2) using a contrast between the visual search and dual task, we delineate the two patterns of temporal coupling and provide a computational framework, which can explain the flexible coupling between the two effectors.

METHODS

Subjects

Eye and hand movements were recorded in 8 and 10 normal right-handed subjects in the dual task and visual search task, respectively (equal number of men and women in both the tasks). They were aged between 21 and 29 yr and had normal or corrected-to-normal vision. Subjects gave their written, informed consent, in accordance with the Institutional Human Ethics Committee of the Indian Institute of Science, Bangalore, which approved the protocol. Subjects were paid for their participation in the experiments.

Behavioral Tasks

Visual search task.

Each trial began with the presentation of a central fixation box where the subjects had to fixate their eyes and hand (Fig. 1A). After a fixation delay of 500-1,000 ms, an array of four stimuli was presented along the cardinal axis. Subjects were instructed to make a movement to the odd target among three similar distractors. The difficulty of the search was modulated using colors. In the easy search condition, the subjects had to discriminate between red [Commission International de l'Eclairage (CIE) coordinates: x = 0.473; y = 0.321; luminance: 25 cd/m²] and green (CIE coordinates: x = 0.299; y = 0.494; luminance: 26 cd/m²) colors, while in the difficult block they had to discriminate between green and blue-green (CIE coordinates: x = 0.304; y = 0.451; luminance: 25 cd/m²) colors. There were alternating simple and difficult search blocks, each having 12–20 trials. Within each block of trials, the color of the target and distractors was randomly swapped. Two consecutive trials were separated by an interval of 1,500 ms. Subjects were given 120–200 trials of practice to get acquainted with the task before the experiment began.

Fig. 1.

A: visual search task. After a fixation delay, the subject was presented with either an easy search array (discrimination between red and green colors) or a difficult search array (discrimination between green and blue-green colors). The figures on the right show the correct, error, and dissociated responses in the coordinated condition. B: eye block of the dual task. Subjects made either only a saccade to the green target after a fixation delay, or both eye (purple) and hand (blue) movements to the target if a tone was also presented. The SOA could be 0, 200, or 400 ms. The figures on the right of the task show the correct response in each case. C: hand block of the dual task. Similar to the eye block, but here the default movement was the hand movement to the red target, with a saccade to the target if the tone was presented.

Subjects performed three separate blocks of trials. In the eye-alone block (∼200 trials), subjects had to make only eye movements, while keeping the hand fixed at the center. In the hand-alone block (∼200 trials), subjects had to make only hand movements while keeping the eyes fixed at the center. Whereas in the coordinated block (∼300 trials), subjects had to make both eye and hand movements. The order of presentation of eye-alone, hand-alone, and coordinated blocks was counterbalanced across subjects.

criterion for determining the outcome of the trials.

The outcome of the trials was ascertained using the direction of the first saccade and first hand movement. We divided the movement amplitude into four segments and considered the direction of the first segment of movement. If the direction of the movement at this point was in the direction of the target, the movements were deemed correct. If the direction of either movement was toward a distractor, it was considered to be an error trial.

rt cutoffs.

RT was calculated as the difference between the movement onset and the target onset and corrected for the system delay. Typically eye RT precedes hand RT by 80–100 ms (Biguer et al. 1984; Prablanc et al. 1979; Sailer et al. 2000); hence trials in which eye onset was after hand onset were removed from the analysis (Fischer and Rogal 1986). This accounted for ∼9% (8 subjects: <5 ± 3%, differences between eye and hand RT in these trials = 46 ± 24 ms; 2 subjects: ∼16 ± 1%, differences between eye and hand RT in these trials = 141 ± 88 ms) of correct eye-hand coordinated trials. To remove outliers, we considered only those trials whose RTs were within the 25th percentile − 1.5 times the interquartile range (IQR), and 75th percentile + 1.5 times the IQR for both eye and hand RTs. In total, ∼5% of eye RT and <1% of hand RT were removed.

Dual task.

The experiment was divided into two blocks of trials: eye block (Fig. 1B) and hand block (Fig. 1C). The sequence of the two blocks was counterbalanced across subjects.

Eye block.

Each trial began with a white fixation box appearing at the center of the screen where the subjects had to fixate their eyes and hand. After a fixation period of 500-1,000 ms, a green target was presented either to the left or right of the central fixation box. Subjects had to make an eye movement to the target as fast and as accurately as possible while keeping the hand fixed at the center (eye-alone trials). In 40% of the trials, an auditory tone (1,000 Hz for 100 ms, ∼70 dB) was presented at 0, 200, or 400 ms after the presentation of the target. The time between the presentation of the target and tone is called the stimulus onset asynchrony (SOA). In these trials, subjects had to additionally make a hand movement to target. Each correct trial was indicated with a green tick displayed on the screen. Eye-alone trials, where a hand movement was made was indicated by a red cross on the screen. Trials in which a response was not initiated within 800 ms of the target onset, or did not finish within 1,600 ms, were aborted. There was no feedback in aborted trials. Each trial was followed by an intertrial interval of 1,500 ms. Subjects executed ∼300 trials of this condition.

Hand block.

Each trial began with the subject fixating his/her eyes and hand on the central fixation box. After a fixation time of 500-1,000 ms, a red target appeared either to the left or right of the central fixation box. In these trials, the subject had to make a hand movement to the target while maintaining eye fixation (hand-alone). In 40% of the trials, a tone was randomly presented at a SOA of 0, 200, or 400 ms after target presentation. In these trials, subjects also had to make an eye movement to the target. Correct trials were indicated on the monitor with a green tick, while the eye fixation errors in hand-alone trials were indicated with a red cross. Trials where a response was not initiated within 800 ms of the target onset, or not finished within 1,600 ms, were aborted. There was no feedback in such trials. An interval of 1,500 ms separated two trials. Each subject performed ∼300 trials of this condition.

rt cutoffs.

The RTs during the eye block and hand block conditions were checked for outliers separately as the mean RT in the two blocks were different. RTs greater than 25th percentile − 1.5 times the IQR and less than 75th percentile + 1.5 times the IQR were included in the analysis. This cutoff removed ∼6% of eye RTs and ∼7% of hand RTs.

criterion for determining the outcome of the trials.

Correct trials were ascertained as those trials in which the direction of the first saccade and the direction of the first hand movement when one-fourth of the movement was finished were in the direction of the target.

Setup of the Experiment

Experiments were conducted using TEMPO/VIDEOSYNC software (Reflecting Computing, St. Louis, MO). This software generated the stimuli and simultaneously acquired the data in real-time at a temporal resolution of ∼1 ms. Eye movements were sampled at 240 Hz by a head-mounted pupil tracker (ISCAN, Boston, MA). Hand movements were sampled at 240 Hz by an electromagnetic tracking system, which reads the position of a tracker placed on the tip of the pointing finger, with reference to a source (LIBERTY, Polhemus, Colchester, VT). Both eye- and hand-tracking systems interfaced with TEMPO in real time with a delay of 8 ± 1 ms.

EMG activity was measured using 10-mm gold cup electrodes (Care Fusion) using a Cerebus data-acquisition system (BLACKROCK Microsystems). The data were sampled at 1 kHz and band-passed between 10 and 250 Hz and stored in TEMPO in real time. The EMG activity of anterior and posterior deltoid muscles was recorded by placing an electrode on the belly of the muscle. The reference electrode was always placed at the elbow, while the ground electrode was placed on the ear lobe of the subject.

Stimuli were displayed on a 24-in. LED DELL monitor (60 Hz), which was placed face down on a wooden frame. Stimuli were reflected on a semitransparent mirror (25% transmission, 75% reflectance) placed at an angle below the monitor. Images were seen on this plane while hand movements were made on an acrylic sheet placed parallel to and below the mirror. This setup gave the impression that both the eye and hand movements were being made in the same plane, while reducing the interference of the monitor on the hand tracker (see Gopal et al. 2015).

Recording Procedures

All of the experiments were conducted in a dark room. Subjects sat with their chin resting at the wooden frame setup looking down at the mirror. A tracker and battery-driven LED were strapped at the tip of the pointing finger to provide visual feedback of their finger position. Head movements were minimized by locking the head at the temple. A head-mounted eye camera was positioned below the eye such that it did not block the stimuli. Each session began with a calibration block in which subjects looked at targets on the screen, and the eye gains and camera positions were adjusted.

Analyzing EMG Signals

For detection of EMG onset times, the full-wave-rectified EMG signal was first smoothed using an 8-ms uniform window. For each trial, the threshold for determining EMG onsets was fixed at a level where 70% of the EMG signal during the baseline period (i.e., from beginning of the trial to target onset) was below the threshold. This measure gave a better detection of the EMG onset compared with a fixed SD above the baseline noise level.

Detection of Eye and Hand Movement Onset

Saccade onset and ending were demarcated when the instantaneous velocity exceeded and fell below 30°/s. Each saccade was further validated using its acceleration-deceleration profile by checking that the peak in acceleration was followed within 100 ms by a peak in deceleration. Furthermore, the saccade beginning and ending were adjusted, depending on the peak velocity of the saccade. After the saccade beginning and endings were marked, only those saccades whose amplitudes were greater than 2° and less than 24° were accepted. These stringent detection criterions made sure that there was no erroneous detection of saccades. Hand movement beginning and end was marked using a velocity cutoff of 10 cm/s. Movements with velocity >300 cm/s were removed from analysis. The hand beginning and end were also modified using the peak velocity of the hand movement. Furthermore, the movements had to satisfy an amplitude criterion (>1 cm) and a duration criterion (>40 ms).

Statistical Tests

The data were first checked for normality using Lilliefors test. If normally distributed, a two-tailed t-test was used; else a signed-rank test was used. Unless otherwise mentioned, the Pearson's correlation coefficient was used. To test if the SDs of two distributions matched, an F-test was used. In Figs. 3–8, to mark statistical significance, the following convention has been used: *P ≤ 0.05 and *P > 0.01, **P < 0.01 and **P ≥ 0.001, ***P < 0.001, and ns, nonsignificant. Cohen's d, which measures the effect size, was also computed.

Fig. 3.

Behavioral evidence of the common accumulator model. A: scatter plot showing that both eye (purple) and hand (blue) RT increases in the difficult search condition compared with the easy search condition. Each dot and cross-hairs represents the mean ± SE RT of a subject, while the square and the cross-hairs represent the mean ± SE RT across the population. The solid dots represent significant difference in mean RT between the easy search and difficult search conditions. The black dotted line represents the unity line. B: scatter plot showing that the SD of eye and hand RT are similar in both the easy search (green) and difficult search (red) conditions. The open dots represent subjects where SD of eye and hand RT distributions was comparable, and the solid dots represent subjects where the SDs were significantly different. The population mean ± SE is represented by the square and cross-hairs. C: scatter plot showing the correlation between eye and hand RT in the easy search (green) and difficult search (red) condition for a single subject. Each dot represents a trial, while the lines represent the least squares fits. D: bar plot comparing the RT correlation in the easy search (green) and difficult search (red) conditions. Each dot represents a subject, and the bar and the cross-hairs represent the population mean ± SE. E: scatter plot showing the linear relationship between the difference in the mean RT between the easy search and difficult search conditions and the difference between the SD of the RT between the easy search and difficult search conditions, for the eye (purple) and hand (blue). The lines represent the least squares fits. The bars (eye, purple; hand, blue) in the inset figure depict the mean ± SE of slope of the least squares fits. ns, Nonsignificant. ***P < 0.001.

Fig. 8.

Architecture in the dissociated trials. A: comparison of the eye-hand RT correlation between dissociated trials (blue) and correct trials (green) in the easy search and difficult search conditions. Each dot represents a subject, with the solid dots representing significant correlation and the open dots representing insignificant correlation. The bars and cross-hairs represent the population mean ± SE. B: comparison of the absolute difference between the SD of eye and hand RT distributions between dissociated trials and correct trials. Markers and colors are the same as in A. C: the percentage of dissociated trials across subjects was negatively correlated with the RT correlation in the easy search (green) and difficult search (red) conditions. Each dot represents a subject, while the lines represent the least squares fit. The black line represents the least squares fit across the population. D: the percentage of dissociated trials across subjects was positively correlated with the absolute difference between the SD of eye and hand RT distribution. The markers and colors are as in C. *P ≤ 0.05 and P > 0.01. **P < 0.01 and P ≥ 0.001. ***P < 0.001. ns, Nonsignificant.

Modeling RT Distributions

RT distributions were modeled using a drift diffusion model (Fig. 2A). This model simulates the activity of a “GO” unit in which sensory evidence is accumulated after an afferent delay of 60 ms. Once this accumulation process reaches a threshold, a movement is initiated. This kind of accumulation to threshold has been observed in many regions of the brain, including movement neurons of frontal eye field (Hanes and Schall 1996), lateral interparietal area (Shadlen and Newsome 2001), superior colliculus (Horwitz and Newsome 1999), dorsal premotor cortex (Song and McPeek 2010) and basal ganglia (Ding and Gold 2010). At each time instance, the level of accumulation can be expressed as:

Fig. 2.

Diffusion models. A: schematic showing drift diffusion accumulation to threshold. The diffusion process starts after an afferent delay of 60 ms and gathers sensory evidence. Once this diffusion process reaches a threshold (gray bar), a movement is initiated. The green, pink, and yellow traces indicate diffusion processes with gradually decreasing slopes, respectively. The RT distributions resulting from the diffusion processes are shown on the top. A schematic representation of the target display with a central fixation spot and green peripheral target and red distractors is shown. The dotted black line indicates the time when the search array was displayed. B: approximately linear relationship between mean and SD of RT distributions. Family of lines show the relationship between the mean and SD of the predicted RT distributions. Each dot represents the mean and SD of the RT distribution predicted by simulating a diffusion process with a particular (μ_GO, σ_GO). Each color represents a particular level of σ_GO with the cooler colors indicate a greater σ_GO of the diffusion process. C: schematic of the common accumulator model. The blue dotted and solid purple line in the target selection stage represent the sensory evidence, which is common for both effectors. In the movement preparation stage, the sensory evidence is integrated by a common accumulation after an afferent delay of 60 ms. Once the diffusion process reaches the threshold, an eye movement is generated (the time is indicated with a dotted purple vertical line), while the hand movement can start only after a peripheral delay (shaded green region, the time of hand onset is indicated with dotted blue line). D: schematic representation of separate accumulator model. The solid purple and dotted blue line represents the sensory evidence of the eye and hand during target selection. The sensory signals are perfectly correlated. During movement preparation, the sensory evidence is accumulated by the eye (purple) and hand (blue) accumulators. Furthermore, there are interactions from the hand accumulator onto the eye accumulator (blue dotted arrow), and vice versa (pink dotted arrow). The time of saccade and hand movement onset is indicated by purple and blue dotted lines, respectively.

$${\alpha }_{\text{GO}}={\alpha }_{\text{GO}}\left(t-1\right)+{\mu }_{\text{GO}}+{\epsilon }_{\text{GO}}$$ (1)

where α_GO is the level of accumulation at time t; μ_GO is the mean drift rate, which represents the mean strength of the input sensory signal; and ε_GO is a Gaussian noise term with mean 0 and SD equal to that of the GO unit, which represents the noise in the input signal.

The accumulation was modeled to be ideal, i.e., without any leak with a time step of 1 ms. The stochastic nature of the accumulation process leads to the generation of a distribution of RT, which changes with change in μ_GO. The variability of the accumulation process at any time point t during the accumulation process is governed by the equation:

$${\sigma }_{\text{GO}}\left(t\right)=k\cdot {\sigma }_{\text{noise}}\sqrt{t}$$ (2)

where σ_GO is the SD of the accumulation process at time t, σ_noise is Gaussian noise, and k is a scaling constant. In other words, slower RT distributions should have higher SD as RT becomes more variable with greater accumulation of sensory noise. Hence, the drift diffusion model predicts that the SD of a RT distribution scales approximately linearly with its mean (Wagenmakers et al. 2005; Wagenmakers and Brown 2007).

Mean-SD Relationship of the RT Distribution

The relationship between the means and SDs of the predicted RT distribution was formally tested by simulating RT distributions using Eq. 1. Initially, σ_GO was kept constant, and μ_GO was systematically changed. Then σ_GO was changed as well, thus generating RT distributions for a range of values of μ_GO and σ_GO, which was chosen such that it could give rise to behaviorally relevant RT distributions. The relationship between the predicted mean and SD of the RT distributions was approximately linear for all the levels of σ_GO (r > 0.99 in all of the test conditions, Fig. 2B), thus validating the approximately linear relationship between the mean and SD of the predicted RT distributions.

Estimation of Model Parameters

For estimating μ_GO and σ_GO, a range of values that could generate behaviorally relevant RT distributions were uniformly and “coarsely” sampled, to simulate RT distributions of 2,000 trials, using Eq. 1. Only those parameters that generated a RT distribution with a mean and SD within 30 ms of the mean and SD of the empirical RT distribution were fed into MATLAB's fmincon function for minimization. A nonlinear constraint was applied whereby 70% of the simulated RT distribution would have to lie within the extent of the empirical RT distribution. The error was calculated as the difference between inverted Gaussian weighted cumulative distribution functions (CDF) of the empirical and simulated RT distributions. The Gaussian weights enabled better fits to the tails of the RT distributions to obtain a more reliable estimate of the SD of the distributions. This was preferred to just fitting the mean of the empirical RT (Ratcliff et al. 2016). For each μ_GO and σ_GO, the minimum error solution typically converged within <20 iterations. Multiple runs of minimization were simulated using different starting points to ensure that the best parameters were obtained.

Preliminary analysis showed that other measures, like Kolmogorov-Smirinov statistic, difference between the histograms, or difference between the cumulative distribution functions, led to under-prediction of the SDs. A combination of brute force with coarse sampling (steps of ∼100 ms) and fmincon approaches provided a better fit to the experimental data. Using brute force with fine sampling is computationally expensive, while just using fmincon often failed to locate the global minimum in the parameter space. Furthermore, fmincon did not provide a good solution if the initial parameters were far from the global minimum. Thus using a combination of these two gave a superior fit to the experimental data.

Models of Eye-Hand Coordination

Common accumulator model.

This model assumes the existence of a single accumulator for both eye and hand (Fig. 2C). Once this accumulator reaches a threshold, an eye movement is generated (Gopal et al. 2015; Gopal and Murthy 2015). To account for greater hand RT compared with eye RT, a delay, representing the delay in activation of hand muscles leading to hand movement, is added at the periphery. The accumulation at time t can be expressed as:

$${\alpha }_{\text{GO}-\text{Com}}\left(t\right)={\alpha }_{\text{GO}-\text{Com}}\left(t-1\right)+{\mu }_{\text{GO}-\text{Com}}+{\epsilon }_{\text{GO}-\text{Com}}$$ (3)

where α_Go-Com represents the activity of the common accumulator, μ_Go-Com represents the mean drift rate of the accumulator, and ε_Go-Com represents the Gaussian noise with mean 0 and SD μ_Go-Com.

Here, μ_Go-Com and σ_Go-Com (SD of the common accumulator) were first estimated from the observed coordinated eye RT distribution, using the estimation method described above. The hand RT distribution in the coordinated condition was then used to estimate the parameters of the hand delay. A sufficiently large range of values were selected for μ_Delay (mean of the hand delay) and σ_Delay (SD of the hand delay), and only those parameters that satisfied the constraints (mean and SD of the simulated hand RT was within 30 ms of the mean and SD of the experimental hand RT; and 70% of the simulated hand RT distribution lay within the extent of the experimental hand RT distribution) were fed into fmincon for minimization. The details of the estimation procedure have been elaborated in Gopal et al. (2015). After estimation of the best parameters, the predicted RT correlation was calculated as the correlation between the predicted eye and hand RT distributions.

Separate accumulators with interaction between accumulators.

In this model (Fig. 2D), separate accumulators representing the developing motor command of the eye and hand were simulated (Gopal et al. 2015; Gopal and Murthy 2015). Each accumulator had a μ_GO and σ_GO, meaning that they accumulated sensory evidence independent of each other. Based on previous work from the laboratory and preliminary analysis, the RT distributions in the coordinated condition were modeled, not as independent accumulators, but as separate eye and hand accumulators with interaction between the two. Thus β_E→H and β_H→E, representing the strength of the interaction from the eye accumulator to the hand accumulator and the interaction from the hand accumulator to the eye accumulator, respectively, were also incorporated. The accumulation process at time t can be represented as:

$${\alpha }_{\text{E}}\left(t\right)={\alpha }_{\text{E}}\left(t-1\right)+{\mu }_{\text{GO}-\text{E}}+{\beta }_{\text{H}\to \text{E}}\cdot {\alpha }_{\text{H}}\left(t-1\right)+{\epsilon }_{\text{GO}-\text{E}}$$ (4)

$${\alpha }_{\text{H}}\left(t\right)={\alpha }_{\text{H}}\left(t-1\right)+{\mu }_{\text{GO}-\text{H}}+{\beta }_{\text{E}\to H}\cdot {\alpha }_{\text{E}}\left(t-1\right)+{\epsilon }_{\text{GO}-\text{H}}$$ (5)

where α_E and α_H represent the level of accumulation of the eye and hand accumulator, respectively; μ_Go-E and μ_Go-H represent the mean drift rate of the eye and hand accumulator, respectively; and ε_Go-E and ε_Go-H represent the Gaussian noise term associated with the eye and hand accumulator, respectively.

The μ_Go-E and σ_Go-E (SD of the eye accumulator) and μ_Go-H and σ_Go-H (SD of the hand accumulator) were estimated based on the empirical eye-alone and hand-alone RT distributions, respectively, while the interaction parameters were estimated from the observed coordinated eye and hand RT distributions. A sufficiently large range of the interaction parameters were used as seeds, and whichever parameters satisfied the constraints (namely simulated eye and hand means and SDs were within 30 ms of the empirical eye and hand means and SDs, respectively) were fed into fmincon for minimization. The inputs to the eye and hand accumulators were considered to be completely correlated. Further details of the estimation procedure has been presented in Gopal et al. (2015).

This sequential method of first estimating the μ_GO and σ_GO of the eye and hand accumulators and then estimating the interaction parameters was preferred over estimating all of the six parameters simultaneously. Previous work from the laboratory has shown that such a sequential approach is more principled and produced a better estimation of the six parameters, as simultaneous fitting often gave rise to multiple solutions (Gopal et al. 2015; Gopal and Murthy 2015). Once the best parameters were estimated, we calculated the predicted RT correlation as the correlation between the predicted eye and hand RT distributions.

RESULTS

Two key aspects of eye-hand coordination were studied to 1) address whether the common accumulator model was applicable in a task involving explicit decision-making; and 2) elucidate how the temporal coupling between eye and hand was modulated by task context. To address these questions, we recorded the eye and hand movements of subjects while they moved either to a target embedded among distractors (visual search task) or to a target where the GO cues for eye and hand were dissociated (dual task). There were two conditions in each of the experiments: easy search and difficult search in the search task; and eye block and hand block in the dual task. Although the behavioral response of the subjects (eye and hand movement to a peripheral target) was identical in the two tasks, the context in which these movements were generated was distinct.

In the first part of the paper, we tested if the common accumulator model was able to predict the behavior in the visual search task. In the process, we also checked whether the common accumulator model reflected only motor preparation or some aspect of decision-making as well. In the second part, we tested whether the pattern of temporal coordination observed in the dual task is similar to that predicted by the common accumulator or the separate accumulator model. Our working hypothesis was that, in the visual search task, the RT distributions would conform to the predictions of the common accumulator model, while in the dual task, which had distinct GO cues for eye and hand, the RT distributions would conform to the predictions of a separate accumulator model.

Eye-Hand Coordination during Decision-Making

RT distributions in the visual search task.

A previous study from the laboratory has demonstrated that the common accumulator model is applicable in a saccade and reach task (Gopal et al. 2015). Here, using a visual search task, we tested whether this model could be extended to tasks involving explicit decision-making. Besides testing the applicability of the common accumulator model in a decision task, analysis of the behavior in this task also allowed us to test if some aspect of the decision was accumulated.

We first tested whether the task performance decreased in the difficult search condition compared with the easy search condition, which would indicate the efficacy of using colors to manipulate task difficulty. As expected, the percentage of correct trials was lesser in the difficult search condition (53 ± 16%) compared with the easy search condition (89 ± 8%) (Wilcoxon signed-rank test: P = 0.002). This drop in accuracy was not a consequence of a speed accuracy trade-off, as both eye and hand RT also significantly increased in the difficult search condition compared with the easy search condition [Fig. 3A; mean eye RT difference = 102 ± 62 ms, t-test: t(9) = 5.2, P < 0.001, d = 1.5; mean hand RT difference = 118 ± 66 ms, t-test: t(9) = 5.7, P < 0.001, d = 1.8]. Furthermore, in both the easy search and difficult search conditions, mean hand RT was significantly greater than mean eye RT [easy search: mean eye and hand RT difference = 89 ± 24 ms, Wilcoxon signed-rank test: P = 0.002, d = 2.1; difficult search: mean eye and hand RT difference = 105 ± 35 ms, t-test: t(9) = 9.5, P < 0.001, d = 1.2], indicating that the temporal order of execution of coordinated eye-hand movements was maintained in both the trial conditions.

The main parameter of interest was the SD of the RT distributions. If the SD of the RT distribution increased in the difficult search condition compared with the easy search condition, it would suggest that at least some part of the decision process was accumulated. Additionally, if within each condition the SDs of eye and hand RT distributions were comparable, it would suggest that the common accumulator model is applicable to the visual search task. Compared with the easy search condition, the SDs of both eye and hand RT distributions increased in the difficult search condition [mean difference in SD of eye RT = 34 ± 23 ms, t-test: t(9) = 4.7, P = 0.001, d = 1.4; mean difference in SD of hand RT = 33 ± 25 ms, t-test: t(9) = 4.2, P = 0.002, d = 1.6]. This is compatible with the idea that the accumulation process involves some aspect of the decision. However, within each condition, the SD of eye and hand RT distributions were comparable [Fig. 3B; easy search: SD(eye RT) = 74 ± 18 ms, SD(hand RT) = 80 ± 18 ms, difference between SD of eye and hand RT distributions = 6 ± 11 ms, t-test: t(9) = 1.7, P = 0.117, d = 0.3; difficult search: SD(eye RT) = 108 ± 29 ms, SD(hand RT) = 113 ± 24 ms, difference between SD of eye and hand RT distributions = 6 ± 11 ms, t-test: t(9) = 1.6, P = 0.154, d = 0.2]. Furthermore, the correlation between eye and hand RT was high and significant in 10/10 subjects (Fig. 3C; single subject) in both easy search (mean r = 0.77 ± 0.08) and difficult search conditions (mean r = 0.8 ± 0.12) and was comparable between the two conditions (Fig. 3D, Wilcoxon signed-rank test: P = 0.275). This showed that the behavioral results in the search task were consistent with the two main predictions of the common accumulator model.

The time of EMG onset (in the anterior deltoid muscle) with respect to the time of saccade onset provided another approach to validate the common accumulator model. If a common signal is send to both eye and hand, then the EMG onset should occur before saccade onset, because eye muscle activation should take some time, albeit much less than the activation of hand muscles. Furthermore, the saccade and EMG onsets should be well correlated. However, if separate signals exist for eye and hand, then the EMG onset should usually begin after saccade onset, as mean hand RT is greater than mean eye RT. Furthermore, the saccade and EMG onset should not be correlated. For this analysis, only trials that entailed movement in the muscle's (anterior deltoid) movement field were considered (top and left targets). The EMG onset was before saccade onset in ∼79 ± 17% and ∼74 ± 25% of trials in the easy search and difficult search conditions, respectively. Additionally, eye RT and EMG onsets were also strongly correlated in both the easy search (mean r = 0.88 ± 0.06) and difficult search conditions (mean r = 0.94 ± 0.03), suggesting that eye and hand movements were initiated by a common signal. Such a high correlation is expected to reduce in the presence of peripheral noise, or a peripheral hand-specific delay, as predicted by the common accumulator model. Consistent with this notion, the correlation between eye-hand RT (easy search: mean r = 0.77 ± 0.08; difficult search: mean r = 0.8 ± 0.12) was lesser than the correlation between eye-EMG RT (Wilcoxon signed-rank test: easy search: P = 0.002; difficult search: P = 0.004).

Interestingly, a strong linear relationship was observed between the difference of the mean RT in the easy search and difficult search condition and the difference between the SD of the RT distribution in the easy search and difficult search condition for both eye and hand (Fig. 3E; eye: r = 0.9, P < 0.001, hand: r = 0.84, P < 0.001). This linear relationship suggests that the additional time taken for the decision in the difficult search condition leads to a concomitant increase in the SD of the RT distribution, providing additional evidence that some aspect of the decision is accumulated.

Validation of the Common Accumulator Model Using Simulations

We simulated the eye and hand RT distributions in the easy search and difficult search conditions to check if the behavioral results were predicted by the common accumulator and separate accumulator models. Since in the common accumulator model the eye RT was directly fitted using the diffusion model (Tables 1 and 2), the comparison between the common accumulator and separate accumulator models was made using the predicted hand RT and the predicted RT correlation.

Table 1.

Experimental and predicted eye RT for the Common and Separate Accumulator models in the Easy Search condition

	Mean			SD
Subject No.	Experimental	CA	SA	Experimental	CA	SA
1	446.11	439.56	444.65	61.89	61.88	69.56
2	443.92	450.89	422.30	79.60	78.25	72.58
3	418.28	418.56	414.54	70.40	67.04	83.79
4	510.77	520.22	516.14	87.37	84.98	118.13
5	450.56	456.65	427.31	54.08	51.74	64.37
6	401.93	400.85	399.34	66.72	68.67	74.62
7	393.04	393.78	411.14	66.36	64.88	76.21
8	419.37	409.97	399.75	63.27	62.11	72.90
9	519.44	527.70	514.85	118.22	119.88	107.09
10	438.89	433.47	412.53	71.22	69.38	74.30

CA, Common Accumulator; SA, Separate Accumulator. Predicted values that are significantly different from the experimental value are shown in bold.

Table 2.

Experimental and predicted eye RT for the Common and Separate Accumulator models in the Difficult Search condition

	Mean			SD
Subject No.	Experimental	CA	SA	Experimental	CA	SA
1	616.43	595.80	557.39	121.42	125.53	136.76
2	602.37	608.44	636.28	124.50	124.28	116.98
3	552.95	547.46	581.69	123.52	123.11	106.29
4	585.82	576.35	520.48	119.34	118.81	149.00
5	588.26	571.60	551.74	101.02	102.32	120.63
6	398.43	400.65	378.68	71.39	70.76	84.45
7	400.90	399.84	395.95	68.40	68.37	73.87
8	488.22	489.77	485.23	69.25	71.32	84.39
9	651.96	658.10	630.01	147.63	142.31	120.32
10	581.73	555.92	537.64	129.16	126.97	136.36

CA, Common Accumulator; SA, Separate Accumulator. Predicted values that are significantly different from the experimental value are shown in bold.

Across the population, the mean of the hand RT distribution was well predicted by both models in the easy search condition [Fig. 4A; common accumulator: 10/10 subjects, t-test: t(9) = 1.4, P = 0.19, d = 0.1; separate accumulator: 4/10 subjects, t-test: t(9) = 1.6, P = 0.138, d = 0.2] and the difficult search condition [Fig. 4B: common accumulator: 10/10 subjects, t-test: t(9) = 1.3, P = 0.212, d = 0.02; separate accumulator: 8/10 subjects, t-test: t(9) = 1.1, P = 0.32, d = 0.1]. Nevertheless, the SD of the hand RT distribution was better predicted by the common accumulator model compared with the separate accumulator model in both the easy search [Fig. 4C; common accumulator: 9/10 subjects, t-test: t(9) = 1.4, P = 0.186, d = 0.1; separate accumulator: 4/10 subjects, t-test: t(9) = 4.2, P = 0.002, d = 0.8] and the difficult search conditions [Fig. 4D; common accumulator: 10/10 subjects, t-test: t(9) = 1.9, P = 0.092, d = 0.2; separate accumulator: 5/10 subjects, t-test: t(9) = 3.5, P = 0.006, d = 0.7]. Furthermore, the RT correlations predicted by the common accumulator model were well correlated with the observed one, while this was not the case with the separate accumulator model in both the easy search condition [Fig. 4E; common accumulator: Spearman ρ = 0.71, P = 0.028; separate accumulator: Spearman ρ = 0.3, P = 0.407] and the difficult search condition [Fig. 4F; common accumulator: Spearman ρ = 0.79, P = 0.01; separate accumulator: Spearman ρ = 0.5, P = 0.143]. Taken together, these results indicate that the common accumulator model predicted the results in the search task better compared with the separate accumulator model.

Fig. 4.

Validating the common accumulator model in the search task using simulations. A: scatter plot comparing the experimental mean hand RT and that predicted by common accumulator (turquoise blue) and separate accumulator (orange) models in the easy search condition. Each dot and cross-hairs represent the mean ± SE of a subject, while the squares and cross-hairs represent the population mean ± SE. Solid dots represent significant difference at the single subject level. B: same as A but for the difficult search condition. C: scatter plot showing that the common accumulator model predicts the SD of hand RT distribution better than the separate accumulator model in the easy search condition. D: same as C but for the difficult search condition. E: scatter plot comparing the experimental RT correlation and that predicted by the common accumulator model and the separate accumulator model. F: same as E but for the difficult search condition. *P ≤ 0.05 and P > 0.01. **P < 0.01 and P ≥ 0.001.

We also tested a variant of the common accumulator model in which a higher threshold was is used for hand effector compared with the eye effector (threshold model) in place of a peripheral delay (delay model). These differential thresholds ensure the proper temporal order of execution of the eye and hand movements. While both the threshold and delay model predicts greater mean hand RT compared with mean eye RT, the two models can be distinguished based on the predicted SD of the hand RT distribution. As the delay model does not contain any accumulation process after the end of the common accumulation, it predicts that the SDs of eye and hand RT distributions will be comparable. Conversely, the threshold model contains an extra accumulation time for the hand; thus it predicts that the SD of hand RT distribution will be greater than that of the eye RT distribution. While the threshold model was able to predict the mean of the hand RT distribution in both easy search (Wilcoxon signed-rank test: P = 1, d = 0.1) and difficult search [t-test: t(9) = 1.2, P = 0.28, d = 0.04] conditions, it over-predicted the SDs of the hand RT distribution in the difficult search condition [t-test: difficult search: t(9) = 2.8, P = 0.02, d = 0.02; easy search: t(9) = 0.1, P = 0.892, d = 0.2]. This suggests that the common accumulator model with a peripheral delay is a better model in the search task.

Eye-Hand Coordination in the Dual Task

RT distributions in the dual task.

Next we analyzed the RT distributions in the 0-ms SOA trials of the eye block and hand block conditions of the dual task, where the GO cues of the eye and hand effectors were presented simultaneously. Hence, the behavioral response expected of the subjects was similar to that in the search task. In the eye block condition (Fig. 5A), the mean hand RT (451 ± 93 ms) was greater than mean eye RT (228 ± 40 ms) in 8/8 subjects and across the population as well [t-test: t(7) = 8, P < 0.001, d = 3.1]. In contrast to that observed in the search task, the SD of hand RT distributions (67 ± 17 ms) was greater than that of the eye RT distribution (38 ± 15 ms) in 7/8 subjects and across the population [t-test: t(7) = 8.6, P < 0.001, d = 1.8]. In the hand block condition (Fig. 5B), 4/8 subjects had significantly greater mean eye RT compared with mean hand RT, 3/8 subjects had mean eye RT greater than that of the hand RT, while the mean eye and hand RT was comparable in one subject. Due to the mixed nature of results, at a population level, mean eye and hand RT were comparable [population mean eye RT = 441 ± 137 ms, population mean hand RT = 425 ± 107 ms, t-test: t(7) = 0.4, P = 0.73, d = 0.1]. Interestingly, the three subjects who showed greater mean hand RT compared with eye RT showed comparable SDs, while the remaining subjects showed greater SD of the hand RT compared with that of the eye RT. At a population level, the SD of the eye RT distribution (134 ± 70 ms) was significantly greater than that of the hand RT distribution (74 ± 42 ms) [t-test: t(7) = 3.6, P = 0.009, d = 1].

Fig. 5.

Behavior in the dual task. A: scatter plot comparing the mean of eye and hand RT distributions in the SOA0 condition of the eye block (violet) and hand block (brown) conditions. The dots and cross-hairs represent the mean ± SE of RT of subjects, while the squares and cross-hairs represent the mean ± SE across the population. The solid dots represent significant difference at the single-subject level. The unity line is shown as a black dotted line. B: scatter plot comparing the SD of eye and hand RT distributions in the SOA0 condition of the eye block (violet) and hand block (brown) conditions. The dots represent the SD of RT distribution of subjects, while the squares and cross-hairs represent the mean ± SE across the population. C: bar plot showing that the RT correlation is low and comparable between eye block (purple) and hand block (brown) conditions. Each dot represents a subject, with the solid ones representing subjects in whom the correlation is significant. The bar and cross-hairs represent the population mean ± SE. **P < 0.01 and P ≥ 0.001. ***P < 0.001. ns, Nonsignificant.

In contrast to that observed in the search task, the RT correlation was poor in both eye block (mean r = 0.23 ± 0.15) and hand block (mean r = 0.38 ± 0.28) conditions. In the eye block condition, the RT correlation was significant in only 2/8 subjects, while, in the hand block condition, the RT correlation was significant in 5/8 subjects. There was no significant difference between the population RT correlation observed in the eye block and hand block conditions (Wilcoxon signed-rank test: P = 0.25, Fig. 5C). Thus, at the population level, the data in both the eye block and hand block conditions was consistent with the predictions of the separate accumulator model. Additionally, the eye-EMG (only left targets were considered) correlation was also lower in the dual task compared with the search task. This correlation was significant in only 1/8 subjects in the eye block condition (mean r = 0.13 ± 0.14), and was significant in only 2/8 subjects in the hand block condition (mean r = 0.36 ± 0.31). Additionally, there was no significant difference between the eye-EMG correlation observed between the eye block and hand block conditions (Wilcoxon signed-rank test: P = 0.148).

Validation of the Separate Accumulator Model Using Simulations

In the eye block condition (see Tables 3 and 4 for eye RT), the mean hand RT (451 ± 93 ms) was well predicted by both the models [Fig. 6A; common accumulator: population mean hand RT = 447 ± 93 ms, t-test: t(7) = 1.5, P = 0.183, d = 0.04; separate accumulator: population mean hand RT = 443 ± 94 ms, t-test: t(7) = 1.2, P = 0.27, d = 0.1] conditions. In the eye block condition, the separate accumulator model correctly predicted the SD of the hand RT in all of the subjects as well, but the common accumulator model could predict the SD of hand RT in only 3/8 subjects (Fig. 6B). Across the population, there was no significant difference between the observed SD of hand RT distribution (67 ± 17 ms) and that predicted by the separate accumulator model (68 ± 18 ms) [t-test: t(7) = 0.4, P = 0.701, d = 0.1], but the common accumulator model significantly under-predicted the SD of hand RT distribution (54 ± 18 ms) [t-test: t(7) = 4.9, P = 0.002, d = 0.7]. At a population level (Fig. 6E), the RT correlation predicted by the separate accumulator model (0.38 ± 0.18) was similar to the observed RT correlation (0.13 ± 0.14) (Wilcoxon signed-rank test: P = 0.2), while the common accumulator model (0.76 ± 0.16) over-predicted it (Wilcoxon signed-rank test: P = 0.008). In the hand block condition, both of the models predicted the mean [Fig. 6C, common accumulator: mean hand RT = 447 ± 92 ms, t-test: t(7) = 1.1, P = 0.31, d = 0.1; separate accumulator: mean hand RT = 420 ± 110 ms, t-test: t(7) = 1.2, P = 0.256, d = 0.1] and SD [Fig. 6D, common accumulator: SD(hand RT) = 70 ± 36 ms, t-test: t(7) = 0.9, P = 0.413, d = 0.1, separate accumulator: SD(hand RT) = 65 ± 34 ms, t-test: t(7) = 2.2, P = 0.07, d = 0.2] of the hand RT distribution. However, comparison of the observed and predicted RT correlation showed that the common accumulator model (0.97 ± 0.07) over-predicted the RT correlation, while the separate accumulator model (0.49 ± 0.32) could correctly predict the RT correlation (Fig. 6E; Wilcoxon signed-rank test: common accumulator: P = 0.008; separate accumulator: P = 0.383). Taken together, this showed that the separate accumulator model could predict the data in the dual task better in both the eye block and hand block conditions.

Table 3.

Experimental and predicted eye RT for the Common and Separate Accumulator models in the Eye Block condition

	Mean			SD
Subject No.	Experimental	CA	SA	Experimental	CA	SA
1	193.3	196.81	195.73	28.53	29.5	35.01
2	235.76	236.7	219.47	43.72	47.6	50.45
3	199.92	201.06	201.72	19.43	19.6	19.17
4	227.16	227.7	226.98	35.58	34.6	30.52
5	290.48	290.44	280.71	47.14	45.25	48.93
6	190.67	192.05	194.45	22.98	20.21	20.47
7	198.41	196.73	198.41	39.11	36.94	40.65
8	284.72	285.95	283.04	67.04	68.49	51.96

CA, Common Accumulator; SA, Separate Accumulator. Predicted values that are significantly different from the experimental value are shown in bold.

Table 4.

Experimental and predicted eye RT for the Common and Separate Accumulator models in the Hand Block condition

	Mean			SD
Subject No.	Experimental	CA	SA	Experimental	CA	SA
1	250.58	251.79	249.81	47.89	46.88	45.58
2	392.11	383.92	407.88	109.94	104.31	115.26
3	620.70	638.72	583.22	231.35	216.76	152.23
4	441.20	424.71	448.41	113.95	112.85	93.58
5	565.69	539.48	557.33	175.09	171.96	160.32
6	253.51	253.89	257.97	38.13	37.29	40.70
7	476.13	424.01	421.90	209.72	167.21	196.47
8	527.92	559.15	513.89	141.74	141.29	126.20

CA, Common Accumulator; SA, Separate Accumulator. Predicted values that are significantly different from the experimental value are shown in bold.

Fig. 6.

Validation of behavioral results using simulations. A: comparison of the experimental mean hand RT with the mean hand RT predicted by the common accumulator (turquoise blue) and separate accumulator (orange) models in the eye block condition. Each dot and cross-hair represents the mean ± SE of a subject's hand RT. Solid dots represent significant difference at the single-subject level. The square and cross-hairs represent the population mean ± SE. The population means of both the models almost coincide. B: same as A but for the hand block condition. The population means of both of the models almost coincide. C: comparison of the experimental SD of hand RT distribution with that predicted by the common accumulator and separate accumulator models. The squares and cross-hairs represent the mean ± SE across the population. D: same as C but for the hand block condition. E: comparison of the experimental RT correlation (gray) and that predicted by the common accumulator (turquoise blue) and separate accumulator (orange) models for the eye block and hand block conditions. Each dot represents the RT correlation for a subject, while the bar and cross-hair represents the population mean ± SE. **P < 0.01 and P ≥ 0.001. ***P < 0.001. ns, Nonsignificant.

We also compared the direction of interaction (from the eye accumulator to the hand accumulator and vice versa, see methods) predicted by the separate accumulator model in the eye block and hand block conditions. In the eye block condition, there was a positive interaction from hand to eye (mean = 0.0044 ± 0.0028), which resulted in the eye-coordinated RT (226 ± 36 ms) being faster than the eye-alone RT [246 ± 44 ms; t-test: t(7) = 4.2, P = 0.003, d = 0.6]. The interaction from eye to hand was not significantly different from zero [mean = 0.0009 ± 0.007; t-test: t(7) = 0.4, P = 0.73, d = 0.3]. In the hand block condition, the hand imposed a negative interaction on the eye accumulator, making the eye-coordinated RT (430 ± 125 ms) slower compared with the eye-alone RT [t-test: t(7) = 4.1, P = 0.004, d = 1.9]. In contrast, the eye accumulator had a positive interaction on the hand accumulator, making the hand-coordinated RT (420 ± 110 ms) faster compared with the hand-alone RT (472 ± 88 ms) [t-test: t(7) = 3.2, P = 0.015, d = 0.5]. This demonstrated that, even in the dual task, the eye block and hand block conditions illustrated distinct patterns of interaction between the two independent accumulators. This further highlights the importance of the task context in determining the nature of eye-hand coordination.

Validation of the Separate Accumulator Model Across SOAs in the Dual Task

In the preceding sections, we have reported the RT distributions in the SOA 0-ms trials in the dual task. Next we tested whether the predictions of the separate accumulator model held across all SOAs in the dual task. The eye and hand RT correlation was poor and often insignificant across all the SOA, both in eye block (SOA 200 ms: mean r = 0.1 ± 0.25, SOA 400 ms: −0.04 ± 0.13) and hand block (SOA 200 ms: mean r = 0.19 ± 0.12, SOA 400 ms: −0.2 ± 0.36) (Fig. 7A). In the eye block condition, at SOA 200 ms, the SD of the hand RT distribution (70 ± 24 ms) was significantly greater than that of the eye RT distribution (58 ± 14 ms) [t-test: t(7) = 2.4, P = 0.044, d = 0.7]. At SOA of 400 ms, except for one subject, all of the subjects showed a greater SD of hand RT distribution (57 ± 29 ms) compared with eye RT distribution (69 ± 29 ms). However, this did not reach significance [t-test: t(7) = 1.5, P = 0.18, d = 0.4] (Fig. 7B). In the hand block condition (Fig. 7C), at SOA 200 ms, the SD of the eye RT distribution (110 ± 37 ms) tended to be greater than the SD of the hand RT distribution (78 ± 28 ms), but did not reach statistical significance [t(7) = 2, P = 0.081, d = 1]. At SOA of 400 ms, at a population level, there was no significant difference between the SD of the eye RT distribution (122 ± 56 ms) and the SD of the hand RT distribution (87 ± 52 ms) [t-test: t(6) = 1, P = 0.379, d = 0.7]. Notably, subjects made few correct responses at higher SOAs, making the SD of RT distributions an insensitive measure of deciphering the architecture underlying the generation of the movements. Overall, this analysis suggested that eye and hand RT at SOAs of 200 ms and 400 ms were largely consistent with the predictions of the separate accumulator model.

Fig. 7.

Validity of the separate accumulator model across SOAs. A: eye and hand RT correlation across all the SOAs in the eye block (violet) and hand block (brown) condition. Each dot represents the RT correlation of a subject, where the open dots represent insignificant correlation. The solid squares and cross-hair connected by a black line represents the mean ± SE of the RT correlation across the population. B: comparison of the SDs of eye (purple) and hand (blue) RT distributions at different SOAs in the eye block condition. Each dot represents a subject. The bars and cross-hair represents the population mean ± SE. C: same as B, but for the hand block condition. *P ≤ 0.05 and P > 0.01. **P < 0.01 and P ≥ 0.001. ***P < 0.001. ns, Nonsignificant.

Dissociated Trials in the Search Task

It could be argued that the evidence of the separate accumulator model being applicable in the dual task was just a reflection of the task which had dissociated GO cues for eye and hand. Evidence against this was observed in a specific fraction of trials within the search task. In the majority of the trials in the search task, both eye and hand went to a common target location, as expected by a common accumulator model. However, in a small fraction of trials called dissociated trials (see Fig. 1A), we observed that eye and hand movements were executed to different directions (easy search: 6.6 ± 5.2%, difficult search: 7.2 ± 4.9%). The percentage of dissociated trials was similar between the easy search and difficult search conditions (Wilcoxon signed-rank test: P = 0.557), indicating that they were not a manifestation of the task difficulty. In these dissociated trials, the hand movement was usually to the correct target in both the easy search condition (85.2 ± 15.6%) and the difficult search condition (60.8 ± 20.5%), while the first saccade usually landed on a distractor. Thus these trials represented a fraction of trials where the target selection was separate for the first eye and hand movements.

We hypothesized that, when target selection is separate, movement preparation stages for the eye and hand are likely to be distinct as well. Within the framework of the drift diffusion model, separate movement preparation stages would entail separate accumulators. This suggests a possibility that the small fraction of dissociated trials seen during the visual search task may be generated by the separate accumulator model rather than the common accumulator model, which generated the majority of the trials. Consistent with this idea, the difference between the eye and hand RT was greater in the dissociated trials (easy search: 228 ± 58 ms, difficult search: 248 ± 92) compared with correct trials (easy search: 89 ± 24 ms, difficult search: 104 ± 35 ms) [easy search: t-test: t(6) = 5.3, P = 0.002, d = 2.5; difficult search: t-test: t(6) = 4.8, P = 0.003, d = 1.9] (subjects with <4 dissociated trials were removed from the analysis). A comparison between correct and dissociated trials showed that the eye RT was faster [easy search: mean eye RT difference = −43 ± 22 ms, t-test: t(6) = 5.2, P = 0.002, d = 0.7; difficult search: mean eye RT difference = −45 ± 31 ms, t-test: t(6) = 3.8, P = 0.009, d = 0.6] and the hand RT was slower [easy search: mean hand RT difference = 90 ± 57 ms, Wilcoxon signed-rank test: P = 0.016, d = 2.1; difficult search: mean hand RT difference = 94 ± 67 ms, t-test: t(6) = 3.7, P = 0.01, d = 1.1] in the dissociated trials. Furthermore, in the dissociated trials, the RT correlation was lower compared with correct trials in both the easy search (dissociated: mean r = 0.41 ± 0.3; correct: mean r = 0.77 ± 0.08; Wilcoxon signed-rank test: P = 0.031) and difficult search (dissociated: mean r = 0.5 ± 0.29; correct: mean r = 0.8 ± 0.12; Wilcoxon signed-rank test: P = 0.047) conditions (Fig. 8A). Also, the difference between the SD of the eye and hand RT distribution was greater in the dissociated trials compared with the correct trials in the difficult search condition (difference between SD of eye and hand RT distribution, easy search: 32 ± 28 ms, difficult search: 24 ± 14 ms) [Fig. 8B; easy search: Wilcoxon signed-rank test: P = 0.156, d = 0.8; difficult search: t-test: t(6) = 2.5, P = 0.049, d = 0.2]. These results are consistent with the predictions of the separate accumulator model, indicating that the dissociations may represent a small percentage of trials where effector-specific target selection and movement preparation stages were instantiated. In accordance with this idea, across subjects, the percentage of dissociated trials was negatively correlated with the increase in RT correlation (Fig. 8C, easy search: r = −0.51, P = 0.135; difficult search: r = −0.86, P = 0.001; combined: r = −0.68, P = 0.001). In addition, the percentage of dissociated trials was positively correlated with the difference between the SDs of eye and hand RT distributions (Fig. 8D, easy search: r = 0.68, P = 0.03; difficult search: r = 0.48, P = 0.165; combined: r = 0.59, P = 0.005). Taken together, these results demonstrate that the computational architecture for eye-hand movements may consist of a mixture of common accumulator model and a separate interactive accumulator model, with the frequency of instantiation of each architecture being modulated by task demands.

DISCUSSION

Previous behavioral studies investigating the temporal coupling between eye and hand movements have found heterogeneous results. The correlations reported ranged from low to high and thus have led to confusion whether the eye and hand initiations are controlled by a common signal or independent signals. Although some studies have implicitly shown the dependence of task context (Frens and Erkelens 1991; Sailer et al. 2000), to the best of our knowledge no study has explicitly contrasted the context dependency of eye-hand coordination. Here, using two behavioral tasks, we were able to delineate the two patterns of temporal coupling and provide a computational framework which could explain flexible and heterogeneous behavior previously reported in the literature.

We propose that, depending on the behavioral context, either a common accumulator model with effector-specific noise (Gopal et al. 2015) or separate interacting accumulators model (Dean et al. 2011) can best explain the temporal pattern of eye-hand coordination. Using two behavioral measures, comparability of SDs of the eye and hand RT distributions and RT correlation, we suggest that the common accumulator model can explain the results observed in the visual search task. Conversely, when the initiation signals for eye and hand were decoupled, the pattern of eye and hand RT followed the predictions of the separate accumulator model. These results were further validated by simulating the common accumulator and separate accumulator model using the drift diffusion framework.

Context Dependency of Eye-Hand RT Correlation

Using a multitude of tasks, previous studies have reported a range of correlations between eye and hand RT. Low RT correlation has been reported by Dean et al. (2011), who trained nonhuman primates to make saccade and reaching movements to a common target but with separate GO cues for the eye and hand. They observed correlations of ∼0.4 at short SOAs, which decreased and often became insignificant at larger SOAs. Such low and often insignificant RT correlations have also been observed in our study. Dissociation of the GO signal presumably biases the brain toward choosing separate networks for initiating eye and hand movements. However, some studies using a saccade and reach task with a single GO signal for eye and hand have also reported low RT correlations. For example, Biguer et al. (1982) observed low correlations (∼0.4) between saccade onset and biceps EMG onset. In a similar task, Prablanc et al. (1979) reported modest RT correlations (0.5). Gielen et al. (1984) additionally used double-step target displacements along with single target presentations and observed low to moderate RT correlation (0.4–0.6). Sailer et al. (2000) studied the eye-hand RT correlation in six different tasks and reported RT correlation of 0.3–0.7, with the lowest RT correlation in the pro-saccade gap condition, where the fixation spot disappeared 200 ms prior to target onset (Fischer and Rogal 1986; Frens and Erkelens 1991). Interestingly, they observed that RT correlation was highest in the anti-saccade gap task (0.7). Such high correlations have also been reported by others. Herman et al. (1981) asked subjects to look and use a stylus to touch targets displayed on a screen and found high RT correlations (∼0.8). Similarly, Fischer and Rogal (1986) observed high RT correlations (up to 0.95) in a saccade and reach task. While it is not entirely clear what aspect of the tasks varied which could give rise to this range of RT correlation, it illustrates the effect of task context on eye-hand coordination. To explain the range of RT correlations observed, Frens and Erkelens (1991) and Sailer et al. (2000) have suggested that the temporal coupling between eye and hand movements may be lower for reflexive movements and higher for intentional movements. It is possible that certain tasks make the movement of one effector more reflexive, which leads to movements being initiated by separate neural commands, resulting in low RT correlations. Our study provides a unifying computational framework where the task context can bias the behavior such that it follows the predictions of the common accumulator model (high-RT correlation) or separate accumulator model (low-RT correlation).

Extending the Common Accumulator Model to an Explicit Decision-Making Task

Previous work from the laboratory has suggested that the common accumulator model could account for eye and hand RT distributions in a saccade and reach task (Gopal et al. 2015) and a redirect task (Gopal and Murthy 2015). Here we extend our previous work and show that the common accumulator model is valid in a signal discrimination context as well. Interestingly, the observed RT correlations in the search task were higher than that reported by Gopal et al. (2015) who found RT correlations of ∼0.6. Thus it seems that the additional decisional component of the task led to stronger temporal correlation between eye and hand RT. The increase in RT correlation is also reflected in the smaller difference between the mean eye and hand RT in the coordinated condition reported here (∼88 ms) than that reported by Gopal et al. (2015) (∼98 ms). Such strong RT correlations (up to 0.95) have also been reported by Fischer and Rogal (1986) and by Herman et al. (1981). However, the observation of comparable SD of eye and hand RT distributions was unique to the study by Gopal et al. (2015), which has also been leveraged in the present study. In short, both the predictions of the common accumulator model, strong RT correlations and comparable SDs of eye and hand RT distributions, were observed in the visual search task, validating the applicability of the common accumulator model in this task. The applicability of the common accumulator model in the search task was also confirmed using simulations of a drift-diffusion model. Two-choice decisional tasks like the visual search task are usually modeled as a race between two diffusion processes or as a single diffusion process with two boundaries (Bogacz et al. 2006; Ratcliff 1980; Ratcliff et al. 2016). However, in this study, a diffusion model of one-choice RT (Ratcliff and Van Dongen 2011) was used, fitting only the correct trials. This allowed the comparison between the results of the search and the dual task, which was the main purpose of the study.

The search task also allowed the study of the nature of computations involved in the accumulation process. In an intentional framework, decision-making and motor preparation are inextricably linked (Gold and Shadlen 2000). However, it is conceivable to imagine that perceptual decisions may not be linked to sensorimotor circuits (Filimon et al. 2013). Many actions may also derive from decisions that have occurred in the past, suggesting the separation of motor preparation from decision-making. A general consensus is that at least two decisions have to be made before a movement can be executed: “where to go” or the stage of target selection, and “when to go” or the stage of movement preparation (Herwig et al. 2001; Reddi 2001; Schall and Thompson 1999). Putative neural signatures of these two stages for eye and hand have been reported in numerous brain regions, including frontal eye field (Hanes and Schall 1996; Murthy et al. 2001; Thompson et al. 1996, 1997), dorsal premotor cortex (Crammond and Kalaska 2000; Song and McPeek 2010), lateral interparietal area (Roitman and Shadlen 2002; Shadlen and Newsome 2001), and superior colliculus (Dorris et al. 1997; Horwitz and Newsome 1999). Nevertheless, a central question that remains unresolved is how these two stages are coupled together. In the context of the current work, two decision models can be contrasted: one where the strength of the sensory evidence (representing the difference between the colors of the target and distractor) does not change with the increase in task difficulty, instead the target discrimination time gets delayed; and another where the strength of the sensory evidence changes depending on the task difficulty. Thus an accumulation process which integrates the sensory evidence would lead to increased mean RT with an increase in the level of difficulty in both the conditions, but only in the case of the second model would the SD of the RT distribution increase with task difficulty. Since the SD of the RT distributions increased with task difficulty, we suggest that some aspect of the decision is reflected in the accumulation process.

Limitations of the Common Accumulator Model

In the simulations of the common accumulator model, the eye-coordinated RT was used to model the common accumulator. While the eye RT serves as a good proxy for the finishing time of the common accumulator, this overestimates the finishing time by ∼30 ms (Gopal et al. 2015; Hanes and Schall 1996). This is because the eye RT is cumulative of the time that the common accumulator takes to reach the threshold and the time taken for the activation of the eye muscles which leads to a saccade. Since access to ocular muscles was not readily available, the eye RT served as a proxy. This shortcoming is reflected in the relation between the observed delay (time interval between the EMG and hand onset) and predicted delay (μ_Delay obtained during fitting the common accumulator model) which showed an underestimation of ∼40 ms, consistent with the idea that eye RT is merely an estimate of the end of accumulation.

Potentially, the time when the hand EMG activation began can be considered as a marker of when the common signal reaches the two effectors. One conceivable drawback of this is that EMG signals are noisier than eye movement signals, and hence detection of EMG onsets might not be as accurate as saccade onsets. An additional aspect to consider is the relationship between central processing that reflects the time it takes to accumulate activity to some threshold and the onset of the EMG activity. Although in the common accumulator model the electromechanical delay is implicitly modeled as a serial stage, beginning once the accumulation reaches threshold (Gopal et al. 2015; Gopal and Murthy 2015), these two stages may be cascaded or coupled in a more complex manner that is yet to be elucidated (Cisek et al. 2009; Van Acker et al. 2016). In the absence of simultaneous recordings between neural activity and motor unit activation, this aspect of the model awaits physiological verification. Highlighting this issue is the observation that little or no correlation was observed between the predicted and observed delay, suggesting a more complex relationship between the end of accumulation and the electromechanical delay. Nevertheless, other aspects of the delay signal appear to be compatible with the prediction of the common accumulator model. For example, it is expected that the temporal delay in overcoming the inertia should decorrelate eye and hand onsets. Thus the highest correlation was observed between hand and EMG onsets, followed by eye and EMG onsets, and then by eye and hand onsets. This was observed in both the easy search and difficult search conditions. This potential limitation of using the eye RT as a proxy to estimate the finishing time of the common accumulation process is also likely to be the reason for over-prediction of the RT correlation observed in the search task.

Modeling Eye-Hand Coordination using the Separate Accumulator Model

There can potentially be two modes of operation for the two effectors: either they can be initiated by a common neural command, or they can be initiated by separate neural commands. These two architectures have been compared in the present study. In contrast to the common accumulator model, which is mostly consistent with the data in the search task, the separate accumulator model predicts low RT correlations and scaling of SDs with its corresponding mean RTs. The results from the dual task matched these predictions. Nevertheless, the eye and hand accumulators were not strictly independent, but instead interacted differentially depending on which effector was chosen as the primary task. Such interactions manifest their effect as changes in RT between the alone and coordinated conditions. Such differential patterns of interactions has also been observed by numerous studies (Bekkering et al. 1994, 1996; Gopal et al. 2015; Gribble et al. 2002; Lunenburger et al. 2000; Mather and Fisk 1985). In the eye block condition, the mean eye RT became slower, while the hand RT did not show any systematic bias at the population level, compared with when the two effectors moved independently. In the hand block condition, the mean eye RT became slower, and the mean hand RT became faster compared with the alone condition. This is similar to what has been observed by Mather and Fisk (1985), Bekkering et al. (1994), and Gopal et al. (2015) and opposite to what was observed by Lunenburger et al. (2000). Thus the differential interactions observed between the eye block and hand block conditions could be due to a subject's strategy of choosing which effector is considered as the primary movement. Differential interactions could also arise because the eye-alone and hand-alone RT used to model the eye and hand accumulators in the separate accumulator model stemmed from two separate blocks of trials (conducted in the same session for each participant). Hence some part of the inconsistency in the shifts in RT between the alone and coordinated condition could be attributed to the differential motor set or strategy used by a subject in each of these blocks. Potentially a better task design would be where eye-alone, hand-alone, coordinated trials with negative and positive SOA, i.e., eye before hand and hand before eye, respectively, were randomly intermixed in a session. This design was avoided as it became confusing for the subjects, and their performance dropped considerably. Nevertheless, it is important to note that, although there are differences in the interaction between the eye and hand blocks, there is a concomitant scaling of the respective SDs and low RT correlation, which is consistent with the separate accumulator model.

The frame-work of using effector-specific accumulators for initiating eye and hand movements is an attractive one, as it provides flexibility. A previous study by Dean et al. (2011) recorded eye and hand movements of nonhuman primates using a dual-task context (similar to the one used in this study) and suggested that two effector-specific, mutually excitable accumulators could predict the behavioral results. On the other hand, Gopal et al. (2015) noted that this model failed to explain the SD of the RT distributions and the RT correlation in a saccade and reach task. Although the specific details of the fitting procedures and the parameters used between the two studies were different, in essence they were similar, as both used two interacting diffusion processes which led to movement of an effector when it reached the threshold. Using the same fitting procedure as reported by Gopal et al. (2015), we found that this architecture could potentially explain the RT distribution in the case of temporally decoupled eye and hand movements. One noteworthy difference in the simulations of the separate accumulator model between the study by Gopal et al. (2015) and the present study was that the former study used specific directions of interaction between the accumulators based on the shifts in RT between the coordinated and alone conditions. Hence, they used a positive interaction from the eye accumulator onto the hand accumulator, and a negative interaction from the hand accumulator onto the eye accumulator. For the present study, the interaction terms were left unconstrained and were allowed to take either a positive or negative value. In addition, the fitting procedure employed by Dean et al. (2011) varied with that employed in the present study. Dean et al. (2011) had five free parameters in their model which were optimized concurrently. However, the approach undertaken here was sequential, where the μ_GO and σ_GO of the eye-alone and hand-alone RT distributions were first optimized, followed by optimization of interaction parameters between the two accumulators. This sequential approach was a conservative one which reduced the ability of the separate accumulator model to fit the data, but it also reduced the chances of generating multiple solutions. However, in spite of the differences in the fitting procedure, we demonstrate that the separate accumulator model fitted the data in the dual task better than the common accumulator model.

For the simulations of the separate accumulator model, completely correlated inputs were considered. It is possible that the correlation between the inputs to the eye and hand accumulator was in between 0 (which results in RT correlation close to 0) and 1, and hence the observed and predicted RT correlation was not correlated. Since the extent of correlation of the visual inputs was not directly tested in the simulations, this reasoning is a conjecture at this point in time. Potentially the extent of RT correlation and the extent of the correlation between the inputs could be used as additional parameters in the estimation process, but this was avoided as it increases the chances of generating multiple solutions. Nonetheless, the simulations allowed us to reveal an interesting double dissociation where the common accumulator model, but not the separate accumulator model, could explain the behavior in the search task, while the separate accumulator model, but not the common accumulator model, could explain the behavior in the dual task.

The use of effector-specific accumulators in the dual task may seem intuitive, as the GO cues of eye and hand were dissociated in time. However, an alternate pattern of responses is possible. Instead of decoupling the two movements, subjects can delay the first response and then chunk the two responses together (Boyd et al. 2009; Shea et al. 2006). Unpublished data from the laboratory has shown that, when the eye and hand target locations are also decoupled in the dual task, subjects often chunk the eye and hand movements together. Additional corroboration of the validity of the separate accumulator model was also observed in a subset of trials in the visual search task in which target selection and motor planning were distinct, as revealed by the dissociations of the initial movement direction of the effectors. Consistent with the predictions of the separate accumulator model, we observed low temporal correlations and greater SDs of hand RT distributions compared with that of the eye RT distribution, indicating the presence of separate accumulators. Thus it must be noted that the different architectures mentioned were not a mere manifestation of the dual task per se. Instead, the task context may bias the brain toward one of the two architectures. Differences between the computational architecture underlying eye-coordination in the dual and search task may reflect the role of a central executive (Seeley et al. 2007; Sridharan et al. 2008), which allows the switch from a common to separate accumulators mode and vice versa. Further studies are required to elucidate the neural basis of both the common and separate accumulators and their modulation by executive control process.

GRANTS

This study was supported by grants from the Department of Science and Technology (IRPHA), Government of India, and a DBT-IISc partnership programme grant and a DBT grant to A. Murthy. S. Jana was supported by a fellowship from the Council of Scientific and Industrial Research, India, and the Indian Institute of Science. A. Gopal was supported by a fellowship from National Brain Research Center, India.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

S.J. and A.M. conceived and designed research; S.J. performed experiments; S.J. analyzed data; S.J., A.G., and A.M. interpreted results of experiments; S.J. prepared figures; S.J. drafted manuscript; S.J., A.G., and A.M. edited and revised manuscript; S.J., A.G., and A.M. approved final version of manuscript.