Visuomotor Associations Facilitate Movement Preparation

Abstract

Many motor skills require rapidly choosing a movement goal and preparing a movement to that goal, such as in sports where circumstances often change quickly and many actions are possible. Humans can benefit from learning the perceptual cues that predict the requirements of movement so that the choice of a movement goal and movement preparation can occur earlier. However, there remains uncertainty about how these perceptual cues are learned. Here we investigate the use and learning of these perceptual-motor associations. First, we ask if episodic memory for associations can support learning. In Experiment 1, participants first memorized associations between symbols and movement goals. When these symbols were subsequently presented as cues, reaching movements were prepared as efficiently as if the goals themselves were previewed, without the need for additional practice. Next, we ask whether statistical learning can be used to learn the associations. In Experiment 2, participants had to learn the associations during the movement task itself. This learning enabled efficient movement preparation, and the rate of improvement scaled with the number and complexity of associations. These findings suggest that movement preparation can be facilitated by perceptual cues via statistical learning and memory recall, highlighting a potential role for learning and memory systems not conventionally implicated in motor behavior.

Consider the problem faced by baseball batters: They have very little time to plan their swing given the speed and distance of pitches, and so must rely on subtle perceptual cues to guide motor preparation, such as the angle of the pitcher’s arm, the hand grip, and the position and spin of the ball (Bahill et al., 2005). This example highlights several characteristics of motor behavior and learning. First, it is common to have uncertainty about the goal of a movement, especially in circumstances when movements need to be executed within a limited window of time. Second, perceptual events in the environment may reveal the goals of movement, such as reaching for a target, if only through an arbitrary association between the event and the goal. How do humans learn to decipher the goal-appropriate movement from informative but arbitrary perceptual cues?

In the laboratory, this kind of learning behavior can be studied with visuomotor association (VMA) tasks, in which human participants must remember or learn how to respond to the appearance of an arbitrary perceptual cue. Typically, a visual cue (e.g., a letter or a shape) is presented to a participant followed by a forced-choice task in which the participant selects one among several targets (Asaad et al., 1998; Murray & Wise, 1996). The visual cue predicts the subsequent appearance of a particular movement target and the participant learns these associations through experience. Learning is usually inferred by observing the participant’s choice accuracy and response time (RT).

Evidence points to hippocampal-dependent associative memory as one mechanism for this form of learning. Damage to the hippocampus impairs learning the cue-target associations in both humans (Petrides, 1997) and nonhuman primates (Brasted et al., 2003; Murray & Wise, 1996). Furthermore, changes in the firing properties of hippocampal neurons occurs across VMA learning in primates (Wirth et al., 2003), and in humans (Mattfeld & Stark, 2015; Stark et al., 2018). It has long been known that the hippocampus is necessary for forming new episodic memories for one-time events (Cohen & Squire, 1980; Eichenbaum, 2000; Scoville & Milner, 1957; Squire, 1986), which could be responsible for VMA learning. Yet, the hippocampus also contributes to other distinct forms of learning and memory. For instance, statistical learning (Hunt & Aslin, 2001; Nissen & Bullemer, 1987) has been shown to be hippocampal-dependent (Covington et al., 2018; Schapiro et al., 2014), and to engage specific hippocampal pathways (Schapiro et al., 2012) in a process that is unique from the role of the hippocampus in episodic memory (Norman & O’Reilly, 2003; Schapiro et al., 2017). Thus, the mechanism of statistical learning may also contribute to VMA learning.

We sought to test whether episodic memory and statistical learning contribute to VMA learning in humans. To do so, we introduce a novel VMA task that controls the level of goal-uncertainty on each trial and enables measuring task performance with high specificity. The task design controls preparation time (PT), which we define to be the time-difference between the appearance of a movement target and the onset of movement. We control the level of goal-uncertainty on each trial by varying the PT, because there is high goal uncertainty when there is little PT (Ghez et al., 1997; Haith et al., 2015). On some trials in this task, a visual cue (which we refer to as a precue) in the form of an arbitrary symbol that informs the participant of which target will appear is presented prior to the target’s appearance. The precue removes goal-uncertainty, provided that the participant has learned the association between the symbol and the target that it signifies. As a positive control, trials are included that show the target itself as the precue (called direct precues), and as a negative control, trials are included that show no precue. We quantified task performance by measuring the kinematics on each trial and computing the direction of movement at the start of each reach. This latter measure allowed us to determine whether reaches were directed toward the correct target, to quantify movement error relative to the target, and also to compute across-trial variability of reach direction. The critical test to determine whether episodic memory or statistical learning contribute to VMA learning was to evaluate whether symbolic precues enable better task performance compared with the no precue condition when the PT was very short. We expect that if symbolic precues are learned and incorporated into movement planning they will enable task performance that matches the task performance of direct precues.

This task design has advantages over other VMA tasks. Speed–accuracy trade-offs can occur in VMA tasks, introducing potential confounds in interpreting performance (Liu & Watanabe, 2012; Pew, 1969; Wood & Jennings, 1976). For instance, different memory mechanisms or cognitive strategies might be responsible for choice behavior when the RT is short compared with long (Dang et al., 2018; McElree et al., 2006; Peebles & Bothell, 2004). Furthermore, the mechanism underlying behavior might change with practice such that higher choice accuracies are possible at shorter RTs (Hardwick et al., 2019). This type of transformation could be obscured if RTs are not carefully controlled, as participants may maintain a longer RT than necessary (Haith et al., 2016), which can happen out of habit (Wong et al., 2017). The task design we used here allowed us to measure response accuracy and movement kinematics as a function of the amount of time available to prepare the response (i.e., PT), and thus avoid the aforementioned challenges inherent in using either RT or choice accuracy alone as a dependent measure.

Here, we present two experiments to test our hypotheses while taking advantage of the benefits of the VMA task and controlling for potential confounds. In Experiment 1, participants memorized the arbitrary associations between symbolic precues and movement goal locations in advance of the movement task. This design allowed us to test if episodic memory for the associations between symbols and targets is used to better prepare movements in the VMA task. We expected that symbolic precue trials will be as effective at facilitating movement preparation in advance of the appearance of the target as direct precue trials. In Experiment 2, the associations between symbols and targets had to be learned through experience during the movement task itself in a design that included cross-situational associations (i.e., multiple symbolic cues were paired to each movement goal locations; Yu & Smith, 2007). This design allowed us to test whether statistical learning was used to acquire the associations. We expected learning to be gradual and to modulate with the number of associations to be learn.

Experiment 1

We introduce a variant of the VMA task to investigate the consequences of memory recall on movement preparation. In this task, visual perceptual cues in the form of symbols (symbolic precues) were paired with movement-target locations and human participants were told to memorize these associations prior to beginning the experiment. For comparison, trials were included that displayed the target itself as a precue (direct precue trials), or that displayed no informative precue (no precue trial). We hypothesized that when symbol-target associations were memorized ahead of time, participants would use knowledge of the associations to prepare movements to the target just as efficiently as during direct precue trials.

Method

Participants

Twenty-one undergraduate students (average age: 19, age range: 18–21, 14 females) were recruited for this experiment. All participants provided informed consent to a protocol approved by the Yale Institutional Review Board. One participant was excluded for poor compliance with the instructions of the task. Power analyses (described in detail in the Statistics section) determined that a sample size of N = 10 would be sufficient to achieve 80% power.

Apparatus

Participants made right upper-limb planar reaching movements while seated at a table. A computer monitor (Dell P2717H) with a 6-ms response latency and a 60-Hz refresh rate was positioned vertically approximately 90–100 cm in front of the participant. A webcam (4 Mega Pixel, 60 frames per second, Kayeton Technology Co., Shenzhen, China) positioned 53 cm above the surface of the table recorded the kinematics of movement at 60 Hz by detecting the location of a colored marking warn on the participant’s hand (Figure 1a). The camera’s resolution at that distance from the table surface was 0.3 mm per pixel. The location of the colored marking was determined by computing the median pixel-location among pixels that contained the unique color of the marking. A cursor (black dot 0.64 cm diameter) representing the position of the participant’s hand was displayed in real-time on the monitor. Given the sampling rate of the camera, the refresh rate of the monitor, and the monitor response latency, a delay of approximately 40 ms was expected between movements of the colored marking and updates of the cursor location. Any such delay in the system was consistent across all conditions and so cannot explain condition-wise differences. However, absolute measures of response latency (and thus PT) was biased by the system delay and introduced noise in the measure of PT for each trial.

Figure 1.

Schematic of Apparatus and Task

Note. (a) The kinematics of planar reaching movements were measured using a digital camera that detected the location of a colored marking on an individual’s hand and displayed that location to the participant in real time (red cursor on the computer screen). (b) The mapping showing which symbol was associated with each target for Experiment 1. The symbols are displayed at the target location only for visualization purposes, they appeared at the center of the screen over the start position during the experiment. (c) Trials consisted of either no precuing, direct precuing, or symbolic precuing. Horizonal bars indicate times in the trial during which the symbol or the target were displayed. Musical notes indicate times in the trial during which auditory tones were presented.

Procedure

Reaches were planar, began at a central location on the table, and were directed toward targets represented as filled gray circles (1.25 cm diameter) on the display. Targets were 11 cm from the start position. Participants were instructed to make ballistic movements to four possible target locations (Figure 1b). Trials began when the cursor was within 2.5 mm of the start position. Participants were trained to initiate their movement coincident with the third auditory tone in a sequence of three tones, “the metronome” (first two tones: 1 kHz monotone of 0.1 s duration; third tone: 1.7 kHz monotone of 0.1 s duration; 0.4 s inter-tone-interval). The target for each trial was presented 0.25–0.80 s prior to the designated movement initiation time (Figure 1c). Varying the target presentation time parameterized the amount of time available to view the location of the movement goal prior to movement initiation; we refer to this time as the PT. Participants were trained to prioritize accurate movement initiation times and to always choose a reach direction, even if the presentation of the target occurred only shortly before the third tone or they were unable to detect it altogether. During a brief training block that consisted of 12 trials with no precue, participants were verbally instructed to initiate their movement coincident with the third tone, and were provided immediate, verbal feedback by an experimenter. Following the initial training block, participants completed the remaining 180 trials alone and without verbal feedback.

The experiment contained three trial conditions (Figure 1c). For no precue trials, a wait period of 3.25 s was imposed prior to the start of the metronome. For direct precue trials, the target was displayed for 0.25 s, followed by a wait period of 3 s. For symbolic precue trials, a symbol (Figure 1b) was displayed at the center of the screen covering the start position for 0.25 s followed by a 3-s wait period. For both direct and symbolic trials, the target that ultimately appeared was consistent with the precue on 90% of trials; when a different target appeared than was initially indicated (a catch trial), it was selected with equal probability among the remaining possible target locations. Trials were divided into blocks, with each block lasting 8.7 min on average. Catch trials did not occur during the first 10 trials of any block.

Participants memorized associations between symbols and target locations before the experiment began. A mapping of the associations (Figure 1b) was shown to the participants on a sheet of paper and they were given as much time as needed to view this mapping (typically less than 1 min) before beginning a practice session of 12 no precue trials. Following the practice block there were three main blocks of 60 trials each. For these blocks, the three trial types were pseudorandomly interleaved such that there was an average of 20 trials of each type, which included catch trials.

Data Analysis

Data were analyzed using MATLAB (The MathWorks, Natick, MA). The PT for each trial was computed as the difference between when the target appeared and the time at which the distance between the start position and the cursor first exceeded 0.5 cm. The reach direction for each trial was computed by measuring the angle of the movement velocity when the reach was 3 cm from the start position. Reaches were labeled as correct when the angular reach was within 30° of the target direction. These kinematic data were used to compute the probability of choosing the correct target (“success probability”), the peak velocity of the reach (“peak velocity”), and the variability of reach-directions toward the chosen target (“directional variability”) within bins of PT 130-ms wide. These measures were used to ascertain the state of movement preparation under the three trial types and in catch trials across PTs and included all trials within the PT bin regardless of whether that trial was directed to the correct target or not.

An individual’s minimum PT was computed according to the formula below (see also Supplemental Figure 1):

$$\mathit{\text{minPT}}=\frac{\text{argmax}}{t}(\text{Pr}(s|P\text{T}>t)+\text{Pr}(u|P\text{T}<t))$$

Here, minPT is the minimum PT, PT is the preparation time, t is the time within the trial, and s and u refer to, respectively, whether the reach on a given trial was directed towards the target (successful) or in a different direction (unsuccessful). This automated determination of minimum PT was manually verified and participants were excluded if the determination was incorrect. This could occur as a result of insufficient sampling across PTs due to participant noncompliance. One participant (5%) was excluded for this reason from Experiment 1. Minimum PT was computed using only trials without precues. The computation of minimum PT is in general more accurate when more trials without precues are included across all PTs.

Statistics

Statistical analyses were conducted in R (https://www.r-project.org). We tested the significance of the relationship of PT, trial type, and their interaction on the measures of movement preparation described above. Linear mixed-effects models were fit for each dependent variable (probability correct, peak velocity, and directional variability) using PT, trial type, and their interaction as fixed effects, and subject as a random effect. The significance of each fixed effect was determined by comparing the log-likelihood of a model that included the fixed effect to a nested model without that effect (Tang et al., 2014). Cohen’s f² (Selya et al., 2012) values were computed for each test and were reported as effect sizes.

Post hoc analyses were conducted for each trial type in the event that an interaction term was significant. For these analyses, linear mixed effects models were fit using subject as a random factor and PT as a fixed factor, and a likelihood ratio test was used to test the significance of PT by comparing the log-likelihood of a model that included PT versus a constant model. Cohen’s f² values were obtained and reported as effect sizes.

An analysis was conducted to determine whether additional PT was required to initiate movements to the presented target during catch trials compared with during no precue trials. For this analysis, PT was transformed to the difference between the measured PT for each trial and a participant’s minimum PT, with positive values indicating more PT was available than minimally necessary, and negative values indicating less PT was available than minimally necessary. Sigmoidal functions were fit to binary data (i.e., success or failure) as a function of PT according to the following equation:

$$p=\alpha +\frac{1-\alpha }{1+e^{-\beta PT+\gamma }}$$

In this model, p is the success probability, α is the vertical offset, β represents a rate of change with PT, and γ represents the horizontal offset. Parameters were found that maximized the log likelihood of the binary data given the model. To determine whether catch trials affected the time it took to initiate a correct response, we tested for a difference in the γ parameter between the no precue condition and catch trials using bootstrap resampling. For each of 1,000 iterations, we sampled participants with replacement and fit the model separately to no precue trials and catch trials. The p value was computed as the fraction of iterations on which the mean difference in the γ parameter between the no precue condition and catch trials was less than zero.

An analysis of statistical power was conducted to determine the minimum sample size needed to achieve 80% power. The key analysis in this study is a mixed-effects linear model that tests for an interaction between trial type and PT in how those factors relate to task success (described above). Computing the power for this test requires a simulation in order to estimate the distribution under the alternative hypothesis of a likelihood ratio test (Brysbaert & Stevens, 2018). Based on behavior from a previous study that used a similar method of controlling the PT of reaching movements (Huberdeau et al., 2019), we expected that task success would strongly depend on PT for trials without a cue, while trials with an informative cue (either direct or symbolic) would not. We simulated data for a measure of task success, “percent correct,” that assumed percent correct would be at chance (.25 for four possible target locations) for the no precue trial type when PT was less than 0.3 s, and would be .9 for all other PTs and trial types; data were assumed to have a residual standard deviation of 0.1. The analysis of power determined that a sample size of N = 10 would be sufficient. The simulated data had a Cohen’s f² effect size of 1.0.

Results

Reaches (Figure 2a) differed in their success probability across PT and by trial types (Figure 2b). There was a significant main effect of PT (f² = 0.27; χ²(1) = 79, p < .001), a significant main effect of trial type (f² = 0.77; χ²(2) = 160, p < .001), and a significant interaction between PT and trial type (f² = 0.60; χ²(2) = 130, p < .001). A linear mixed effects model fit to data from no precue trials revealed that the success probability had a large and significant dependence on PT (f² = 1.3; χ²(1) = 92, p < .001). In contrast, direct precued trials and symbolic precued trials had much smaller effect sizes relative to the no precue condition, although the effects of PT were significant (direct: f² = 0.032; χ²(1) = 3.9, p = .05; symbolic: f² = 0.036; χ²(1) = 4.3, p = .04). These analyses confirm that the success probability differed by trial type, where reaches with a symbolic or direct precue were similarly successful at preparing the correct reach with only a small dependence of PT, while trials without a precue had a large and significant dependence on PT.

Figure 2.

Symbolic and Direct Precuing Both Facilitate Efficient Movement Preparation in Experiment 1

Note. (a) Reaches of one participant showing typical kinematics. The group average for (b) success probability, (c) peak velocity, and (d) directional variability, as a function of preparation time.

Despite the effects of trial type and PT on success probability, reaches were kinematically similar across PTs and trial types, even when all trials (successes and nonsuccesses) were included in the analysis (Figure 2c). There were no significant main effects of peak velocity (PT: f² = 0.005; χ²(1) = 1.5, p = .23; trial type: f² = 0.004; χ²(2) = 1.2, p = .56), nor an interaction between PT and trial type (f² = 0.004; χ²(2) = 1.1, p = .57).

Directional variability differed across trial types in a similar way to success probability (Figure 2d). There were significant main effects of PT (f² = 0.01; χ²(1) = 4.1, p = .044) and trial type (f² = 0.30; χ²(2) = 76, p < .001), and a significant interaction between PT and trial type (f² = 0.04; χ²(2) = 12, p = .002). Directional variability for no precue trials significantly depended on PT (f² = 0.07; χ²(1) = 8.0, p = .004), while the directional variability of directly cued trials (f² < 0.001; χ²(1) = 0.054, p = .82) and symbolically cued trials (f² = 0.02; χ²(1) = 1.4, p = .23) did not. These results are consistent with our hypothesis that direct precues and memorized arbitrary symbolic precues are equally effective at facilitating movement preparation irrespective of PT. In this case, the state of movement preparation is reflected in the variability in initial reach direction.

Did switching the location of the target relative to the cue on catch trials impede movement to the ultimate appearance of the target? An analysis of reach direction error (Figure 3a) revealed that catch trials (Figure 3b) depended on PT differently than the other trial types (Figure 3c). Sigmoidal curves were fit to success binary data as a function of PT separately for each trial type (Figure 3d). A bootstrap resampling analysis determined that catch trials significantly delayed the time at which reaches were successful, that is, the parameter γ was significantly different between no precue trials and catch trials (p < .001).

Figure 3.

Preparation Time Data Aligned to Each Individual’s Minimum PT in Experiment 1

Note. (a) Reach direction as a function of time from the minimum preparation time (PT) and trial type. (b) Same for catch trials. (c) The probability of making a correct reach as a function of the time from the minimum PT and trial type, including catch trials. (d) Sigmoidal fits to each trial type and to catch trials.

Discussion

The results from Experiment 1 support our hypothesis that episodic memory for the associations between the symbols and targets supports VMA learning in humans. When no precue was given and the PT was below an individual’s minimum PT, this constituted a “go before you know” design (Chapman et al., 2010; Ghez et al., 1997; Haith et al., 2015). Trials with no precue thus had a strong and significant effect of PT on success probability and movement variability, while trials with either direct or symbolic precues had much weaker reliance on PT, providing conclusive evidence that movements were planned in advance.

The task design avoided potential confounds in evaluating performance that can be present when using RT or task success alone to quantify the state of movement preparation. Movement goal uncertainty has been shown to delay RT (Scherbaum et al., 2010; Simon, 1969) and increase across-trial movement variability (Krüger & Hermsdörfer, 2019; Song & Nakayama, 2009), and VMA tasks are subject to a speed–accuracy trade-off that can bias results (Liu & Watanabe, 2012; Pew, 1969; Wood & Jennings, 1976). To quantify movement preparation after controlling for RT, we measured success probability and movement variability. Movement preparation according to these measures was the same for direct precues and symbolic precues, and both conditions were superior to the no precue condition. The kinematics of the actual movements were otherwise similar across all conditions. Any effect of PT on success probability or movement variability was small when either type of precue was given. In contrast, when no precue was given, additional PT was needed to prepare reaches to the correct target.

An additional feature of Experiment 1 was that 10% of trials were catch trials, where a different target appeared than had been cued. Similar experimental designs, in which a change in movement plan is induced through a change in movement goals, have been used to study movement preparation (Ames et al., 2014, 2019; Haith et al., 2015; Selen et al., 2012). In our study, catch trials significantly delayed the minimum time that was necessary to successfully reach toward the presented target. This further demonstrates that precues resulted in movements being planned toward the cued target, as switching to an unexpected target incurred a cost of additional time to prepare a successful reach.

These findings confirmed our hypotheses that recently memorized associations between arbitrary symbols and targets can facilitate movement preparation equally well as direct cues, even without practice or additional learning. This highlights the importance of explicit memory for motor skill learning, as efficient movement preparation in advance of the appearance of movement goals is a critical component to many real-world skills (Bahill et al., 2005).

Experiment 2

Associations between environmental cues and movement goals may not always be deterministic or easy to memorize, as in Experiment 1. Instead, these associations may need to be learned through experience, which may rely on a number of possible learning mechanisms. Prior studies have suggested that hippocampal-dependent associative memory may be important for learning (Mattfeld & Stark, 2015; Murray & Wise, 1996; Stark et al., 2018). Here, we hypothesized that another hippocampal-dependent mechanism, statistical learning (Aslin & Newport, 2012; Frank et al., 2007; Yu & Smith, 2007), may be used to learn the associations. Experiment 2 was designed to test the possibility that statistical learning contributes to VMA task performance. Statistical learning is typically slower than associative memory, so we hypothesized that learning in this task would gradually reveal facilitation of movements by symbols, and that varying the number of associations to be learned would modulate the learning rate.

Method

Participants

Participants were drawn from the same population as in Experiment 1. We recruited 74 individuals for Experiment 2 in total, with 57 participants randomly assigned to one of three initial groups (described below in the Procedure section) who completed different variants of the experiment and 17 additional participants assigned to a fourth replication sample. Five participants in total were excluded for poor compliance with task instructions. Power analyses similar to Experiment 1 (described below in the Statistics section) determined that a sample size of N = 13 per group would be sufficient to achieve 80% power.

Procedure

The same apparatus as Experiment 1 was used to display instructions, stimuli, and targets, and to collect movement kinematics. Unlike Experiment 1, participants were not shown the symbol-target associations before the start of the task. They had to acquire these associations via cross-situational statistical learning, where each target was associated with more than one symbol across trials. That is, participants had to learn that different shapes predicted the same target location. The first block served as a practice session with 60 trials of the no precue condition. These trials without a precue were used to compute each individual’s minimum PT. The next four main blocks contained 60 trials, with 27 trials each of direct and symbolic trial types, as well as three catch trials of each type. A posttest memory assessment for the symbol-target associations was given at the end of the experiment, and was scored by taking the ratio of the number of symbols that were correctly paired to a target to the number of symbols that were incorrectly paired to a target or not paired but should have been.

Three variants of Experiment 2 manipulated the complexity of learning in different participants (all naïve): three targets with two symbols each (N = 21, ages 18–21, 16 female), four targets with three symbols each (N = 19, ages 18–23, 13 female), and six targets with two symbols each (N = 13, ages 18–21, eight female). Symbols differed across these variants, including in visual complexity from simple geometric shapes to arbitrary symbols and amorphous shapes (see Figure 4). A near-replication of the three-target variant (N = 16, ages 18–25, nine female) was conducted as a fourth group after the data above had been collected and analyzed, with the only difference being the addition of a memory test for the associations after the first block of symbol-target association learning.

Figure 4.

Visuomotor Associations and Example Kinematics for Experiment 2

Note. Target locations (gray circles) and symbol mappings (presented in periphery for visualization, always displayed at center) for the variants of experiment 2 with (a) three targets and six symbols, (c) four targets and 12 symbols, and (e) six targets and 12 symbols (b, d, f). Reaches for one participant in each variant showing example kinematics.

Data Analysis

The probability of making a correct reach (“success probability”) among reaches with a PT less than the individual’s minimum PT was computed for each half-block, which comprised 30 trials. Reaches with PTs greater than an individual’s minimum PT were excluded from this analysis to isolate trials that must necessarily have depended on learned associations. That is, reaches with sufficient PT could have been successful even if the association between the symbols and targets was unknown because participants could have responded to the appearance of the target rather than preemptively preparing a movement to the target in response to the precue. Thus, excluding trials with high-PT provides a more accurate estimate of the ability of participants to use associations for movement preparation across the experiment. On average, there were 13 trials that had a PT lower than the individual’s minimum PT for each set of 30 trials.

Experiment 2 additionally allowed us to track learning of the symbol-target associations. The probability of reach success when a reach was conducted with a PT less than an individual’s minimum PT was computed as a function of the number of times that a given cue had appeared in the experiment. This analysis was performed separately for symbolic and direct precue trials. As a comparison and baseline, we repeated this analysis on data from Experiment 1, where learning across symbol-appearances was not expected. Exponential functions of the form:

$$y=\beta \left(1-e^{-\alpha t}\right)$$

were fit to success probability data, where y was taken to be the success probability, t the number of occurrences of a symbol, β the offset, and α the rate of learning. These parameters were obtained by finding those parameters that maximized the log likelihood of the binary (success/failure) data given the model. Their significance was tested using a Wald test.

Statistics

To determine whether success probability changed across practice, a linear mixed effects model was fit to success probability using block number, trial type, and their interaction as fixed effects, and subject as a random effect. The significance of each fixed effect was tested using a likelihood ratio test, and Cohen’s f² was used as a standardized effect size, as in Experiment 1. This analysis was completed for each variant of Experiment 2. Additional analyses were conducted to test whether success probability changes across blocks for symbolic trials and direct trials individually. Paired t tests were conducted to test whether the explicit memory score differed from the success probability and whether the explicit memory score differed from chance. Cohen’s d was computed as the effect size for these tests. The use of linear models for probabilities of this type remains a topic of study (Ferrari & Comelli, 2016). Even when assumptions of normality are violated, however, Ferrari and Comelli (2016) found that statistical results are typically robust.

An analysis of statistical power was conducted to determine the minimum sample size needed to detect an effect of block on success probability with 80% power. We assumed that the success probability increased linearly from chance (i.e., .33 for the three-target version) to .9 across eight blocks of practice, and that the residual standard deviation was 0.2. This analysis used the same procedure as the power analysis for Experiment 1 and indicated that a minimum of 13 participants were needed. The effect size for these simulated data was f² = 0.7.

An across-group analysis was performed to determine if the learning rate of success probability differed across groups. For this analysis, a linear model was fit for success probability as a function of block number for each individual in each group. An analysis of variance was conducted to test whether the rate parameter differed across groups, and η² was reported as the effect size for this test.

Exponential functions were fit to the success probability data as a function of symbol-occurrence, as described above. The significance of each parameter of the exponential fits was tested using Wald tests. The values of the rate and offset parameter are reported.

Catch trials were analyzed for Experiment 2 in a similar way to Experiment 1, except with an additional factor for trial number to account for potential learning effects across practice. The interaction between trial type, PT, and trial number was tested using a likelihood ratio test, as in Experiment 1. When significant, an analysis was performed to test whether catch trials differed significantly in their reliance on PT from no precue trials using a likelihood ratio test.

Across Experiments 1 and 2, five independent groups of participants prepared reaches to a target with varying numbers of potential alternatives. This design allowed us to test for any relationship between the number of choice alternatives and the minimum time required to prepare a successful movement (minimum PT) when no precue was given. We fit a linear model with the number of potential targets as the independent variable and the minimum PT as the dependent variable. We compared this model to an intercept-only model using the Bayes factor. The Bayes factor is approximated by the following formula (Wagenmakers, 2007):

$$BF=e^{\frac{BIC(\mathit{\text{model\hspace{0.17em}}}1)-BIC(\mathit{\text{model\hspace{0.17em}}}2)}{2}}$$

where BIC is the Bayesian information criterion for a given model. This approximation assumes uniform priors for each model (Wagenmakers, 2007).

Results

When two symbols were paired with each of three target locations (Figure 4a), reaches (Figure 4b) with PT lower than an individual’s minimum PT had a success probability that increased across blocks (Figure 5a). This was confirmed by linear mixed effects models that fit success probability as a function of block, trial type, and their interaction (see Table 1). Critically, there was a significant interaction between block and trial type. The effect of block number on success probability was significant for the symbolic trial type, but not for the direct trial type. The replication of this experiment (Figure 5b) revealed similar results: a significant main effect of trial type and a significant interaction between block and trial type, with a significant main effect of block for only the symbolic trial type. Thus, in both the original study and the replication study, there was evidence of learning across trials for the symbolic precue condition.

Figure 5.

Probability of Making a Correct Reach Among Movements With a PT Below the Minimum PT in Experiment 2

Note. Black squares indicate the group average memory test score. Results for (a) the three-target variant, (b) the three-target replication, (c) the four-target variant, and (d) the six-target variant. PT = preparation time.

Table 1.

Results of Mixed-Effects Models for Experiment 2

Experiment version	Factor	Effect size (f2)	Chi-squared (all DOF = 1)	p-value
Three-target	Interaction	0.05	13	<.001
	Block	0.03	9.0	.003
	Trial type	0.21	57	<.001
Direct trial type	Direct: block	0.002	0.35	.55
Symbolic trial type	Symbolic: block	0.13	17	<.001
Three-target replication	Interaction	0.05	11	<.001
	Block	0.01	3.2	.07
	Trial type	0.21	45	<.001
Direct trial type	Direct: block	0.02	2.1	.14
Symbolic trial type	Symbolic: block	0.09	9.4	.002
Four-target	Interaction	0.04	13	.001
	Block	0.09	25	<.001
	Trial type	0.76	160	<.001
Direct trial type	Direct: block	0.03	4.1	.04
Symbolic trial type	Symbolic: block	0.21	24	<.001
Six-target	Interaction	0.004	0.96	.33
	Block	0.01	2.0	.16
	Trial type	2.7	270	<.001

Note. Direct trial type and symbolic trial type rows under each experiment version refer to post-hoc analyses that test the effect of block on each trial type individually. DOF = degrees of freedom.

The variant with four targets and three symbols per target (Figures 4c–d) exhibited the same pattern of results (Figure 5c). There was a significant interaction between block and trial type, and success probability depended significantly on block for the symbolic trial type. There was a small but significant block-effect for the direct trial type, suggesting a practice effect. In the variant with six targets and two symbols per target (Figures 4e–f), there was no evidence of learning (Figure 5d): no main effect of block and no interaction between block and trial type, though there was a significant main effect of trial type. There was also a significant difference in learning rate for the symbolic trial type across the four groups (η² = .14), F(2, 67) = 5.6, p = .005.

We also examined success probability as a continuous function of the total number of times a symbol had been previously encountered by the participant across experiments (see Figure 6). Exponential functions were fit to binary success data in all experiments and groups. There were significant effects in the three-target variant (β = 0.73, z = 8.1, p < .001; α = 0.80, z = 79, p < .001) and its replication (β = 0.73, z = 6.0, p < .001; α = 0.82, z = 70, p < .001). Similar effects were found for the four-target variant (β = 0.55, z = 6.6, p < .001; α = 0.66, z = 30, p < .001). The six-target variant also had a significant learning rate and had an asymptote that was significantly different from zero, although it was still low, at 0.30 (β = 1.2, z = 3.4, p < .001; α = 0.30, z = 20, p < .001).

Figure 6.

Progression of Learning Individual Associations

Note. The success probability on symbolic precued trials (blue) when the movement preparation time (PT) was below an individual’s minimum PT, as a function of the cumulative number of exposures to a symbol in (a) experiment 1, and in the (b) three-target, (c) three-target replication, (d) four-target, and (e) six-target variants of experiment 2. For comparison, analogous curves for direct precue trials (green) are shown as a function of target occurrences.

Explicit memory for the learned associations was tested through a symbol-target matching quiz. In all three variants of Experiment 2, the mean test accuracy closely matched the probability of making a correct reach when precued with a symbol (Figure 5; three-target variant: d = 0.31; t(19) = 1.4, p = .17; four-target variant: d = 0.006; t(17) = 0.54, p = .59; six-target variant: d = 0.33; t(12) = −1.2, p = .24). In the replication study of the three-target variant (Figure 5b), which contained a memory test after the first block and at the end of the experiment, again the test accuracy closely matched reaching performance at the corresponding block (test following first block: d = 0.18; t(14) = 0.66, p = .52; test following final block: d = 0.20; t(15) = 0.78, p = .45). Instead, test accuracy was significantly higher than chance accuracy for the three-target (d = 2.2; t(20) = 10; p < .001) and four-target (d = 1.4; t(18) = 6.1; p < .001) variants, and for the three-target replication (first block: d = 1.9; t(14) = 7.5; p < .001; final block: d = 2.4; t(15) = 9.5; p < .001), but not for the six-target variant (d = 0.41; t(12) = 1.5; p = .17).

Catch trials, in which a different target appeared than had been cued, were analyzed to test whether receiving an incorrect cue delayed the onset of successful movements. For the three-target condition (f² = 0.01; χ²(7) = 42; p < .001), the four-target condition (f² = 0.01; χ²(7) = 40; p < .001), and the six-target condition (f² = 0.04; χ²(7) = 94; p < .001) there was a significant interaction between trial number, trial type, and PT; the three-target replication group had a small effect that was marginally significant (f² = 0.005; χ²(7) = 13; p = .08). For each group, there was also a significant difference between catch trials and no precue trials (three-target group: f² = 0.01; χ²(1) = 25; p < .001; three-target replication: f² = 0.02; χ²(1) = 32; p < .001; four-target group: f² = 0.02; χ²(7) = 23; p < .001; six-target group: f² = 0.03; χ²(7) = 27; p < .001), replicating the findings from Experiment 1.

As a secondary analysis, we sought to determine whether minimum PT differed depending on the number of choice alternatives, as predicted by Hick’s law (Hick, 1952). A linear model was fit to the minimum PT as a function of the number of targets across groups (see Figure 7). This model was compared with an intercept-only model by computing the relative evidence for each model using a Bayes factor. This analysis showed positive evidence for the intercept-only model over the linear model (Bayes factor = 4.5), inconsistent with Hick’s law.

Figure 7.

The Minimum PT of Reaches With No Precues as a Function of the Number of Alternative Potential Targets

Note. Black squares (with error bars representing the standard error) indicate the group average for an individual experiment or variant, and black dots indicate individual participants. PT = preparation time.

Discussion

Experiment 2 investigated the mechanism involved in learning VMAs when no prior knowledge of the associations was provided. We hypothesized that statistical learning would be used to acquire knowledge about these associations. We tested this hypothesis using a task design that included cross-situational associations in which multiple symbols were paired with each movement target and by measuring task success across trials.

Success probability significantly changed across blocks in all but the six-target condition and task success mirrored explicit memory for the symbol-target associations. The fact that the memory test tracked performance in the VMA task suggests that learning was mnemonic in nature. Past experiments that used a cross-situational learning design, as was done in this experiment, have suggested that humans may utilize a propose-but-verify technique to acquire knowledge about associations with multiple associates (Berens et al., 2018; Trueswell et al., 2013). Such a mechanism would be consistent with the data in the present cross-situational learning design because the associations were explicitly retrieved.

The number of symbols associated with each target and the number of potential targets modulated the rate of learning. This pattern is consistent with a statistical learning mechanism, as the rate of learning in a cross-situational learning context is expected to be modulated with the number of possible associates (Blythe et al., 2010). Participants exhibited a learning curve for the association of each symbol with its affiliated target that was well-fit by an exponential function. These learning dynamics led to a gradual improvement in the success probability across trials of practice and are also consistent with learning mechanisms that are partially reliant on associative memory (Collins & Frank, 2012).

An unexpected finding from this experiment was that the minimum PT when no precue was given did not vary by the number of alternative target choices. This finding violates expectations based on Hick’s law—the observation that reaction time (RT) scales as a function of the number of alternative choices (Christina et al., 1982; Fischman, 1984; Henry & Rogers, 1960). An important methodological difference between the experiments presented here and previous findings is the use of the timed-response method, which forced responses across varying response times (i.e., the time between the presentation of a target and the actual movement onset). We expect that had the experiment allowed RT to vary freely, RT would have scaled with the number of potential targets when no precue was given. This finding suggests that the origins of Hick’s law resides in deliberative processes, and not motor preparation.

General Discussion

Prior studies examined how actions are planned when prompted with an arbitrary visual cue (Asaad et al., 1998; Murray & Wise, 1996; Wirth et al., 2003), but the mechanisms of early VMA learning have not been characterized previously. We hypothesized that memorized associations would require no additional learning to be successfully deployed in a reaching task. An analysis of the time course of performance across trials in Experiment 1, where the VMAs were instructed and memorized ahead of time, revealed that there were no changes across practice. This demonstrates that the translation of known perceptual cues into movement goals does not require practice. Further, Experiment 2 demonstrated that the process of acquiring the cue-target association requires practice, is well modeled by an exponential process, and also appears mnemonic in nature given the correspondence between reports of explicit knowledge of the cue-target associations with movement success. These results give credence to the idea that motor learning is a cognitive activity (Stanley & Krakauer, 2013).

What mechanisms account for learning visuomotor associations over time? We found exponential dynamics of learning, consistent with a previous VMA study in nonhuman primates (Brasted et al., 2003). In that study, transection of the fornix, and thus deactivation of hippocampal contributions to recall, impaired VMA learning, suggesting a role for the hippocampus (see also Murray & Wise, 1996). Indeed, both the acquisition and expression of associations from cross-situational learning and statistical learning have been linked to the hippocampus (Berens et al., 2018; Covington et al., 2018; Eichenbaum, 2000; Schapiro et al., 2014; Wirth et al., 2003). This suggests that the hippocampus may be important for linking and recalling a movement goal for an arbitrarily perceptual cue.

Although convention holds that motor learning is independent of the hippocampus (Corkin, 1968; Milner, 1962), more recent research has suggested otherwise (Roy & Park, 2010; Stanley & Krakauer, 2013). Hippocampal-dependent processes might be necessary in motor learning through its capacity to memorize and learn associations, and to facilitate cortical processing (Kok et al., 2013; Kok & Turk-Browne, 2018). This model of hippocampally mediated movement planning could underlie many motor behavior phenomena, such as evidence that memory recall enables faster adaptation to visuomotor perturbations (Huberdeau et al., 2015) and contributes to motor sequence learning (Wong et al., 2015). A potentially fruitful direction for future research using this paradigm and conceptual model could be to examine brain activity changes in the hippocampus and other brain areas such as motor cortex across learning of visuomotor associations.

The VMA task that we used involved only ballistic arm reaches, whereas many human motor skills, including the opening example of baseball batters, involve continuous and complex movements. If movement is ongoing, how does visual information update the unfolding action? Some authors have addressed this question with tasks in which a stream of perceptual information is updated continuously (Selen et al., 2012). However, the learning and use of arbitrary perceptual cues has not been studied for continuous movements.

We showed in Experiment 1 that having knowledge of the associations resulted in an immediate benefit to performance in terms of the accuracy of action selection. Although accuracy improved more gradually over time in Experiment 2, this learning was still consistent with a mnemonic process: The probability of selecting the correct response was no different when tested with an explicit memory test versus with the VMA task. This suggests that once the associations were learned, they were accessible to explicit knowledge, an effect that has been shown in other statistical learning paradigms (Batterink et al., 2015). A remaining question, however, is whether the associations in Experiment 2 could have been encoded entirely with episodic memory, without the need for statistical learning through repetition. Although the associations in Experiment 2 were more complex because of the larger number of symbols, the similarity of symbols, and the many-to-one mapping of symbols to locations, it nevertheless might have been possible for participants to encode the associations if they had been instructed explicitly. This could be investigated in future research to help disentangle the relative roles and boundary conditions of episodic memory and statistical learning in movement preparation.

The experiments presented here included a delay between the presentation of the precue and the appearance of the target. This delay permitted a comparison of trials that had no precue against those with a precue so as to precisely measure the contribution of the precue to movement preparation. This method, however, was not designed to evaluate the relative speeds of applying knowledge of the precues to movement preparation. This latter aspect of how arbitrary perceptual information is translated to movement remains to be explored, especially through the timed response method, which enables sampling movements across a continuum of preparation times (Haith et al., 2016).

Our results raise the question of what aspects of practice benefit motor performance. Motor behaviors outside of the laboratory can be more complex than a simple reaching task, and thus may require a qualitatively distinct learning mechanism to reduce movement execution variability (Shmuelof et al., 2014). Moreover, the environmental cues that predict movement goals in natural settings can also be more complex to decipher than the relatively simple stimuli used in the experiments presented here. In our study, stimulus complexity appeared to be a rate-limiting factor for learning, as the four-target and six-target conditions of Experiment 2 had the same total number of associated pairs to learn, but less learning was observed for the more visually similar shapes from the six-target condition. This suggests that identifying the unique identities of cues and the movement goals they denote are key elements to learning what actions are most appropriate. Significant practice may thus be needed to learn associations between perceptual cues and movement goals in the real-world, given that cues may be much more complex (e.g., multimodal, dynamic) and associated with multiple goals in a context-dependent manner.

Public Significance Statement.

What do we learn when we learn a new skill? The most obvious answer is that we learn how to generate movements that were previously unfamiliar to us. A less obvious but critical answer, investigated in this study, is that we learn the perceptual events that occur before and during the execution of a skill, and then use those events to better prepare for upcoming movements.