Taking Aim at the Cognitive Side of Learning in Sensorimotor Adaptation Tasks

Abstract

Sensorimotor adaptation tasks have been used to characterize processes responsible for calibrating the mapping between desired outcomes and motor commands. Research has focused on how this form of error-based learning occurs in an implicit and automatic manner. However, recent work has revealed the operation of multiple learning processes, even in this simple form of learning. This review focuses on the contribution of cognitive strategies and heuristics to sensorimotor learning, and how these processes enable humans to rapidly explore and evaluate novel solutions to enable flexible, goal-oriented behavior. This new work points to limitations in current computational models, and how these must be updated to describe the conjoint impact of multiple processes in sensorimotor learning.

The Versatility of Human Motor Control

Flexible use of the upper limbs is fundamental to our species. The ability to manipulate objects with our hands, coupled with an expanding capacity to plan future states, was critical to our ancestors’ survival [1]. Dexterous arm movements confer a tremendous advantage for efficiently harvesting foods in varied environments, as well as for manufacturing and manipulating tools. Indeed, Darwin argued that humans’ use of thrown projectiles may have been an adaptation brought about by the pressure to hunt, and suggested that this distinctive behavior may be linked to the emergence of bipedalism [2,3]. Though other primates have occasionally been shown to perform analogous upper limb behaviors, these actions are rarely observed and lack much of the precision of human throwing [4–6].

Many classic studies of sensorimotor learning have been based on reaching and throwing movements, with the results helping us gain fundamental insights into foundational ideas such as the trade-off of speed and accuracy [7–10] and the representation of sensorimotor dynamics [11]. One important subfield of motor learning research employs adaptation tasks to ask how an internal model, a representation of body-environment interactions, is calibrated to support feedback and feedforward control [12]. The internal model concept has provided a useful theoretical tool to understand how people adjust their behavior when moving in atypical force fields or when the visuomotor mapping is altered. These paradigms capture computational problems that enable us to skillfully manipulate objects when dynamics fluctuate (e.g., the changing weight of a bottle as we consume its contents) or when environmental factors require that we adjust our movements (e.g., throwing a frisbee on a windy day). Building on a rich body of neurophysiological and neuropsychological evidence, [13–17] and articulated in sophisticated computational models [18–22], this form of incremental motor learning has provided a fundamental characterization of one important function of the cerebellum.

Models of error-based learning have provided a reasonable approximation of behavior. For example, a simple state space model [18] in which an error signal is used to recalibrate an internal model from trial to trial, captures the general shape of the learning function, one in which performance changes follow a negatively accelerating exponential (or linear in log-log coordinates, [23]). However, these models fail to capture certain features of performance such as spontaneous recovery and savings [20,24]. The inadequacy of these models reflects the complexity of human motor performance: We are flexible, generalist problem-solvers, and, as shown in studies of learning across diverse task domains [25–27], readily employ multiple learning systems to solve the problem at hand. In studies of sensorimotor adaptation, this means that the learner, when presented with an unexpected and salient perturbation, is likely to generate a compensatory strategy or heuristic. Much as the spear fisher adjusts his aim to account for the refraction of light in water, a participant might opt to aim to the side of a target if an opposing force unexpectedly displaces the limb or a visuomotor perturbation results in a large reaching error.

Until recently, strategy use has been considered a nuisance [13] in studies of sensorimotor adaptation, with the experimental instructions often designed to actively discourage this behavior [14,28]. Moreover, the use of heuristics, such as an explicit change in aiming, has been ignored in computational models of the learning process. However, the flexibility of the human motor system allows us to supplement the calibration process. Strategies can allow us to use our planning abilities to rapidly find “good-enough” solutions, ones that might get performance in the right ballpark as the calibration process slowly and subtly homes in on the precise dynamics. In this paper, we review recent developments in studies of sensorimotor adaptation, highlighting work that has provided a richer picture of the operation of multiple learning processes and new insights into how these processes support skilled motor behaviors.

Using Multiple Learning Processes in Response to Sensorimotor Perturbations

The physics of the body and environment are in a continuous state of flux: Not only do long-term changes arise from growth, development, and injury, but, in the short-term, muscles fatigue and sensory conditions fluctuate. The motor system must rapidly adjust to these variable conditions, and the ease with which we maintain calibration belies its computational complexity [29].

To study this calibration process, researchers have employed a variety of learning tasks — including prism adaptation [14,28,30,31], visuomotor rotations [32,33], and force field learning [11] — in which a perturbation is introduced to alter the relationship between a movement and the resulting sensory feedback. Across a range of contexts, performance typically follows a stereotypical learning function (Fig 1A) driven by a gradient descent process in which the error is reduced in a continuous, monotonic manner. When the perturbation is removed a persistent “aftereffect” is observed, taken as the signature of a recalibrated sensorimotor mapping. Over time, the aftereffect diminishes at roughly the same rate as that observed during the initial acquisition phase, eventually returning to the baseline, non-adapted state.

Figure 1. Explicit processes in motor learning.

(A) The canonical human motor learning curve, with the preliminary baseline period (region 1), the learning block where a sensorimotor perturbation is applied (region 2), and a “washout” period where the motor system is re-calibrated back to baseline (region 3). (B) Data from a subset of participants who “cheated” in a prism adaptation study — that is, using an aiming strategy to adjust their behavior after the first perturbed trial (black X). Although this immediately eliminated the error, performance became worse over subsequent trials (red line). When instructed to stop aiming, the error became larger and reversed sign. Simulated data is modeled after reference [28]. (C) After the first perturbation trial in the strategy task (black X), participants are instructed to counter the rotation by aiming towards a landmark displaced from the target. This results in immediate task success. However, performance subsequently deteriorates (“drifts”) due to the operation of an implicit learning process. Simulated data is modeled after reference [34]. (D) If the training period is extended, the error arising from implicit drift is eventually negated by an adjustment in the strategy. An aftereffect, indicative of recalibration, is evident when the rotation is turned off and the participants are told to reach directly to the target. Simulated data is modeled after reference [35].

However, this formulation misses a common-sense approach to the problem participants face in such experiments. While throwing darts one evening, imagine, after donning a pair of prism glasses, that you see a dart land far to the right of the target. It would be reasonable to suppose that an intelligent agent would take steps to volitionally compensate for the perturbation. For example, you might aim to the left of the target on the next trial. Indeed, such compensatory strategies are essential on windy days for golfers and placekickers.

In one oft-cited prism adaptation example, several individuals displayed extremely rapid learning, completely reducing their error on a throwing task after a single trial [28] (Fig. 1B). When queried, these individuals reported using an explicit strategy, estimating the error induced by the glasses and purposively aiming in the opposite direction to negate the perturbation. Interestingly, this strategy proved to be unstable, with successive movements increasingly overcompensating for the prismatic distortion. When instructed to throw “where the target appeared,” their performance again took the form of the stereotypical learning function, indicating that performance may reflect the combined effects of strategy use and an implicit form of recalibration.

One clever way to directly examine this hypothesis involved a variant on the standard visuomotor rotation task, one in which participants were given explicit information about the perturbation and instructed to use a compensatory strategy [34]. Vision of the hand was occluded and feedback was limited to the display of a circle that indicated the position of the hand at the end of the movement. After an initial block of trials with veridical feedback, a 45° counterclockwise perturbation was imposed. Critically, after two reaches in this altered environment the experimenter intervened, describing the perturbation and instructing the participants to aim in the clockwise direction. To facilitate the use of this strategy, landmarks were positioned at 45° intervals around the target. Thus, by aiming to the landmark 45° clockwise from the target, the perturbation could be fully negated.

As would be expected, participants performed perfectly on the subsequent trial: Using an aiming strategy enabled one-trial learning (Fig. 1C). However, over the next 80 trials, the participants’ movements began to “drift” in the direction opposite the perturbation. This paradoxical behavior — where performance worsened with practice — suggests that the motor system continued to calibrate the motor commands based on the mismatch between the intended reach location (the aiming landmark) and visual feedback, while ignoring feedback about task accuracy (the difference between the target location and observed feedback). In a subsequent experiment, it was shown that the drift reversed with extended training, an effect attributed to an adjustment in the aiming strategy (Fig. 1D, [35]). This non-monotonicity, together with evidence using various other methods, has made clear that strategy use and implicit recalibration constitute dissociable and relatively independent learning processes, with their dynamic integration resulting in the observed task performance [24,34–45].

The instructed-strategy procedure [34] has provided important insights into the computational constraints on these two processes. However, it doesn't address how or whether people develop and modify strategies in a more spontaneous manner; that is, when the experimenter does not intervene and provide explicit instructions. To address this issue, we developed a task that provides a trial-by-trial measure of the contributions of explicit aiming and implicit recalibration [24,37–39,45]. To assay strategic aiming, participants verbally report their aim direction prior to each reach, providing these reports both before the perturbation and over the course of learning (Figs. 2A,B). Using a simple subtractive procedure (reach angle minus aiming angle), we can estimate the precise state of implicit recalibration in a continuous manner. Interestingly, the lion's share of early learning, especially with large perturbations, is associated with aiming and not recalibration [38]. Furthermore, and perhaps more surprising, aiming remains prevalent even in the late stages of learning, a result that challenges the standard belief that asymptotic performance only reflects the state of a recalibrated sensorimotor mapping [37–39]. Various control conditions indicate that these results are not an artifact of the aiming report task [37,39]. The learning curves and aftereffects in this task are similar to that observed in standard visuomotor rotation tasks, suggesting that strategic processes operate even when the task context (e.g., instructions, landmarks) does not prime their use.

Figure 2. Measuring strategy use in a sensorimotor adaptation task.

(A) To obtain a direct assay on aiming strategy, participants are required to explicitly report their aim location prior to each trial. The magnitude of implicit learning can be estimated by subtracting the aiming angle from the measured movement angle. (B) There is a large contribution from explicit re-aiming right after the perturbation, which decreases over time. In contrast, implicit learning is slower and monotonic. Note that the estimated state of remapping matches precisely the magnitude of the aftereffect at the start of the washout phase. Data adapted from reference [37]. (C) The fast and slow components of the two-rate state-space model [20] closely resemble, respectively, explicit and implicit learning [39].

Implications for Computational Models of Sensorimotor Learning

The field of sensorimotor learning has benefitted from the development of rigorous computational models that not only account for observed behavioral results in healthy and neurologically impaired populations, but also generate many testable predictions [18,20,46,47]. As noted in the introductory section, prevailing models of the canonical learning curve use algorithms that capture a gradient descent reduction of error. The most prominent of such models is the two-parameter “state-space” model [18], where a Markovian learning rule is used to update the motor state and account for performance errors on a trial-by-trial basis. One parameter describes a fixed learning rate, the other corresponds to a retention, or memory term.

An important extension of this model was motivated by the idea that performance changes may reflect the operation of multiple learning processes that operate at different time scales [20]; for example, one process might learn quickly with a short retention constant, whereas a second learns more slowly with longer retention. Various puzzling phenomena observed in motor adaptation tasks, including spontaneous recovery [20,48] and savings [24,49–51] can be explained by this multiple-rate model. Moreover, this work inspired new ways of placing constraints on computational and neural mechanisms of sensorimotor learning. For example, whereas different environments may demand rapid changes (e.g., walking on granite or walking on sand), the body is generally stable. It would be advantageous to use error signals that operate at different rates depending on the nature of the representations [52]. In terms of neural systems, it has been proposed that fast cerebellar learning allows for the rapid reduction of error when learning a new skill or mapping, whereas slower learning within the motor cortex is essential for retention [53].

However, it is not clear how processes such as strategic planning fit into the picture, especially in the case of one-trial learning. One solution is to associate explicit processes with the fast process of the two-rate model, and implicit recalibration with the slow process [39] (Figure 2C). This framing is in accord with results showing that explicit, fast learning is more flexible than sensorimotor recalibration, enabling generalization to new target locations, perturbation sizes, and other variations in task demands [38]. Indeed, many markers of human sensorimotor learning, including savings [24] and structural learning [54], are likely products of our flexible ability to quickly select an appropriate movement plan.

A second major issue concerns the nature of the error signals used for learning. Current versions of multi-rate models assume that different learning mechanisms operate on the same error signal. However, it is increasingly clear that implicit and explicit forms of learning respond to distinct error signals (Figure 3A): Implicit recalibration is driven by the difference between the expected and observed outcome, what is referred to as sensory prediction error [29]. In contrast, strategy learning is sensitive to the difference between the goal and observed outcome, or what is refer to as performance error [35]. Note that in most experimental contexts, and in the natural world, these two types of errors are confounded: We usually aim at the target of our movements, so the expected outcome is the same as the goal. However, experimental manipulations such as the instructed-strategy task [34] or aiming report task [24,37–39,45] decouple these error signals. Thus, in the strategy task, the drift phenomenon described above occurs because the recalibration system is presented with a large error signal — the difference between the aiming location and the rotated cursor, even when performance error is negligible (as in the first aiming trials). Indeed, when these error signals are decoupled, it appears that implicit recalibration is completely insensitive to task success [34,35]. The non-monotonic shape of the performance curve in the strategy task reflects the fact that participants have to “re-aim” to offset the consequences of a modular implicit learning process driven by sensory prediction errors [24,35] (Figure 3A).

Figure 3. Multiple error signals in sensorimotor adaptation tasks.

(A) Dissociated error signals for recalibration (sensory prediction error) and strategizing (performance error). (B) A simplified schematic of the primary processes thought to be involved in voluntary movement. Cognitive processes (green box) provide input to implicit motor execution processes (red box). As part of the planning process, an aim is selected based on the task goal. The control policy constitutes the precise movement plan(s) that correspond to the selected goal and results in a motor command to the limb. The motor command not only drives the movement, but is fed into a forward model to generate a sensory prediction. This prediction is compared to the feedback to define the sensory prediction error, a signal that is used to update the forward model and control policy. Performance error feedback influences the planning process, allowing for strategic adjustment. The majority of research in motor learning has focused on details of the forward model and limb dynamics (red box). Further work should also address the computations occurring at the planning stages (green box).

This new conceptualization will require revisiting our computational models. Not only is it necessary to incorporate distinct error terms for explicit and implicit processes, but it may also be necessary to reconsider whether these processes utilize different learning algorithms. Recent work suggests that the gradient descent algorithm may be an inappropriate characterization of implicit learning: The learning function and asymptotic state of recalibration does not appear to be proportional to error size [38,55–57], and when isolated from task performance, recalibration appears to proceed in fixed, discrete-like steps [57]. Likewise, explicit learning appears to be highly non-monotonic, producing behaviors more consistent with active exploration and/or hypothesis testing [37]. Thus, the stereotypical learning curve may not reliably reflect individual learning curves, but may instead be an artifact that arises from the averaging of data across individuals [58]. A more accurate account of the performance function will require models that reflect the combined operation of explicit and implicit learning processes and their respective error signals (Figure 3B).

Neural Systems for Explicit Aiming and Implicit Recalibration

The notion that learning reflects the conjoint operation of multiple learning systems is prevalent in many cognitive domains such as category learning, recognition memory, and reinforcement learning [25,26,59]. The work of Milner and colleagues with amnesic patient HM was, of course, highly influential in the development of memory taxonomies, and in particular, the striking distinction between explicit, or declarative memory and implicit, procedural memory [60,61]. Although the initial demonstration of spared implicit learning in HM came from motor tasks such as mirror drawing, subsequent work revealed varying capacities for implicit learning on a range of perceptual tasks [60]. This work has inspired a half-century of research on the neural correlates of different memory processes, with the insight that learning, even within a taxonomic branch, is likely to be highly distributed.

Neuroimaging studies have shown that areas including, but not limited to, prefrontal cortex, premotor and primary motor cortices, parietal cortex, basal ganglia, and cerebellum are recruited during sensorimotor adaptation tasks [62–69]. Of particular interest here has been the cerebellum. Dating back to the 19th century, this structure has been recognized as essential for motor coordination and learning [70]. Inspired by its unique anatomy and physiology, detailed models of cerebellar learning [71–74] have been developed and refined, using tasks that involve adaptation of eye movement reflexes [51,75]. In terms of reaching studies, patients with cerebellar degeneration consistently show attenuated adaptation in response to sensory perturbations [14,15,36,47,76]. Furthermore, cerebellar activity is correlated with sensory prediction errors, the putative signal for sensorimotor recalibration [77]. Taken together, there is general consensus that the cerebellum is essential for keeping the motor and sensory systems calibrated across a range of contexts.

What are the putative neural substrates for the more cognitive contributions to motor learning? In terms of explicit processes such as strategy use, it is noteworthy that frontal lobe regions, including lateral and medial aspects of prefrontal cortex, as well as premotor cortex, often exhibit increased activity during the early phases of sensorimotor learning [62,64,78]. Although the functional role of the prefrontal activations have typically been described in terms of meta-cognitive control processes such as planning, working memory, or monitoring [67,68,78], it would also be reasonable to suppose that these regions are essential for strategic changes in aiming, consistent with a more general view of the frontal lobes being essential for action selection when the sensory-motor mapping is novel or arbitrary [79]. Aiming, at least when invoked to hasten learning in response to a perturbation, requires an indirect mapping with the direction of the action displaced from the target, similar to the spear fisher accounting for the refraction of light in water.

The instructed-strategy task [34] has revealed intriguing differences between the effects of cerebellar and frontal lobe damage: Patients with cerebellar degeneration actually perform more accurately than matched controls on this task, showing attenuated drift after implementing an aiming strategy [36]. Their impaired sensitivity to sensory prediction errors confers a form of “immunity" to maladaptive recalibration in this task. In contrast, patients with prefrontal lesions from stroke tend to show greater drift than their matched controls [80]. We assume this pattern reflects a deficit in being able to adjust their aiming strategy, even when intact recalibration has led to a gradual increase in performance error. In line with this hypothesis, older adults, assumed to have mild forms of frontal lobe dysfunction, show intact implicit learning but reduced explicit learning in visuomotor adaptation tasks [44].

We speculate that these results suggest a key role for the frontal lobe in aiming. However, there are reasonable alternative hypotheses to consider. For example, the excessive drift in our study with PFC patients [80] could reflect an insensitivity to performance error, perseveration, or even a “hyper-sensitive” calibration system (e.g., a cerebellum unchecked by the cortex). Future work that directly manipulates and measures different markers of implicit and explicit processes will be required to advance our understanding of the functional contributions of different neural systems to sensorimotor learning.

Beyond Adaptation: Towards a Broader View of Motor Learning

Tools from statistical decision theory and Bayesian statistics may prove useful in developing descriptive models, as well as offering new ways to characterize mechanisms of motor learning [81,82]. Aiming locations could be thought of as (indirect) spatial goals, cached motor commands as action options, and the planning and execution of a specific command as an enacted decision. The honing of a true motor skill, as opposed to adaptation to an external perturbation, has been theorized to entail a model-free reinforcement learning process [83]. Thus, learning in a reaching task can be characterized by a trade-off between exploration and exploitation [84], where strategic processes initially explore the manifold of actions that may yield task success, and, once a solution is found, the rewarded movement is reinforced over time. This approach helps shed light on the relationship between motor variability and learning [85]: The operation of cognitive strategies, especially prominent during early learning, may confer rapid dimensionality reduction (i.e., reducing the space of possible solutions).

Insights gained from the study of strategy use point to other aspects of cognition that are likely to be relevant for the study of sensorimotor learning. For example, there can be costs in motor performance from cognitive control [86], an idea captured by the folk psychology notion that experts are wise to not “think”, but just “do,” and reflected in the venerable model of Fitts and Posner on the stages of skill acquisition [87]. However, we cannot assume that the reduction in cognitive contributions to performance implies that all learning has shifted to the implicit calibration system. There appear to be multiple forms of implicit learning: In addition to error-based remapping, the evidence suggests that changes in performance also reflect contributions from associative processes such as use-dependent learning and operant conditioning [50,88,89].

Moreover, it is unclear if this explicit component ever really “disappears.” For instance, professional riflemen adeptly use “Kentucky Windage” to adjust their aim to correct for the direction of the wind. In this and other cases, cognitive strategizing is the mark of an expert, not an amateur. Indeed, the kinds of cognitive strategies discussed here are not limited to motor tasks: A generalized capability for one-trial learning has obvious implications for learning writ large.

Ultimately, it is critical to incorporate the influence of cognitive planning into any realistic and comprehensive model of human sensorimotor learning. High-level motor planning is not just relevant to spearfishing, darts, or shooting: The ability to execute aimed movements — to rapidly, accurately, and flexibly perform planned, multi-joint movements to interact with the environment — is a hallmark of human behavior.

Outstanding Questions.

How should aiming strategies be modeled and integrated into standard models of sensorimotor learning? How can such models be modified to include ideas from work on decision making and reinforcement learning to provide a comprehensive picture of motor performance and learning?

What are the putative neural substrates contributing to the cognitive processes underlying motor strategies and heuristics, like aiming? How much do these substrates overlap with the known neural architecture involved in planning and decision-making?

Do strategic processes and recalibration processes directly interact, or are the systems psychologically and neurally “quarantined” from each other?

Do explicit strategies become proceduralized over time as a true skill is acquired?

What aspects of “cognitive” mechanisms for motor learning are shared with other species and which, if any, are unique to humans?

TRENDS Box.

Behavioral, computational, and neuropsychological studies have provided a detailed picture of the processes involved in sensorimotor adaptation tasks. This work has been based on laboratory studies in which sensorimotor feedback is perturbed, using tools such as prism glasses, force fields, and visuomotor rotations. Performance changes have been attributed to learning mechanisms that modify a sensorimotor mapping based on sensory prediction errors, the difference between predicted and observed feedback. However, a growing body of research points to the operation of additional learning processes, including the use of cognitive strategies and heuristics. In adaptation experiments, such strategies can be characterized as “aiming”, one form of a flexible, goal-oriented motor plan.

Theoretical models are being revised to address the interaction of multiple learning processes, specifying computational constraints concerning the teaching signals used by different learning mechanisms. Furthermore, the search for the neural substrates of motor learning has cast a wider net, going beyond the role of the cerebellum and motor cortex, to include, for example, frontoparietal areas involved in planning and decision making.