Cerebellar and Prefrontal Cortex Contributions to Adaptation, Strategies, and Reinforcement Learning

Abstract

Traditionally, motor learning has been studied as an implicit learning process, one in which movement errors are used to improve performance in a continuous, gradual manner. The cerebellum figures prominently in this literature given well-established ideas about the role of this system in error-based learning and the production of automatized skills. Recent developments have brought into focus the relevance of multiple learning mechanisms for sensorimotor learning. These include processes involving repetition, reinforcement learning, and strategy utilization. We examine these developments, considering their implications for understanding cerebellar function and how this structure interacts with other neural systems to support motor learning. Converging lines of evidence from behavioral, computational, and neuropsychological studies suggest a fundamental distinction between processes that use error information to improve action execution or action selection. While the cerebellum is clearly linked to the former, its role in the latter remains an open question.

1 INTRODUCTION

The rules of American baseball define the strike zone as a region with a fixed width (17″) and variable height based on the distance between the hitter’s chest and knees. The hitter is most vulnerable to low pitches, ones that cross the zone near, or just below, the knees. For such pitches, hitters are successful in reaching base less than 20% of the time, considerably lower than their overall success rate (Encina, 2013). Given these probabilities, the pitcher would be wise to consistently aim for this location. However, this strategy not only takes considerable practice but also entails considerable risk. Pitches that are just a few inches too high end up right where hitters have their best success; a hoped for strike out pitch is suddenly a fan’s home run souvenir.

Given these challenges, we can ask, how does the pitcher master this skill? One possibility is that learning centers on updating processes involved in action execution. By this view, to improve accuracy, the pitcher might aim to the same location each time, using the outcome of recent throws to adjust a learned sensorimotor relationship with the goal of reducing variability. However, low variability entails its own cost in baseball. The pitcher must vary the targeted location so that the hitter cannot focus on one region of the strike zone. A successful pitcher has to use outcome information to also improve action selection. Perhaps, the next pitch should be aimed slightly higher or lower in the strike zone, or, depending on the previous outcome, require a shift to a new region of the strike zone.

In this review, we focus on recent developments in the motor learning and skill acquisition literature that explore how people use a multiplicity of learning processes to improve action execution and action selection. This theme has been advanced in behavioral, computational, and neuroscientific studies. With respect to the latter, sensorimotor learning has long been assigned to the functional domain of the cerebellum, inspired by models of how this subcortical structure is essential for error-based learning. However, the multiple learning systems perspective underscores the need to consider the cerebellum within the broader context of a distributed learning network and point to interesting ways in which the cerebellum interacts with other subcortical and cortical systems during skill acquisition.

2 THE CEREBELLUM AND ERROR-BASED LEARNING

The role of the cerebellum in coordination and movement regulation took hold in the nineteenth century. Ablation of this structure in a variety of animals produced profound impairments of coordination, even in the absence of weakness (Dalton, 1861; Fine et al., 2002; Flourens, 1824; Marshall and Magoun, 1997). Similarly, early descriptions of individuals with lesions of the cerebellum focused on the decomposition of multijoint movements (Babinski, 1896; 1902) and abnormalities in the regularity, rate, and force of muscle activations (Holmes, 1939), a constellation of symptoms now referred to as cerebellar ataxia. A defining notion of ataxia is that this disorder produces problems in the execution of goal-directed movements, even if the affected individual still retains the intent, or representation of the goal of the desired action (Holmes, 1939).

While the early neurology literature focused on the loss of coordination in ataxia, the second half of the twentieth century witnessed a paradigm shift as the study of learning came to the forefront. As detailed pictures of the idiosyncratic anatomy and physiology of the cerebellum began to emerge, computational neuroscientists took up the challenge to develop functional hypotheses of the cerebellum. Two highly influential papers, the first published by David Marr in 1969 and the second by James Albus in 1971 (Albus, 1971; Marr, 1969), laid out the core ideas of how the cerebellum subserves an essential role in error-based learning, a hypothesis that remains central in current studies of cerebellar function.

While a thorough review of this work is beyond the scope of this chapter, it is instructive to review the key features of the Marr–Albus theory. Both papers sought to explain why the Purkinje cells of cerebellar cortex receive two unique inputs, the parallel fibers and the climbing fibers. Parallel fibers are the axonal extensions of granule cells, with each fiber making single synapses on hundreds of thousands of Purkinje cells. The parallel fibers carry information from the ascending tracts of the spinal cord, many subcortical nuclei, and, via the pontine nuclei, large parts of the cerebral cortex (Jansen and Brodal, 1954). The integration of parallel fiber activity causes the Purkinje cells to generate simple spikes, high-frequency bursts of firing. In contrast, climbing fibers originate in the inferior olive. Each of these fibers targets at most a few Purkinje cells, but through extensive innervation patterns across the Purkinje cell dendritic arbor, the climbing fiber can result in a massive action potential, the complex spike.

Marr and Albus recognized that the simple-complex spike arrangement offered an ideal situation for supervised, error-based learning. In this model, the parallel fibers generate a representation of the state of the system, a state that incorporates information about the state of the body as well as a state that has access to current motor commands (e.g., efference copy). The climbing fibers serve as the teacher, generating complex spikes when an unexpected event is encountered. With rather simple, yet elegant, algorithms, this interaction provides the essential ingredients for error-based learning. In the Albus model, this learning was hypothesized to entail a weakening of synaptic strength between the parallel fibers and Purkinje cells, an idea that anticipated long-term depression (Albus, 1989; Ito et al., 1982). The Marr–Albus theory has been elaborated and modified over the past 40 years, but the core idea of the cerebellum as a system for supervised, error-based learning has become established as one of the central tenants of cerebellar theory (Ito, 2006).

Early experimental tests of the Marr–Albus model came from studies of the vestibular-ocular reflex, with gain changes in reflex correlated with variation in simple spike activity (Fukuda et al., 1972; Ito, 1974). More causal tests came about with the seminal discoveries of Richard Thompson and colleagues on classical conditioning of the eyeblink response in the rabbit (McCormick and Thompson, 1984a, b). This work provided compelling evidence that the conditioned response was localized to the cerebellum: Focal lesions of either the cerebellar cortex or deep cerebellar nuclei resulted in the abolition of the CR with minimal effect on the UR (Yeo et al., 1985). The eyeblink paradigm also allowed experimenters to conduct strong tests of the Marr–Albus model, replacing the effects of the CS and US by direct stimulation of the parallel fibers or inferior olive, respectively. Eyeblink conditioning and VOR adaptation continue to be amazingly fruitful tasks, serving as model systems for studying the cellular, molecular, and genetic basis of sensorimotor learning (Boyden et al., 2006; Gao et al., 2012; Raymond et al., 1996; Schonewille et al., 2011). These tasks have also been used in behavioral and imaging studies in humans, with the results providing converging evidence of an essential learning role for the cerebellum (Cheng et al., 2008; Logan and Grafton, 1995; Schubert and Zee, 2010; Timmann et al., 2010).

Critical to both eyeblink conditioning and VOR learning is the presence of an error signal. In the former, the airpuff constitutes an error (i.e., unexpected aversive stimulus) and the animal learns to attenuate the negative effects of this US by closing the eyelid in response to predictive conditioning stimulus such as a tone or light. In the latter, the error comes in the form of retinal slip, the difference between the position of the eye and the stimulus being tracked. The notion of error representation in the cerebellum also came from reaching studies in the primate, where climbing fiber discharge was observed when the animal experienced an unexpected sensory event (Gilbert and Thach, 1977).

Most pertinent to the current review is the body of literature that has amassed over the past 25 years involving studies of sensorimotor adaptation during volitional action. Here, researchers have employed a range of environmental perturbations, asking how cerebellar pathology affects adaptation. The simplest task, at least experimentally, is to have participants wear prism glasses and make reaching movements to visual targets (Held and Hein, 1958; Helmholtz, 1909/1962). Participants learn to adjust their reaching or throwing movements in a direction that offsets the prismatic distortion. The time course of learning generally follows an exponential function, one in which the error is reduced in a roughly proportional manner across training (Fig. 1). When the prismatic lens are removed there is a prominent aftereffect, such that it takes several reaches or throws to return to original (nonprism) eye-hand calibration. Cerebellar damage, experimentally induced in nonhuman primates or naturally occurring in humans, results in attenuated learning (Baizer and Glickstein, 1974; Martin et al., 1996; Weiner et al., 1983). The subjects continue to produce large errors even after many reaches when wearing the prismatic devices, and, correspondingly, show a smaller aftereffect than control participants.

FIGURE 1.

Hypothetical learning curve during adaptation to an arbitrary visuomotor perturbation. The perturbation is imposed during movements 100–200. Target errors are initially in the direction of the perturbation, but, with training, adaptation occurs. The perturbation is removed on trial 201 and an aftereffect is observed in which target errors are in the direction opposite to the perturbation. The size of the visuomotor perturbation is in arbitrary units (percent).

This cerebellar-mediated learning impairment has been confirmed in many sensorimotor adaptation studies in which the participants make reaching movements within virtual reality environments. For example, participants can be asked to reach in a force field, with the hand displaced in a direction orthogonal to the path of motion by a force that is proportional to velocity (Smith and Shadmehr, 2005) or a visuomotor perturbation can be imposed by rotating the position of a feedback cursor relative to true hand position. Across these various types of perturbations, the picture is quite consistent in showing that patients with cerebellar pathology exhibit a marked impairment in adapting to sensorimotor perturbations (Criscimagna-Hemminger et al., 2010; Gibo et al., 2013; Izawa et al., 2012; Morton and Bastian, 2004; Rabe et al., 2009; Smith and Shadmehr, 2005).

An appealing feature of virtual reality environments is that they provide the experimenter with control over the perturbation, and, as such, offer the opportunity to manipulate the magnitude and form of the error signal. For example, a force-field or visuomotor perturbation can be introduced abruptly or in a gradual manner. In the former, the participant is aware that the environment has been altered, even though their response to the perturbation may or may not be under strategic control, an issue we return to below. In the latter, the participant is generally completely unaware of the perturbation, at least during the early trials of learning. While one paper indicated that patients with cerebellar pathology showed a much more pronounced deficit in adapting to an abrupt perturbation (Criscimagna-Hemminger et al., 2010), subsequent work indicates that the patients’ deficit is similar to both types of perturbations (Gibo et al., 2013; Schlerf et al., 2013).

When analyzed at a group level, cerebellar pathology clearly disrupts learning across a range of adaptation tasks. However, a more fine-grained analysis points to some degree of domain-specificity within the cerebellum. Indeed, one of the first studies of prism adaptation (Martin et al., 1996) pointed to a dissociation between cerebellar contributions to learning and coordination. Ataxia was especially marked in patients with lesions of the superior regions of the cerebellum, lesions that encompassed the classic arm/hand representation in lobule V. However, these patients tended to show modest deficits in adaptation. In contrast, patients with more inferior lesions showed marked deficits in learning, despite minimal evidence of ataxia. More recent work using sophisticated lesion reconstruction methods has revealed intriguing dissociations within the cerebellum on different adaptation tasks. Force field adaptation deficits are associated with lesions of more superior aspects of the cerebellum relative to visuomotor adaptation deficits (Donchin et al., 2012; Rabe et al., 2009), a pattern that is consistent with anatomical differences in the representation of task-relevant information. Whereas the errors for force-field adaptation are primarily somatosensory, visuomotor adaptation is primarily driven by a visual error signal. There is a crude superior-inferior gradient in terms of the relative projections of somatosensory and visual inputs to the cerebellar cortex.

3 COMPUTATIONAL MODELS OF SENSORIMOTOR ADAPTATION

The neuropsychological literature discussed above provides compelling evidence that patients with cerebellar lesions are impaired on tasks requiring sensorimotor adaptation. However, the specific computational role of the cerebellum in such tasks has been the subject of considerable debate. The Marr–Albus model predicts that error signals, arising from the climbing fibers shape parallel fiber-Purkinje cell synapses to modulate the representation of the system’s state, with this modulation producing changes in future responses to similar states. However, most adaptation studies utilize a block design in which the perturbation (e.g., prisms, forces, and rotations) is applied for a fixed period of time. The lack of variance in the perturbation makes it difficult to elucidate a trial-by-trial relationship between error signals and changes in motor output. Through the introduction of randomly varying perturbations, two seminal studies with healthy individuals were able to identify the relationship between error and adaptation on trial-by-trial basis (Scheidt et al., 2001; Thoroughman and Shadmehr, 2000). The results of these studies showed that the amount of change on a single trial was proportional to the size of the perturbation or motor error on the preceding trial.

This learning process can be characterized by a linear dynamical system, with various instantiations being realized in state–space models (Thoroughman and Shadmehr, 2000), autoregressive models with exogenous inputs (Scheidt et al., 2001), or hidden Markov models (Schlerf et al., 2013). For a visuomotor rotation task, the dynamical system can be represented as a state–space model as follows:

$$e_n=r_n-{\widehat{r}}_n$$ (1)

$${\widehat{r}}_{n+1}=A{\widehat{r}}_n+{Be}_n$$ (2)

Equation (1) represents the error on trial (n), which captures the idea that sensorimotor adaptation requires learning an internal model (r̂_n). When the system is properly calibrated, the output anticipates the effects of the perturbation (r̂_n). Learning in this model is error-driven: In Eq. (2), B reflects the learning rate, or the proportion of the error that is compensated for from one trial to the next. The value of B tends to be between 0.10 and 0.30, meaning trial-to-trial corrections adjust for approximately 10–30% of the error. While learning would occur more rapidly with higher values of B, such systems tend to be unstable. The other parameter A is considered a memory term, indicating how well the system retains a memory of the internal model from trial-to-trial. Values for this are almost always quite high (A>0.99), at least for relatively simple tasks such as reaching.

Equations (1) and (2) describe the simplest form of a state–space model, capturing the effects of learning in a range of adaptation tasks through the operation of a single learning process. More sophisticated versions have focused on the idea that error information and the updating process may occur over multiple timescales (Smith et al., 2006). For example, a fast system may operate with a large learning rate (B) and a small memory term (A), whereas a slow system may use a smaller learning rate (B) and a large memory term (A). Multirate models have been employed to account for signatures of interference, forgetting, and recall within the linear dynamical system framework.

These models have also been used to specify the learning impairments observed in patients with cerebellar degeneration. Tseng et al. (2007) used an adaptation task in which participants learned to reach in the face of a 20° visuomotor rotation (Tseng et al., 2007). They compared two conditions: In one, the participants were provided with continuous online feedback and required to terminate the movement at the target. In the other, the participants were instructed to produce “shooting movements,” attempting to pass through the target until they contacted a virtual pillow. Contrasting these two conditions allowed the authors to evaluate different hypotheses for the patients’ learning deficit. It is possible that learning deficits are secondary to control problems. For example, the patients’ ataxia might make it difficult to use online feedback or control the terminal phase of a movement, with the added control problems placing demands on resources that could otherwise be used for learning. By including the shooting condition, the experiments sought to reduce the control demands on the patients, both by eliminating online corrections and providing an external support to aid movement termination. However, the results showed that the patients were equally impaired in both conditions (Fig. 2A). More important, a model-based analysis revealed a common learning-rate deficit in both tasks. Whereas the learning rate for controls ranged from 0.10 to 0.34, the values for the patients clustered around 0.03 (Fig. 2B). Taken together, the two studies provide strong evidence that the patients’ learning deficit centers on an impairment in trial-by-trial adaptation and is not secondary to problems related to their ataxia.

FIGURE 2.

Impaired adaptation in patients with cerebellar ataxia. (A) Top: Control participants learned to counter a 20° perturbation (dark shaded area) with either pointing (left) or shooting movements (right) and showed a large aftereffect. Bottom: Patients with ataxia were unable to counter the perturbation and showed a smaller aftereffect. (B) The adaptation rate, as measured by a state–space model, was similar for the two types of movements. The adaptation rates for the patients cluster near the lower values for both tasks.

This hypothesis is further reinforced in a study that compared two types of visuomotor rotations, one in which a 20° perturbation was introduced abruptly and another in which the rotation was introduced gradually in 4° increments (Schlerf et al., 2013). In both cases, the patients adapted at a slower rate than the controls, reached lower levels of asymptotic performance, and showed reduced aftereffects. The data were analyzed with a model designed to assess if the performance deficit might reflect a credit assignment problem: Intuitively, one might assume that someone with ataxia may attribute an error in performance to their inability to control their movements rather than attribute the error to a change in the environment (and thus should be incorporated in an internal model of that environment). To address this question, estimates of the participants’ motor variability in the absence of a perturbation were obtained. These values were then used in a probabilistic model based on a Markov-chain process to estimate learning rates. The results indicated that the ataxic individuals exhibited a reduction in learning rate, even when the differences in motor noise were incorporated into the model (see also, Smith and Shadmehr, 2005).

4 MULTIPLE LEARNING MECHANISMS IN SENSORIMOTOR ADAPTATION

Linear dynamical systems have provided a simple, yet elegant tool to account for performance on sensorimotor adaptation across a range of tasks. However, a single-process version, such as that described by Eqs. (1) and (2), have proven to be inadequate to account for more complex learning phenomena such as generalization, spontaneous recovery, and asymmetries between the rate of adaptation and the rate at which the aftereffect washes out once the perturbation is removed (Zarahn et al., 2008). As noted above, one solution has been to posit that error-based learning may operate over multiple timescales: both the error and decay parameters can be expanded to influence performance over multiple trials or change as a function of time. For example, the two-rate model of Smith et al. (2006) not only accounts for the rather abrupt shape of many learning functions but, more importantly, can account for patterns of interference observed when participants are successively exposed to conflicting perturbations (Smith et al., 2006).

Multiple-rate models fail to capture one phenomenon observed in many studies of human learning: savings in relearning (Zarahn et al., 2008). Savings is defined as faster learning upon reexposure to something that had been previously learned, but then “forgotten.” In classical conditioning studies, faster acquisition of a conditioned response following extinction compared to initial acquisition would constitute savings. The classic account of this phenomenon is that extinction did not really abolish the conditioned association, but rather, induced the animal to learn a second association, one in which the CS is not paired with the US, and thus does not generate a CR. Savings occurs because the repairing of the CS and US invokes the original context, allowing the dormant CS–CR association to be unmasked. Linear dynamical systems are incapable of producing savings since such systems do not retain a memory of previous states: learning in such systems involves recalibrating the state of a single representation, rather than the acquisition of multiple representations. As such, once the aftereffect is washed out, learning would have to begin anew, even if the original perturbation was reintroduced. This prediction, however, is at odds with a number of empirical reports (Huang et al., 2011; Kitago et al., 2013; Zarahn et al., 2008). Adaptation occurs much more rapidly, especially in the initial trials when people are reexposed to a previously learned perturbation.

Observations such as these have led motor learning theorists to consider that performance changes in sensorimotor learning tasks involves the operation of multiple learning processes (Huang et al., 2011; Kitago et al., 2013). Indeed, this trend brings the study of motor learning into closer alignment with memory research where theorists have long entertained the idea of multiple learning systems and processes. This issue was, of course, brought to the forefront in Milner’s classic studies with HM (Scoville and Milner, 1957). Not only did this case indicate that the medial temporal lobe was essential for the formation of selective types of memories, but the case also reemphasized the classic observation that severe impairments in learning can exist even when long-term memory is relatively spared.

A similar question can be asked with respect to the cerebellum: While the evidence clearly indicates that this structure is essential for sensorimotor adaptation, or learning, is it also essential for storage, or consolidation of the acquired memory? Or is there a partition between learning and consolidation for sensorimotor adaptation, similar to what has been proposed for episodic and declarative memory?

This issue has been addressed in the eyeblink conditioning literature. Lesions of either the cerebellar cortex or deep cerebellar nuclei preclude the acquisition of the conditioned response. However, if the lesions are introduced postacquisition, the conditioned response is only abolished following lesions of the DCN (Clark et al., 1984; McCormick and Thompson, 1984a). In contrast, animals with postacquisition lesions restricted to the cerebellar cortex continue to produce conditioned responses (McCormick and Thompson, 1984b). Interestingly, the adaptive timing of these responses is disrupted (Koekkoek et al., 2003; Perrett et al., 1993). Rather than produce CRs that are maximal at the anticipated time of the US, the animal now produces CRs that occur in immediate response to the CS. Thus, the cerebellar cortex is essential for learning (especially the precise temporal features of the CR), but consolidation of the acquired association may be independent of the cerebellar cortex (Kellet et al, 2010).

The question of whether learning and consolidation are functionally distinct in sensorimotor adaptation has received relatively little attention. Recently, this problem has been taken up in studies comparing the relative contribution of the cerebellum and cerebral cortex during sensorimotor learning. Galea and colleagues (2011) evaluated performance changes during visuomotor adaptation following transcranial direct cortical stimulation (tDCS) (Galea et al., 2011). Anodal tDCS has been found to facilitate learning in a range of tasks (Nitsche et al., 2010; Reis et al., 2009), presumably by putting the targeted region into an “up” state by increasing neuronal excitability at baseline. In the Galea study, tDCS of the cerebellar cortex increased learning rates during adaptation compared to tDCS of the motor cortex or sham stimulation, but had no effect on the recovery from the aftereffect when the perturbation was removed. tDCS of the motor cortex had the opposite effect: learning rate during adaptation was unaffected, but the aftereffect persisted for a longer period of time. Although washout only constitutes a very early probe of consolidation, this double dissociation suggests a selective role for the cerebellum in learning, with consolidation being cortically mediated. The latter hypothesis is further supported by evidence showing that consolidation is selectively disrupted when TMS pulses are applied over motor cortex during force-field adaptation (Hadipour-Niktarash et al., 2007).

Computationally, the cerebellum and neocortex have been hypothesized to use distinct learning processes. The cerebellum, with its unique physiology, is viewed as the prototypical system for error-based learning, with adaptation driven by the difference between the predicted and actual sensory outcome of a movement. In contrast, learning within the cortex, may be primarily driven by Hebbian processes, with synaptic efficacy strengthened as a function of coactivation. Within the motor learning field, the behavioral signature of Hebbian learning has come to be called use-dependent learning, reflecting the fact that repetition alone can be sufficient to increase the likelihood of a movement (Diedrichsen et al., 2010), or introduce a bias in the execution of other, related movements (Verstynen and Sabes, 2011). A use-dependent process can account for savings (Huang et al., 2011); as such, savings may not arise from facilitation of error-based learning processes associated with the cerebellum, but rather from the reactivation of movement patterns stored in the cerebral cortex. In this view, savings is linked to processes associated with action selection, with the reintroduction of the perturbation serving as a cue for memory recall (Morehead et al., 2013).

Models of decision making have focused on yet another learning process, reinforcement learning, to account for how organisms learn to select the optimal response for a given context (Daw et al., 2006; Sutton and Barto, 1998). For the rat in the maze, reinforcement learning processes can explain how the animal chooses to turn toward the baited arm when approaching the branch point in a T-maze. For the human at the casino, a similar process is hypothesized to dictate whether a gambler continues to hammer away at one slot machine in expectation of a jackpot, or switches seats to try her luck on another machine. As with error-based learning, reinforcement learning operates by comparing an expected and realized outcome. However, the expectation here is on anticipated reward (Sutton and Barto, 1998). If an outcome produces a greater than expected reward, the likelihood of repeating that action is increased; if the outcome is less than the expected reward, the likelihood of repeating that action is decreased. Dopamine activity in the basal ganglia, and in particular, the ventral striatum, correlates with the size of these prediction errors (Schultz, 1998).

An important difference between standard models of reinforcement learning and error-based learning relates to the information content of the error signal. In error-based learning (see Eq. 2), the error is vectorial: the sensory prediction error provides information on how the movement should be modified in order to be successful on future actions. For example, if the reach lands to the left of the target, then the internal model has to be recalibrated to reduce this deviation, a form of gradient descent. In reinforcement learning, the error is either categorical (e.g., the rat either obtained the reward or failed to obtain the reward) or, if metrical, indicates the difference in the value of the reward. Receiving a small payoff from one slot machine does not provide information concerning which of the other slot machines is likely to provide a bigger payout. In general, reinforcement learning has been applied to account for how organisms choose which action to take, rather than explain how a selected movement is executed or optimized.

In principle, the performance changes observed during sensorimotor learning could come about from reinforcement learning, error-based, or some combination of these processes. In a recent study of visuomotor adaptation, Izawa and Shadmehr (2011) provided a particularly clever comparison of reward-based and error-based learning during a visuomotor adaptation task (Fig. 3). In both conditions, a rotation of 8° was introduced gradually over 320 trials. In the error-based condition, participants were provided with online feedback of the cursor. In the reward-based condition, they received binary feedback, indicating if the reach had intersected the target or missed the target. Participants modified their trajectories in both conditions, effectively counteracting the effects of the rotation. That is, performance improved in a similar manner with either reward- or error-based feedback.

FIGURE 3.

Compensating for a visuomotor rotation with different types of feedback. (A) Feedback was either provided online or in the form of binary signal indicating success or failure (reward). (B) Performance for two representative participants over the course of 500 trials in which an 8° perturbation was gradually introduced. (C) Sensory remapping, as measured by the localization task, was present for the online feedback group and negligible for the reward feedback group.

A test of sensorimotor remapping, however, indicated that the representational change was quite different in the two conditions (Fig. 3C). For this test, participants were required to reach to a position with the trained limb and then reach to the same position with the untrained limb. Participants trained with error-based feedback showed a discrepancy between the final hand position of the two hands, suggesting a recalibration of the sensorimotor mapping associated with the trained hand. In contrast, participants who received reward-based feedback did not show a difference between the judged position when reaching with either hand, suggesting that no remapping had occurred. Thus, reward-based feedback was sufficient information to counter the gradually introduced rotation, but was insufficient to train an internal model.

It is important to ask how participants in the reward condition learned to counter the rotation since they could not see the size or direction of their movement errors. Theoretically, there are two, related possibilities. First, due to simple random variation in reaching performance, reaches that deviate in the opposite direction of the rotation would be positively reinforced. The selective reinforcement of these reaches would induce a systematic shift in reaching direction, one that counters the rotation. Second, success with reward-based feedback could come about through a more exploratory process. That is, the participant might explore different reaching directions, with reinforcement-learning mechanisms biasing the system toward actions that result in rewards and away from actions that fail to produce rewards. With this process, the exploratory process would have to be repeated and expanded as the size of the rotation increases.

In support of this second idea, participants in the reward-based feedback condition showed an increase in reach variance compared to the error-based feedback condition (Fig. 3C). Indeed, Izawa and colleagues (2011) developed a reinforcement-learning model that learned to counter the rotation through random exploration that resulted in the reinforcement of successful action policies. Learning within this model does not involve the adaptation of an internal model, consistent with the observation that the reward-based participants did not show a change in performance on the intermanual matching task.

The study by Izawa and colleagues highlights that the same visuomotor rotation can be solved through two very different forms of learning. With vectorial errors, performance can improve through the adaptation of an internal model; as described in Eqs. (1) and (2), the error provides a supervised signal that informs the system about how an internal model should be changed. This process is not possible with categorical errors. As such, performance improvements come about via reinforcement-learning mechanisms that promote changes in action selection. It is interesting to note that in both conditions, learning was implicit. Whereas adaptation is widely recognized as an implicit process, models of reinforcement learning make transparent that changes in action selection can also result from the operation of implicit processes.

Supposing that learning can result from either adaptation or changes in action selection helps resolve some lingering discrepancies in the sensorimotor adaptation literature. A number of studies have reported that the size of the aftereffect, the cleanest probe of adaptation, frequently falls well short of the size of the rotation when participants are not provided with online feedback (Hinder et al., 2008; Peled and Karniel, 2012; Shabbott and Sainburg, 2010). It may well be that, under such conditions, performance changes reflects the combined effects of two (or more) learning processes. Error-based adaptation would be strongest with online feedback (and thus produce larger aftereffects), whereas endpoint feedback might promote changes in action selection. In Section 5, we will turn to recent work on sensorimotor adaptation in which experimenters have developed methods to directly examine the simultaneous operation and interaction of multiple learning processes.

5 STRATEGY USE DURING SENSORIMOTOR ADAPTATION

Error-based, use-dependent, and reinforcement learning all entail the operation of continuous mechanisms that gradually guide performance to the correct solution. As noted above, learning rates in adaptation studies tend to be between 0.10 and 0.30, indicating that only a fraction of the error is accounted for when updating the internal model. The rate of learning in reinforcement-learning studies varies as a function of the task context and number of action choices, but tends to fall in the higher end of this range (Daw et al., 2006; Li and Daw, 2011; O’Doherty et al., 2003; Rutledge et al., 2009; Stoloff et al., 2011; Wittmann et al., 2008). Use-dependent learning is likely a slower process, especially if dependent on Hebbian mechanisms where synaptic changes require multiple repetitions.

A notable feature of human learning is our ability to rapidly modify our behavior in response to perceived changes in the context. In the classic learning literature, this phenomenon is sometimes referred to as one-shot learning (Carey and Bartlett, 1978; Lake et al., 2011), although such changes need not be achieved in a single trial. For example, the field goal kicker on a football team may note that there is a strong crosswind blowing from left to right and choose to aim his kick to the left of the goal posts. Strategic changes such as this can lead to abrupt changes in performance. Recent studies have begun to ask how such processes impact sensorimotor learning, and, correspondingly, what types of information constrain the utilization of strategies.

The idea that motor learning can benefit from strategic and explicit processes has a long history in cognitive psychology. Fitts and Posner (1967), in their classic work, proposed that skill learning could be characterized by three sequential stages (Fitts and Posner, 1967). Learning begins with a cognitive stage in which verbally based, cognitive strategies are used to establish the task goals and general movement features required to achieve these goals. In this stage, the person must determine the appropriate sequence of actions. In a second, associative stage, the sensorimotor space is explored, establishing the mapping between the desired sequence of actions and their associated movements. Finally, an autonomous stage entails the consolidation of the motor commands, allowing actions to be performed in a fluent and flexible manner.

While this idea has been around for nearly 50 years, the contribution of explicit strategies to sensorimotor adaptation tasks has largely been ignored in the experimental literature. One reason is that strategies tend to be idiosyncratic and highly variable across individuals. In addition, it can be difficult to assess strategies within the context of the task, and reports at the end of learning are of questionable reliability given that the strategy may change over time. For example, participants have difficulty describing complex force-field perturbations, even if they have a sense that something about the environment has been altered with the onset of the perturbation. For these reasons, studies of explicit processes in sensorimotor adaptation tasks have generally used indirect probes such as changes in reaction time (Fernandez-Ruiz et al., 2011), self-reports (Heuer and Hegele, 2008; Hwang et al., 2006), or susceptibility to dual-task interference (Galea et al., 2010; Taylor and Thoroughman, 2007, 2008).

A direct approach was introduced by Mazzoni and Krakauer (2006) in a study of visuomotor rotation. The experimenters described the 45° clockwise perturbation to the participants and instructed them to aim in the counterclockwise direction of the perturbation as a compensatory strategy (Fig. 4). To facilitate strategy use, landmarks were added to the display, positioned 45° apart. Thus, when the target appeared at one location, the participant simply had to aim to the neighboring landmark in order to negate the perturbation. Not surprisingly, the participants were able to use the strategy and immediately succeed in compensating for the rotation. However, as training continued, their performance deteriorated. The reaches drifted in the direction of the strategy (direction was greater than 45° counterclockwise), with the error growing to over 20°.

FIGURE 4.

Experimental conditions used by Mazzoni and Krakauer (2006) to assess the effect of strategy use in countering a visuomotor rotation. (A) Top left: Before the rotation the participant reaches to a goal target (gray-filled circle) and the visual feedback is veridical. Top right: The participant experiences two trials of the 45° visuomotor rotation. Bottom left: The participant is then instructed to offset this rotation by aiming to a landmark 45° clockwise from the target (unfilled circle). Bottom right: As training continues, target errors (performance) drift in the direction of the aiming target. Note that in all displays, eight circles arranged along an invisible ring, indicating possible target locations were always visible. One turned into the goal target. (B) The top panel shows a standard adaptation with a gradual reduction in directional error over trials. In the bottom panel, the instructions are provided after the first two trials with the rotation. Target error immediately drops to near 0, but then increases (drifts) in the opposite direction.

How can we account for this paradoxical result, a situation where performance actually gets worse with practice? Mazzoni and Krakauer (2006) hypothesize that this task design pits two processes against one another, an explicit strategy and implicit sensorimotor adaptation, with the latter winning out. The key here is to consider the error signal for adaptation. We typically think of the error as the difference between the target location and the feedback location. However, their findings suggest that, the error signal for adaptation is not target error but rather is based on the difference between the predicted and actual outcome, that is, a sensory prediction error. With the instructed strategy, the prediction is no longer at the target location; it is now at the aiming location. Thus, when feedback appears at the target location, the adaptation system is presented with a large error. The internal model is adjusted to reduce this error, resulting in the drift phenomenon where performance error increases across trials. Consistent with this account, participants showed a sizable aftereffect when the rotation was turned off and they were instructed to reach directly to the target. If adaptation were driven by an error based on a comparison of the target location and feedback, neither drift nor an aftereffect should have been observed. These results suggest that adaptation is insensitive to whether or not the movement achieves its goal.

Movement goals and sensory predictions are usually well-aligned: In the standard visuomotor adaptation task, participants aim for the target location and expect their movement to terminate at that location. One might suppose that Mazzoni and Krakauer (2006) have created a clever, yet ecologically unrealistic situation. However, there are likely many situations in which the final outcome of an action deviates from the initial planned action. Consider our baseball example again in which a pitcher is attempting to throw a 12–6 curveball, one in which the pitch initially looks to cross the plate near the hitter’s chest and then drops down to the ankles (thus, 12–6 as in the positions on a clock face). For this pitch, the 12 o’clock aiming direction does not match the final 6 o’clock position. Given the results of Mazzoni and Krakauer (2006), we might expect our pitcher to start elevating the pitches as they correct for the difference between the predicted and actual position of the ball (at least if the adaptation system does not have a sense of the effect of a curveball). At present, we can only rely on anecdotal evidence with respect to pitching—certainly there are many regretful pitchers who have watched a mislocated curveball result in a home run.

It is also possible that, with practice, skilled actions come to reach a balance between the effects of strategic and adaptation processes. Mazzoni and Krakauer limited practice with the rotation to just 80 trials, observing the endpoint error rise to approximately 25°. By their model, we would expect that this error would, with extended practice, continue to grow up to 45°, at which point the sensory prediction error would be zero. We tested this prediction by using their aiming strategy task in an extended training session (Taylor and Ivry, 2011). Consistent with their results, participants initially drifted in the direction of the strategy. However, with continued training, the target errors began to decrease and, by 200 trials or so, were near zero (Fig. 5A). Interestingly, when the rotation was turned off after 320 trials, an aftereffect of around 20° was observed.

FIGURE 5.

Extended training while using an explicit strategy to counter a 45° rotation. The target error becomes small when the strategy is implemented (Trial 42). The target error drifts in the direction of the strategy for about 80 trials and then reverses with performance eventually becoming asymptotic with little target error. However, a large aftereffect is observed when the rotation is turned off and participants stop using the strategy, revealing the magnitude of implicit adaptation. Circles: Observed data for the group. Solid curve: Fit of the two-process model fit. (B) Implicit adaptation is based on a sensory prediction error (aiming error, gray), defined as the difference between the aiming location and the feedback location. Strategy adjustment is based on target error (black), the difference between the target location and the feedback location.

To account for this nonmonotonic behavior, we developed a novel state space model in which performance is the result of two processes, each driven by its unique error term (Fig. 5B; Eqs. 3 and 4).

$$e_n^{\text{target}}=(r_n-{\widehat{r}}_n)+S_n$$ (3)

$$e_n^{\text{aiming}}=(r_n-{\widehat{r}}_n)$$ (4)

$${\widehat{r}}_{n+1}=A{\widehat{r}}_n+{Be}_n^{\text{aiming}}$$ (5)

$$S_{n+1}={Es}_n+{Fe}_n^{\text{target}}$$ (6)

Target error is directly influenced by the strategy (s). It can be immediately offset by the introduction of a strategy to offset an external perturbation. Aiming error, in contrast, is not directly influenced by the strategy. Rather, it represents a sensory prediction error, defined as the difference between the aiming location and the feedback location (see Taylor and Ivry, 2011 for derivation). While, the aiming error signal is used by an implicit adaptation system to update an internal model (Eq. 5), the target error is used to update the strategy (Eq. 6). As such, the goal, or performance error is used to adjust the strategy. This two-process, two-error signal model was capable of capturing the nonmonotonic learning behavior of the participants, as well as more subtle features such as the relationship between the size of error and the size of the aftereffect (Taylor and Ivry, 2011).

6 CEREBELLAR AND NEOCORTICAL CONTRIBUTIONS TO SENSORIMOTOR ADAPTATION

The aiming task introduced by Mazzoni and Krakauer (2006) provided a tool for observing the operation of two processes, one based on the use of an explicit strategy and the other driven by implicit adaptation of a forward model. Our modeling work suggests that these two processes operate in a concurrent and, to some degree, independent manner. As a further test of this two-process model, we sought to determine if these processes were associated with distinct neural systems by testing patients with different neurological conditions.

As outlined above, many studies have shown that patients with cerebellar pathology are impaired in tasks requiring sensorimotor adaptation. The aiming study offers a novel test of this idea, one in which the patients should actually perform better than controls. Indeed, we set out to test two predictions by comparing the performance of patients with cerebellar ataxia and matched controls on the aiming task. First, we assumed that the patients would have little difficulty using an aiming strategy when given explicit instructions. As such, we expected that, similar to the control participants, they would be able to immediately compensate for a 45° rotation by successfully aiming to a neighboring landmark. Second, and most interesting, we expected that the patients would show attenuated drift given the assumption that the cerebellum is essential for adaptation. As can be seen in Fig. 6, both of these predictions were confirmed (Taylor et al., 2010). After instructed about the strategy, the patients and controls immediately reduced target error. Over the next 80 trials, however, their performance diverged: While the reaches for the control participants showed the characteristic drift pattern (mean maximum drift=11.3°), the patients’ movements remained highly accurate in terms of terminating near the target location (mean max. drift=5.4°). These results provide compelling evidence, not only that the integrity of the cerebellum is essential for adaptation, but that this process is driven by a sensory prediction error. We assume that these patients have difficulty in generating a prediction of the expected outcome of the movement and are therefore unable to update an internal model.

FIGURE 6.

Neuropsychological studies of explicit strategy. (A) A 45° counterclockwise rotation was imposed for trials between the dotted vertical lines and participants were instructed to reach to the landmark clockwise to the target. Patients with cerebellar degeneration (light circles) showed less drift than the Control participants (dark), as well as smaller aftereffects, consistent with predicted impairment in adaptation. (B) Patients with unilateral lesions in the prefrontal cortex (light) showed greater drift than their matched control group (dark), although the aftereffects were similar. This pattern is indicative of an impairment in strategy change with intact adaptation. (C) Lesion reconstruction for patients with prefrontal cortex damage. The reconstructions are overlaid and individually colored for each patient.

The cerebellar results on the aiming task provide a single dissociation, indicating that the adaption component can be selectively disrupted. Stronger neuropsychological evidence for our two-process model requires showing the reverse, namely, that a different group exhibits a selective disruption of the strategic process. To date, the neural systems associated with a strategic process for sensorimotor learning are unknown. One potential candidate is the prefrontal cortex. Classically, the prefrontal cortex is associated with cognitive control, a catchall phrase to encompass processes such as goal representation, planning, and performance monitoring (Miller and Cohen, 2001; Milner, 1963). While these functions are not typically linked to motor learning, a number of neuroimaging studies have shown that the prefrontal cortex, specifically dorsolateral prefrontal cortex, is consistently activated during the sensorimotor adaptation tasks (Floyer-Lea and Matthews, 2004, 2005; Sakai et al., 1998; Shadmehr and Holcomb, 1997). This activation is especially apparent during the early stages of learning (Seidler and Noll, 2008). Indeed, the rate at which participants compensate for visuomotor perturbations during the early stage of learning correlates strongly with the magnitude of the BOLD response in dorsolateral prefrontal cortex activation (Anguera et al., 2010). One hypothesis offered to account for this pattern is that the large performance errors observed early in learning engage spatial working memory, either as part of a monitoring process or to develop compensatory strategies to respond to the perturbations (Anguera et al., 2009, 2010, 2011, 2012).

Indirect evidence supporting a role of prefrontal cortex in sensorimotor learning comes from studies on aging. Older adults consistently show slower learning curves in visuomotor adaptations tasks compared with younger adults (Bock, 2005; Fernandez-Ruiz et al., 2000; Hegele and Heuer, 2010; Hegele and Heuer, 2013; Heuer and Hegele, 2008; Heuer et al., 2011; McNay and Willingham, 1998). Interestingly, these impairments appear to be related to reduced awareness of the perturbation and use of explicit, compensatory strategies (Heuer and Hegele, 2008; McNay and Willingham, 1998). Heuer and Hegele (2008) found that, while younger adults showed less error than older adults when tested on a visuomotor adaptation task, the aftereffects for the two groups were very similar, suggesting that motor adaptation was largely intact in the older adults. To directly test for explicit knowledge of the perturbation, the participants were asked to rotate a ray, originally connecting the start and target position, to an orientation that indicated the direction they should move in order to hit the target. On average, the older adults did not rotate the line as much as the younger adults, suggesting that they had less explicit knowledge of the perturbation. In fact, when this proxy of strategy use was taken into account, the learning curves for the two groups were similar.

Finally, lesions of PFC, including naturally occurring lesions and those transiently induced with TMS, can profoundly impair learning on motor learning tasks (Gomez Beldarrain et al., 1999; Ivry et al., 2008; Pascual-Leone et al., 1996; Slachevsky et al., 2001; Slachevsky et al., 2003). In the case of a visuomotor perturbation, patients with PFC lesions have a complete lack of awareness of the perturbation, even when the perturbation is quite large (Slachevsky et al., 2001, 2003). Furthermore, even when the patients are aware of the perturbation, they have difficulty describing it and, perhaps more importantly, have difficulty reporting what action would be required to compensate for the perturbation.

Taken together, the neuroimaging, aging, and lesion studies provide evidence not only that the integrity of prefrontal cortex is important for motor learning, but also that it may be specifically related to the employment of strategic processes. Motivated by these findings, we recently tested a group of patients with unilateral lesions of lateral prefrontal cortex (LPFC; eight left-sided lesion, two right-sided lesions), on the strategic aiming task. The lesions for these patients are quite variable, both in size and position, but, as can be seen in Fig. 6C, they all encompass LPFC. We selected patients with minimal receptive language problems so that they could understand the strategy instructions. In addition, the majority of the patients did not suffer significant hemiplegia (at least to the degree that they could make reaching movements). For two of the patients, we had to do the testing with their ipsilesional limb; for the others, the task was performed with the contralesional limb.

A deficit in using strategic processes could manifest in different ways. One possibility is that a patient with such an impairment would be unable to implement the strategy or inconsistent in applying the strategy. By this hypothesis, we might expect patients with LPFC damage to show performance functions similar to those observed in typical visuomotor adaptation studies, with a gradual decrease in target error over the course of training. Alternatively, an impaired strategy system might result in amplified drift, with the patients unable to modify their behavior in response to the increasing target error as adaptation builds up.

The results were generally consistent with the latter prediction (Fig. 6B). Patients with PFC damage were able to follow the instruction to use the explicit strategy to counter the rotation. However, on average, they showed increased drift (mean maximum drift=21.8±8.9°) compared to age-matched controls (11.5± 7.4°), although this difference was only marginally reliable (t(17)=1.7, p=0.1). Interestingly, the PFC patients and controls showed similar adaptation as assessed in a final eight trials in which feedback was removed and the participants were asked to stop using the strategy when reaching to the targets. The size of the aftereffect, based on the average of these eight trials was 8.4±2.9 for the PFC group compared to 9.2±3.7° for the controls (t(17)=0.5, p=0.6). Although we need to be cautious in interpreting null results, the results suggest that the LPFC group has difficulty modifying an instructed strategy, even though the size of the target error becomes quite pronounced due to adaptation. From visual inspection, only one of the 10 PFC patients showed an abrupt change in target error during training, the clearest signature of strategy change. In contrast, five of the nine age-matched controls showed large trial-by-trial fluctuations in target error. Failure to modify a strategy could be viewed as a form of perseveration, a common problem observed in individuals with PFC lesions on tests of cognitive control (Heaton, 1981; Milner, 1963). Unfortunately, however, we did not include a sufficient number of training trials with the rotation to be confident in our estimates of overt changes in strategy.

Taken together, the performance of the cerebellar and PFC group on the strategy-aiming task constitutes a double dissociation. By the model outlined in Section 5, the cerebellar group fails to use a sensory prediction error signal to adapt an internal model. In contrast, the PFC fails to use a target error signal to modify a strategy. While these results converge with previous reports using standard sensorimotor tasks, the aiming task offers a cleaner way to isolate these processes, one in which adaptation and strategy change pull the system in opposite directions.

Future work will be required to assess the computational role of other neural regions in sensorimotor learning. One obvious candidate is the basal ganglia given its widely discussed role in skill acquisition (Doyon et al., 2009; Shmuelof et al., 2012). A few studies have examined the performance of patients with basal ganglia pathology on visuomotor adaptation tasks. Individuals with Parkinson’s disease show normal learning rates and sizable aftereffects, suggesting that internal model adaptation is intact (Fernandez-Ruiz et al., 2003; Weiner et al., 1983). However, these patients show reduced savings when retested in a second session, a result interpreted to implicate the basal ganglia in motor consolidation (Bedard and Sanes, 2011; Fernandez-Ruiz et al., 2003; Marinelli et al., 2009). This hypothesis is interesting to consider given the important role of dopamine in reinforcement learning (Fiorillo et al., 2008; Frank et al., 2004; Schultz, 2006; Wise, 2004). If the cerebellum mediates internal model adaptation during learning and conveys this information to cortex (Galea et al., 2011), we can speculate that the basal ganglia helps consolidate this newly formed memory through dopaminergic modulation of cortex (Hosp and Luft, 2013; Koralek et al., 2012).

This hypothesis predicts that variation in the level of dopamine, even in neurologically intact individuals, would affect consolidation following motor learning. For example, participants who experience a greater degree of success during initial motor learning, and presumably, have increased reward-related dopamine release, show greater retention compared to participants who learned more slowly, even if the degree of learning during the initial session was comparable between groups (Trempe et al., 2012). Moreover, a recent study found that rewarding motor performance is critical for the retention and expression of the newly acquired motor memories (Pekny et al., 2011).

These results suggest that the basal ganglia are not involved in internal model adaptation, but contribute to motor learning through their role in consolidation and, perhaps, the expression (selection) of learned movements. It is also possible that the basal ganglia support processes associated with strategy change, perhaps through reinforcement learning. To date, reinforcement-learning models have been designed to look at classical and instrumental conditioning. In terms of adaptation tasks, it is possible that learning driven by target error is dependent on reinforcement learning, a hypothesis that would suggest a direct role of the basal ganglia in sensorimotor learning. Alternatively, the contribution of the basal ganglia may be more indirect, providing a modulatory input to the cerebral cortex. There is clearly a need to test patients with basal ganglia pathology, or employ dopaminergic manipulations in healthy participants, on tasks that provide probes on adaptation, strategy use, and other learning mechanisms.

7 SYSTEMS INTERACTION IN SENSORIMOTOR LEARNING

Our neuropsychological studies provide an example of how cerebellar and cortical learning systems interact to support one form of motor learning. The idea that learning, even for a simple task such as reaching in a perturbed environment, involves the coordinated operation of cortical and subcortical areas is one that has been broadly promoted. For example, neuroimaging studies of skill acquisition consistently show the engagement of a distributed cortical–subcortical network, with many areas showing similar changes in activation patterns over the course of learning (Doyon and Benali, 2005; Keele et al., 2003; Seidler, 2010).

Less clear is what is meant by “interact.” A priori, we tend to assume that these systems operate in an interdependent manner, perhaps with some degree of functional specialization. Our work with the strategy-aiming task, though, requires considering that these different neural systems may operate with considerable independence, reflecting their distinct computational principles. Strategic processes, associated here with the prefrontal cortex, appear to focus on the task goal, using outcome success to evaluate the utility of selected actions. Adaptation processes, associated here with the cerebellum, are concerned with ensuring that an executed movement produces its intended consequences. In standard adaptation studies, these two computations are confounded: the task goal and sensory prediction converge on the same location and we see learning converging in a monotonic fashion toward more accurate movements. From this point of view, it is reasonable to infer that the processes operate in a “coordinated” manner to promote task success.

The strategy manipulation allows these signals to be unconfounded, and importantly reveals considerable independence. While there are a number of limitations of our two-process model, it does make explicit a few important points concerning systems interaction. First, the drift phenomenon makes salient that input to the cerebellum is severely constrained. This system does not appear to have access to information about the task goal; in the strategy-aiming task, this means that the cerebellum does not have access to the strategy. Superficially, this would appear to be a very “dumb” system, imposing corrections to sensory prediction errors even if this undermines successful task performance. However, this “dumbness” reveals an appealing simplicity. It may be advantageous to computationally isolate processes designed to handle action selection and action execution. The cerebellum need not consider whether or not the right action has been selected. It is simply given its marching orders to take a motor command and determine if the sensory outcome of the movement matches expectations. Evaluating whether the motor command was appropriate given a particular task context is deferred to noncerebellar systems.

A second feature of our two-process model is that both the strategy and adaptation processes operate in a continuous manner. The nonmonotonic function evident in Fig. 5, with an initial rise in target error followed by a reversal, might be viewed as indicative of the successive operation of two processes. However, within the framework of our model, this nonmonotonicity is an emergent property, reflecting the fact that the magnitude of the two-error signals changes over time. With the initial application of the strategy, the prediction error signal is large (45°), while the target error signal is small (around 0°). As adaptation occurs, the former becomes smaller and the latter larger. Thus, one need not posit a switch from adaptation to strategy adjustment. Both processes remain operative at all stages of performance. A dynamic tension between the two learning processes is reached, allowing performance to stabilize with reaches successfully terminating near the target location.

Indeed, even when performance becomes asymptotic, the two learning systems continue to operate, pushing performance in opposite directions. As long as sensory feedback appears at a location other than the intended (aiming) location, there will remain a sensory prediction error to drive adaptation. Should this increase target error, an adjustment in the strategy will have the opposite effect. Evidence consistent with this hypothesis is seen in the fact that, with extended training at asymptote, the magnitude of the aftereffect is larger than the magnitude of maximum drift. Drift is constrained by changes in the strategy, whereas the aftereffect provides a probe that is independent of the strategy. One might expect that if participants were trained for an infinite amount of time, adaptation might eventually reach the size of the perturbation. However, even in standard adaptation tasks, performance generally asymptotes at a value less than the perturbation, perhaps because there is some trial-to-trial forgetting of the internal model.

Further evidence that overall performance reflects a dynamic tension between two learning processes comes from experiments in which we manipulated the salience of the sensory prediction error (Taylor and Ivry, 2011). In one condition, the aiming landmarks disappeared as soon as the participant initiated each reach. In another condition, the landmarks were only visible during training blocks (no rotation), used to teach participants how to reach 45° away from a target location. During the rotation+strategy adaptation phase, only the target was visible. In both conditions, participants were successful in compensating for the rotation when given the aiming strategy. However, the manipulations affected adaptation. When the aiming landmarks disappeared, drift was about half the size of that observed with fully visible landmarks. When the landmarks were absent, drift was minimal (see also Benson et al., 2011). Similarly, the aftereffect was reduced in both conditions.

These effects were captured by our model by a single parameter that represented a weight assigned to the sensory prediction error signal, the signal driving adaptation. We note that, in theory, sensory prediction error is based on the difference between predicted and actual sensory information, information that is dominated by the visual modality in studies of visuomotor adaptation (Block and Bastian, 2010; Sober and Sabes, 2005). As such, there is no need for the aiming landmarks if the person can generate a representation of the predicted location of the feedback. However, these results indicate that the landmarks serve as a salient proxy of the predicted location, providing a visible point of comparison with the feedback. When the landmark disappears or is absent, the strength of this signal is weakened.

Our two-process model provides a computational account of how two learning systems interact during motor learning (Taylor and Ivry, 2011). While results from the strategy-aiming task emphasized the need to consider performance as the composite of two, qualitatively different processes, the basic idea of systems interaction has been advocated in many studies of motor learning (Heuer and Hegele, 2008; Heuer et al., 2011; Hwang et al., 2006; Michel et al., 2007; Redding and Wallace, 1996; Redding et al., 2005; Sulzenbruck and Heuer, 2009; Taylor and Thoroughman, 2007, 2008). Indeed, some researchers have argued that multiple learning processes may best be viewed as competitive; for example, prior explicit or declarative knowledge has been shown to interfere with statistical learning (Bonatti et al., 2005; Finn and Hudson Kam, 2008) and performing declarative tasks during or subsequently following motor learning can affect recall (Brown and Robertson, 2007; Keisler and Shadmehr, 2010; Taylor and Thoroughman, 2008). However, in these studies, as well as in our strategy-aiming task, the evidence for multiple learning systems has largely been indirect, measured through changes in learning rate, size of aftereffects, or postexperiment tests of knowledge of the perturbation. As noted previously, postexperimental survey data are notoriously unreliable, especially when adaptation becomes complete. Participants may report they were aware that the environment had been perturbed, but after a few hundred trials, fail to recall if they utilized a strategy to facilitate performance.

We have recently devised a new task to directly assess systems interactions, focusing on the interplay of strategic aiming and adaptation (Taylor et al., 2014). Participants were provided with a continuous array of visual landmarks surrounding the target and were required to report their aiming direction prior to each movement (Fig. 7). During initial training, the aiming requirement must have seemed odd to the participants: They would report the number “0” prior to each reach and then move directly to the target. Our interest, though, was in their behavior once a 45° rotation was introduced. The participants rapidly reduced their target error, with the data revealing that performance was a combination of a change in the aiming direction and adaptation of an internal model. Interestingly, the time courses for these two processes were quite different. Adaptation was slow and proceeded in a monotonic fashion, with the final state of learning during rotation training approximating the size of the aftereffect. Aiming direction, on the other hand, was highly nonmonotonic, exhibiting large fluctuations early in training, before settling into smaller adjustments late in training. The dynamics here provide further evidence of the interaction between two learning processes. The large changes in the aiming report data provided a quick fix to the perturbation. But the solution must be modified over time because adaptation continued to operate. This result again underscores the inflexible and independent nature of the cerebellum, implementing its learning rule even in the face of effective performance.

FIGURE 7.

Experimental task to directly probe strategy utilization. (A) Prior to reaching, the participant is required to verbally report their aiming location (numbered locations). On the critical trials, the feedback location is rotated 45° counterclockwise from the target. (B) Target error for the Instruction group (black) and a control group that was not required to report their aiming location (light). The rotation was present between 56 and 376 (dashed vertical lines). (C) Angle of aiming location reports for the Instruction group. (D) Estimate of internal model adaptation, calculated by subtraction of the aiming direction (in C) from the target error (in B).

Less clear is how to consider constraints associated with strategy change. In our original model (Taylor and Ivry, 2011), we applied a state–space model to account for strategy change, with learning designed to monotonically reduce the target error. That is, we used the same gradient descent algorithm to capture strategy change and forward model adaptation, with the former driven by a target error signal and the latter by an aiming error signal. While this formalism seems consistent with cerebellar-based adaptation, it may not be an appropriate characterization of strategy change. The aiming report data are highly variable, at least in the early stages of learning, suggestive of an exploratory process. Similarly, in our original strategy study (Taylor and Ivry, 2011), some of the individual performance functions revealed categorical-like adjustments to the aiming strategy. Strategy change may require an alternative learning process, one that is more amenable to one-trial learning or some sort of exploratory process.

Exploratory behavior has generally been considered from the perspective of reinforcement learning. Within this general class of models, one distinction has been made between model-free and model-based (Daw et al., 2011; Sutton and Barto, 1998). In model-free reinforcement learning, choices are made by evaluating the expected values for different actions and selecting the option expected to maximize reward (Daw et al., 2011; Haith and Krakauer, 2013; Sutton and Barto, 1998). This may be sufficient to capture the implicit changes in performance observed by Izawa and colleagues (Izawa and Shadmehr, 2011), where people learned to compensate for a gradual visuomotor rotation with categorical feedback. In model-based reinforcement learning (Daw et al., 2011; Sutton and Barto, 1998), the participant develops a representation of the action space. For sensorimotor adaptation, this might be a representation of the relationship between movements of the hand and the cursor. For a given target location, the participant would use the model to select the action expected to counter the rotation. Alternatively, participants may learn to employ a simpler rule or heuristic of the kind “when the error is to the left, go right; when the error is to the right, go left.” Consistent with this hypothesis, we observed a win-stay/lose-shift pattern in the aiming direction time course data, such that the aim was less likely to change on the next trial if the previous trial was successful. Thus, the application of reinforcement-learning ideas to motor learning tasks may require a hybrid model that incorporates these ideas. Upon first encountering the rotation, rapid learning may be facilitated by an exploratory process where different solutions are tested. Later on, a model-based or lose-shift process might become dominant as the participant makes small changes in the strategy to compensate for ongoing adaptation.

While it remains for future work to determine the best formalism of strategy use and strategy change, it is instructive to consider the general applicability of the multiple process idea to sensorimotor adaptation. As a first step toward addressing this question, we tested a group of participants on a standard visuomotor adaptation task (no aiming landmarks, no report) and compared their performance to the group who were provided with landmarks and asked to report their aiming location (Taylor et al., 2014). While forward model adaptation was greater for those tested on the standard task, their aftereffect was considerably less than the actual rotation, despite the fact that performance at the end of the training phase showed minimal error (Fig. 7B). These results suggest that visuomotor adaptation, even in conventional paradigms, entails multiple processes, with adaptation supplemented by an additional aiming “strategy,” even if that strategy may operate at an implicit level (which would suggest that the term “strategy” is a bit of a misnomer).

8 CEREBELLUM AND SENSORIMOTOR LEARNING: BEYOND ADAPTATION

To this point, we have emphasized the critical role for the cerebellum in using sensory prediction errors for adapting a forward model. The drift phenomenon observed in the explicit strategy task highlights the modular nature of this mechanism, making clear that this cerebellar process does not have access to the strategy. However, this result does not need to reflect a general feature of the cerebellum. It would be unwise to treat cerebellar computations as reflecting a single process given the extensive connectivity between this structure and multiple cortical and subcortical regions (Buckner et al., 2011; Krienen and Buckner, 2009; O’Reilly et al., 2010; Strick et al., 2009). Moreover, it may well be that there is an asymmetry in communication between the cerebral cortex and cerebellum. While our work suggests that (some parts of) the cerebellum have little access to cortical representations (e.g., goals/ strategies), cortical representations may be modulated by cerebellar processing. Anatomically, this connectivity has been described as entailing relatively closed loops, with symmetric projections from the neocortex to the cerebellum and vice versa. Functionally, it may be that there is some degree of asymmetry. Our work with the aiming task suggests that (some parts of) the cerebellum have little access to cortical representations; that is, cerebellar adaptation does not appear to be modulated by strategies or goal outcomes. Nonetheless, it may be that cortical representations of these constructs may be modulated by cerebellar processing.

This asymmetry hypothesis is motivated, in part, by one intriguing paradox that comes about from the multiple system perspective. As is made transparent in our studies with aiming targets, performance reflects the joint contribution of adaptation and strategy utilization. If the domain of the cerebellum is restricted to the former, we might expect the patients to rely on nonadaptation processes to compensate for a sensory perturbation. For example, they might overcome a visuomotor rotation by making greater use of a strategy. Indeed, this shift should be facilitated when the rotation is large because the target error experienced by the patients is much larger than that experienced by control participants, especially as training proceeds. This prediction is not supported by the data. In both force-field (Gibo et al., 2013; Smith and Shadmehr, 2005) and visuomotor adaptation (Schlerf et al., 2013; Vaca-Palomares et al., 2013) studies, the patients are similarly affected when presented with small or large perturbations.

These results suggest that the requirement for cerebellar integrity is not limited to adaptation, or that the operation of adaptation, in some manner, constrains the operation of other learning processes. For example, the cerebellum, through its connections with prefrontal cortex, may also contribute to the operation of strategic processes. Various functional hypotheses have been put forward to account for processing within a cerebellar–prefrontal network. Perhaps the predictive capability of the cerebellum extends beyond sensory prediction error. For example, the ability to use semantic information to predict the final word of a sentence is disrupted by rTMS of the cerebellum (Lesage et al., 2012), suggesting a more general role of this system in prediction beyond the sensorimotor domain. Or cerebellar–prefrontal loops may be part of a working memory system, helping maintain action plans such as the current state of the strategy (Spencer and Ivry, 2009) or simulating outcomes for different aiming locations (see Strick et al., 2009).

In a similar vein, the cerebellum may also be a critical node in a reinforcement-learning network. While studies of reinforcement learning have focused on the basal ganglia and orbitofrontal cortex, the BOLD response in the cerebellum has also been found to be correlated with reward prediction error (O’Doherty et al., 2003; Seymour et al., 2004). Moreover, one study showed that patients with focal cerebellar lesions were impaired in reward-based learning tasks, having difficulty in both learning to associate arbitrary objects with different reward values and in reassigning these values when the stimulus-outcome associations were reversed (Thoma et al., 2008).

The computational role for the cerebellum in such tasks remains unclear, but recent studies have identified anatomical projections between the cerebellum and basal ganglia (Bostan et al., 2010; Hoshi et al., 2005). One hypothesis to consider is that this pathway allows the reward prediction system to differentiate between errors in selection and errors in execution. In standard reinforcement-learning models, the failure to obtain an expected reward comes about because the selected object was erroneously over-valued. However, an expected reward will also be missed if the required action is not executed properly. To return to the world of baseball example, consider a pitcher who believes a particular batter cannot hit a curveball. If the pitcher delivers a beautiful curve and the batter hits a home run, the pitcher should use the negative prediction error to update his beliefs, becoming hesitant to throw a curveball when the batter next hits. But what if the home run occurs when a pitched curveball fails to curve? In this context, the pitcher might be wise to ignore the negative prediction error and maintain his beliefs about the batter. We suggest that cerebellar projects to the basal ganglia, and perhaps also to prefrontal cortex, are essential for discriminating between different types of errors. That is, the output from the cerebellum could modulate reward prediction errors, with the occurrence of a sensory prediction error providing a signal to deemphasize a reward prediction error.

This hypothesis offers a novel account for the failure of patients with cerebellar pathology to use strategic processes in a compensatory manner in sensorimotor adaptation tasks. A noisy sensory prediction error system not only impairs adaptation but also removes a modulatory input to the reward prediction error system. Even if the patient develops an appropriate strategy, misreaches resulting from poor adaptation or execution would not be discounted and lead to negative reward prediction signals that diminishes the value of that strategy. These ideas remain to be tested, but offer a computational perspective on how interactions might arise between the cerebellum, cortex, and basal ganglia that build on their unique representational capabilities.

9 CONCLUSIONS

The work reviewed here makes clear that sensorimotor learning is not the result of a single process, but rather involves a multiplicity of different learning processes. We have emphasized that learning, even in simple perturbation studies, can occur at multiple levels: cerebellar-based internal model adaptation can improve action execution, while prefrontal cortex and basal ganglia based processes may improve action selection. A critical question for future study is to determine the extent to which these systems operate in relative independence, or, more generally, describe the manner in which they interact. Nonetheless, it is important to keep in mind that their joint operation is advantageous in terms of hastening overall motor learning, at least when learning is defined as improved task performance.

The current architecture reflects the operation of distinct evolutionary processes, the adaptation of a host of mechanisms that were selected to solve different problems (pardon the pun). In combination, a multipronged learning system can be flexible and produce performance gains over multiple timescales. While the cerebellum can learn to make predictions about the sensory consequences of a selected action to improve motor execution, this process is relatively slow, limited, by stability considerations, to small trial-by-trial increments. Human competence is greatly enhanced by our ability to employ explicit strategies, utilize social learning, and exploit feedback in evaluating different action options.