Computations underlying sensorimotor learning

Abstract

The study of sensorimotor learning has a long history. With the advent of innovative techniques for studying learning at the behavioral and computational levels new insights have been gained in recent years into how the sensorimotor system acquires, retains, represents, retrieves and forgets sensorimotor tasks. In this review we highlight recent advances in the field of sensorimotor learning from a computational perspective. We focus on studies in which computational models are used to elucidate basic mechanisms underlying adaptation and skill acquisition in human behavior.

Introduction

Motor learning has traditionally focused on how predictive or feedforward control is updated by past errors and has generally assumed that adaptation is driven by a single process, that variability is undesirable, and that learning and control are shaped by a trade-off between speed and accuracy. Recent research, reviewed below, has begun to examine how learning shapes both predictive and reactive control mechanisms and has shown that adaptation involves multiple, interacting processes, that the motor system tunes variability to the learning task, and that accuracy is not necessarily gained at the expense of speed. Here we review developments in the field over the last two years and refer the reader to recent reviews that cover earlier periods for additional background [1–3].

Time course of sensorimotor learning

Motor memories can be stored and protected over extended periods of time and a major focus of sensorimotor research in recent years has been on the mechanisms of acquisition, interference and retrieval of such memories. Recent models propose that trial-by-trial sensorimotor learning arises from multiple interacting processes which can be approximated by a fast process which learns quickly but also decays quickly and a slower process that learns slowly but also decays slowly [4]. These processes effectively compete to learn, with early task improvement dominated by the fast process with the slower process accounting for the major share of task improvement after more extensive practice. This model is able to account for several key features of motor learning including the rather surprising re-expression of learning that can occur following rapid de-adaptation. Recently this model has been extended by incorporating additional processes with different timescales including an ultra-slow system in prism adaptation that is only activated by prolonged exposure on the order of several hundred trials [5].

The evidence for distinct processes with different timescales prompted many groups to examine whether these processes might be linked to distinct forms of learning. For example, by asking participants to verbally report where they were aiming while adapting to a visuomotor rotation, Taylor and colleagues isolated explicit learning (target relative to aiming location) from implicit learning (aiming location relative to final reach location) components [6*]. An examination of the time courses of these components suggest that the fast process may correspond to explicit learning whereas the slow process may correspond to the implicit learning [7]. Moreover, the two state model, fit to individual participants, can also account for changes in learning seen with aging [8]. Older adults exhibit decreased retention in the slow learning process compared to younger adults. In addition, within the older group, poor explicit memory was associated with reduced retention of both the fast and slow processes. These results suggest that explicit memory resources may relate to impairments in the both the fast and slow processes, but that aging effects on the slow process are independent of explicit memory.

Decay and savings in sensorimotor learning

In accord with state-space models of learning, motor memories decay, independent of error signals, on a trial by trial basis. A recent study claimed to provide evidence that on removal of a perturbation such decay requires the detection of a context change [9]. However, a subsequent study has highlighted problems in analysis and design [10] and shown that decay of motor memory is an implicit process that does not depend on explicit context-change detection. However, the amount of decay for a particular movement has been shown to be linked to the magnitude of a context change (e.g., change in reach direction). Thus, the decay of memory associated with a given reach direction is greatest for movements made in that direction, with memories associated with different directions being protected as they show less decay [11]. Such decay from trial to trial means that there will always be a residual error even after extended learning [12]. A study which amplified viewed errors found that residual error decreased when errors were magnified, consistent with increased trial-to-trial learning [13].

Following adaptation and subsequent deadaptation (i.e., washout) savings can be observed in the form of faster re-learning of the same task. Savings have also been examined without washout, in which case participants can rapidly recall prior learning after just a few trials. The latter kind of savings can be observed with as few as 5 trials of initial learning [14]. Washout can have a dramatic effect on savings. For example, gradual adaption (of which participants are largely unaware) does not lead to saving when it is washed out [15] but results in nearly full saving when it is not washed out [16].

Skills acquisition in sensorimotor learning

Motor learning tasks can be separated into those that require a recalibration of an existing control policy, such as when learning to move in a force-field or under a visuomotor rotation, and skill-learning tasks, requiring the acquisition of a new control policy, such as when learning to trace a specified path [17], to generate force which has a novel relation to muscle activation [18] or to reach under a visuomotor reversal [19*–22]. Teglen and colleagues [19*] found that whereas performance decays between successive rotation learning sessions, gains are observed between successive reversal learning sessions. Moreover, they found that, in contrast to rotation learning, in reversal learning accurate performance requires substantial preparation as reflected by decreasing error with increasing reaction time (see also [20]). These authors therefore proposed that such shifts in time-accuracy tradeoff and offline gains are hallmarks of skill-learning task [19*]. Interestingly when learning a large visuomotor rotation, the expression of learning (i.e., error reduction) is still positively correlated with prolonged preparation time [23] and, after learning, savings are reduced on trials in which participants are only allowed limited preparation [24]. These results suggest that rotation learning can involve elements of skill learning, and it may be that participants attempt to implement an new explicit control policy.

Variability in sensorimotor learning

State-space models initially assumed that learning rates are fixed. However, several recent studies have shown that task features can substantially alter the learning rate. For example, as an applied perturbation becomes more variable, adaptation rates tend to decrease and, moreover, feedback gains within a movement tend to increase [25]. Changes in learning rate are accounted for by a model that suggests the learning rate can be specified for different error magnitudes, with increased sensitivity to recently experienced error magnitudes [26]. Variability in the form of exploration is also important for real world learning and a recent study showed that natural movement variability is positively correlated to the speed with which participants could learn a new motor task [27**]. Conversely, in some tasks variability can hinder performance and it has been shown that participants are able to adapt their strategy dynamically to minimize the effect of variability on task performance [28], using so called risk-aware control [29].

Interference in sensorimotor learning

Numerous studies have shown that there is often substantial interference when people are asked to learn multiple tasks (e.g. opposing force fields or visuomotor rotations) even when contextual cues are used to indicate which task is currently active (e.g. [30]). Recent studies have started to identify key contextual cues that allow multiple motor memories to be learned. For example, it has been shown that the recent state of the limb (i.e. the lead-in to a movement) can be used to separate motor memories, associated with adapting the movement, and that such learning generalizes to novel lead-ins [31, 32]. Moreover, follow-through movements can also engage different motor memories for learning prior to the follow-through [33]. That is, when different follow throughs are associated with opposing tasks, interference is dramatically reduced compared to the same opposing tasks without follow through. Moreover, when learning a single motor task, a consistent follow through led to faster learning compared to the same task with a variable follow though. Together these studies suggest that the lead-in and follow-through to a motor task determines the motor memory where that skill will be stored. Importantly, most natural manual tasks, including object manipulation tasks, involve a sequence of actions that are seamlessly integrated [34, 35], and the above results suggest that surrounding actions may provide state-based context allowing different task parameters to be learned with limited interference.

Reward in sensorimotor learning

Recent studies have also focused on the way punishment and reward can affect learning. For example, it has been shown that reward can accelerate motor learning [36] but that learning is even faster with punishment at the cost of poorer retention compared to reward [37]. Indeed, the willingness of subjects to explore, as indicated by movement variability, appears to be at least partially controlled by recent rewards, with variability decreasing as average reward increases [38]. A recent study found that participants who weigh immediate rewards more heavily than future rewards exhibit faster saccadic eye movements, presumably because they more strongly value the immediate reward of fixating a target [39]. Moreover, reward can act as a motivating influence that can break the typical speed-accuracy trade-off leading to both faster and more accurate movements [40**]. This seemingly paradoxical result that subjects can increase both their speed and accuracy can be reconciled by positing that subjects can exert different levels of control over variability associated with neural noise. However, this comes at a cost, termed the cost of control, which can apparently be manipulated via motivation.

Structures in sensorimotor learning

Motor learning typically involves learning novel kinematic and dynamic transformations between motor outputs and sensory inputs that are determined by the structure of the task. Sensorimotor interactions can share the same structure (e.g. the kinematics and dynamics of scissors) but have different parameters (e.g. scissor length and hinge friction) and one proposal is that learning a structure facilitates generalization to different tasks that share the same structure [41]. For example, people who had never used a computer show less generalization after visuomotor learning compared to computer users, although just two weeks of computer use removed this difference [42]. Recently, other accounts of learning and generalization have been proposed for simple one dimensional tasks whereby participants learn how to respond to the specific visuomotor errors they experience [26] or store memories of specific visuomotor rotations [43]. Although these models may account for some simple features of generalization it is unlikely that they will be able to explain more complex features of generalization that can be accounted for by structural learning [41].

Bayesian processing in sensorimotor learning

The idea that the sensorimotor system uses probabilistic models to makes inferences about unobserved variables (e.g. state of the limb or features of a novel task) in line with Bayesian statistics is well established. For example, in reaching movement people take into account knowledge of their own variability (i.e. prior) in an attempt to optimize reach performance. Recent studies have examined how people represent such priors [44, 45] and suggest that they may be constructed from a mixture of very simple basis functions [46**] and that in some situations there can be substantial mismatches between the represented prior and true variability [47]. In addition, the prior used to represent one’s own errors tends to be narrower than the true distribution of errors and narrower than the priors applied when simply observing errors without action, providing a potential explanation of why we tend to rate our own performance as better than others [48]. When representing priors, the way the mean and the variability of the prior generalize across space appears to be dissociated [49]. A key use of Bayesian inference is to perform model selection to account for data, trading off model complexity against the fit to the data. A recent study showed that humans are in accord with such model selection tending to prefer simpler models (Occam razor) when accounting for data [50].

Rapid motor responses and sensorimotor learning

Optimal feedback control (OFC) provides a unifying principle for motor planning in which a cost of a movement is regarded as a trade off between task success and effort. OFC involves specifying time-varying gains for a set of feedback controllers including rapid motor responses that automatically correct for movement errors that arise through movement variability or task change (for a review see [51]). Such rapid motor responses have been shown to have access to newly acquired knowledge of object dynamics without the need to experience a perturbation with the new dynamics [52*]. Moreover, rapid visuomotor responses can be adapted to have different gains at the same location in space for either different directions of motion of the hand or different perturbation directions suggesting a flexible fractionation of control [53]. Whereas the majority of models focusing on OFC assume rather simple linear systems, recent work has started to try and apply new modelling approaches to capture the nonlinear nature of the true sensorimotor system [54, 55].

Conclusion

Significant recent advances have been made in the field of sensorimotor control through the continued integration of computational modelling and experimental work. Here we have highlighted recent innovative computational work that provides a strong theoretical foundation for different aspects of sensorimotor control and learning, and that is applicable to a wide range of tasks from saccadic eye movements to complex object manipulation. The exciting challenge ahead is to extend and integrate these models so as to understand learning and control of real-world tasks and to link the behavioral and computational processes to their neural implementation.