Selecting and Executing Actions for Rewards

Rewards shape human actions. The mere possibility of earning a reward induces substantial improvements in the way we choose and execute actions (Chen et al., 2017). This observation has raised hope for rehabilitation: reward is regarded as a promising means to magnify the positive effects of practice on motor control (Quattrocchi et al., 2017). Yet, this branch of research is only burgeoning, and neuroscientists have yet to identify the mechanisms through which reward improves movements.

At present, two distinct spheres of study have provided insights into how reward improves motor control. First, studies on action selection show that reward can speed up reaction times (RTs; the time elapsed between stimulus presentation and action initiation; Klein et al., 2012) and enhance selection accuracy (subjects select the “right action” more often when reward is at stake; Derosiere et al., 2017a,b). Second, studies on action execution reveal a beneficial effect of reward on movement times (MTs; the time elapsed between action initiation and completion; Reppert et al., 2015) and execution accuracy (e.g., subjects execute faster and more precise movements when reaching to a rewarding target) (Manohar et al., 2019). Strikingly, most work investigating the effects of reward on action selection and execution has examined these effects in separate studies (Chen et al., 2017), impeding the genesis of an integrative understanding of how reward shapes the two processes in more natural settings.

Even when considered in isolation, the precise mechanisms underlying the effects of reward on selection and execution processes have remained obscure. An important gap in our knowledge concerns how reward improves execution accuracy. One possibility is that the presence of reward increases limb stiffness, enhancing the resistance of the moving effectors to internal and external perturbations (Gribble et al., 2003) and ultimately reducing movement variability (so-called “motor noise”; Manohar et al., 2015). Yet, the contribution of stiffness to reward-driven improvements in execution accuracy has been speculative.

In a recent article published in The Journal of Neuroscience, Codol et al. (2020) addressed the two issues mentioned above. In a series of experiments, the authors asked human subjects to use reaching movements to displace a manipulandum from a starting position to one of four target locations. Before starting each movement, subjects were informed of the maximum reward they could obtain in the trial (0, 10, or 50 pence). In 10p and 50p trials, the magnitude of the reward ultimately obtained by the subject depended on her/his performance in the trial (see below).

The first aim of the study was to test the impact of reward on the speed and accuracy of action selection and execution in a single setting. To do so, on 10p and 50p trials, Codol et al. (2020) provided rewards that were inversely proportional to the RT and the MT combined together (reflecting the speed of selection and execution processes, respectively). Reward magnitude also depended on the accuracy of both selection and execution. Importantly, some trials required subjects to ignore distractor cues; initiating a movement toward these cues was classified as a selection error and thus unrewarded. Furthermore, trials on which the final position of the manipulandum fell >4 cm away from the target center were classified as an execution error and thus unrewarded. Hence, to maximize reward in 10p and 50p trials, participants had to select and execute reaching movements as quickly as possible while keeping both selection and execution accuracy high.

A second objective of the study was to test the contribution of limb stiffness to reward-driven improvements in execution accuracy. To investigate this, the authors had subjects perform the same task as described above, with the addition that some trials involved a displacement of the manipulandum after movement completion, pushing subjects' arm away from the target. Arm stiffness was evaluated by measuring the amount of force exerted by the subject during this perturbation. The authors were able to assess the impact of reward on stiffness by comparing this measure of force in 50p versus in 0p trials. In a control experiment, the authors also tested the effect of reward on arm stiffness before movement initiation (i.e., the displacement of the manipulandum pushed subjects' arm away from the starting position).

The results indicated that, when considered in a single task, reward can have a dissociable impact on action selection and execution. Indeed, 10p and 50p trials were not associated with any change in selection speed (i.e., no significant effect on RTs, compared with 0p trials), but entailed a boost of execution speed (i.e., a reduction in MTs). Conversely, selection accuracy was enhanced in rewarded trials (i.e., a smaller proportion of movements were initiated toward distractor cues than on 0p trials), whereas execution accuracy remained unchanged (i.e., the deviation between the manipulandum final position and the target center was stable). Interestingly, computational analyses revealed that the maintenance of high execution accuracy in rewarded trials (despite faster MTs) could be in part attributed to a reduction in motor noise. Most importantly, this reduction in motor noise was associated with a substantial increase in arm stiffness in 50p compared with 0p trials that was observed specifically at the end of the reaching movement (and not before movement initiation), thus confirming the contribution of endpoint stiffness to reward-driven improvements in execution accuracy.

The dissociable effect of reward on selection and execution speeds is striking. Indeed, one major framework in motor neuroscience views selection and execution processes as part of a continuum with a shared neural basis centered on the motor system (Cisek, 2007). In this view, altered activity in specific neural structures (e.g., in the case of reward processing, midbrain dopaminergic neurons; Schultz, 2015) could produce changes in both selection and execution processes at the behavioral level. The roots of this idea lie so deep within the field that researchers often consider RTs and MTs together as a single measure, thought to reflect action vigor (Shadmehr et al., 2019). The findings of Codol et al. (2020) ask us to reconsider carefully this vision, suggesting that, in some conditions, the speed of action selection and execution can be regulated by independent (yet likely interacting) neural structures. Consistent with this hypothesis, a recent study revealed the existence of distinct subpopulations of midbrain dopaminergic neurons, with some cells encoding behavioral choice and others sensitive to movement features (Engelhard et al., 2019).

An alternative explanation for the lack of effect of reward on RTs may arise, however, if one concedes that this measure not only reflects the speed of action selection but also the rapidity of sensory processing (Haith et al., 2016; Vassiliadis et al., 2020), and that reward could have affected these two processes in opposite ways. Indeed, the task described above puts a considerable demand on sensory processing, as it required participants to discriminate between four target locations and, in some trials, to avoid distractor cues. This time-consuming process relies on attentional mechanisms that amplify and suppress neural responses in visual neurons encoding target and distractor cues, respectively (Itthipuripat et al., 2019). The prospect of reward may have strengthened the emphasis on such attentional mechanisms, slowing them down to take more time to sharpen visual activity. Importantly, this interpretation offers a potential mechanistic explanation for how subjects may have improved selection accuracy in rewarded trials. Notably, if such a scenario holds true, the lack of effect of reward on RTs may have emerged from a concomitant, antagonistic hastening of action selection. In this case, the increase in selection speed would be concurrent to the boost of execution speed, and would be therefore in accordance with the continuum framework mentioned above. This hypothesis suggests new avenues of research, aiming to disentangle the effects of reward on the different processes occurring between sensation and action.

Another important finding of the study is that reward reduced motor noise through increased limb stiffness, limiting the potential negative consequence of high execution speed on accuracy. Interestingly, the movement pattern reported by Codol et al. (2020), a parallel increase in movement speed and stiffness during rewarded trials, is similar to that observed when participants are exposed to unpredictable perturbations of their movements during execution (Crevecoeur et al., 2019). This pattern is thought to reflect the implementation of a specific strategy of the motor system (so-called “robust strategy”), minimizing the impact of perturbations on action execution in uncertain environments (Bian et al., 2020). Critically, the results of Codol et al. (2020) suggest that the presence of reward also influences the reliance on such a robust strategy. More generally, the reliance of the motor system on this strategy may depend on the expected outcome of a movement: it increases both when the risk of execution failure is high (i.e., in uncertain environments) and when adequate execution can lead to a reward.

The finding of a reward-driven increase in stiffness has at least two major implications for the development of rehabilitation protocols. First, high stiffness may induce muscular fatigue, a process that might reduce the magnitude of rehabilitative learning (Branscheidt et al., 2019). Therefore, therapists should track patients' fatigue systematically when training involves reward. Second, the ability to regulate limb stiffness could be a relevant marker of whether a patient may or may not benefit from reward-based rehabilitation. For instance, patients with excessive stiffness (e.g., because of poststroke spasticity) may not display the reward-driven improvements in execution reported by Codol et al. (2020), at least not without appropriate antispastic treatment.

In conclusion, the study by Codol et al. (2020) builds on timely questions regarding the mechanisms underlying the impact of reward on motor control. In a series of experiments, the authors show that the presence of reward can have dissociable impacts on action selection and execution, with effects on the latter process associated with increased arm stiffness. As we discussed, these findings provide mechanistic insights and have implications for future clinical translation.