Teaching the cerebellum about reward

Abstract

Selecting the most rewarding action and performing it accurately are two separable brain functions that are thought to rely upon different neural systems. New evidence suggests that the cerebellum could learn to do both.

The cerebellum is a brain area long known for its critical role in supervised sensorimotor learning¹, i.e. learning to perform an action, like a precisely-timed swing of the racquet, in response to a specific sensory event, like a rapidly approaching tennis ball (Fig. 1). As anybody who has played sports can attest, sensorimotor learning is hard—it can often be nearly impossible—and very frustrating without lots of practice and the help of a good teacher. Two studies published recently in Nature Neuroscience^2,3 have uncovered new, fascinating details about the neurons that teach the cerebellum and about the type of didactic information these neurons provide. The studies suggest that these ‘teacher neurons’ may help the cerebellum learn not only how to perform an action correctly (Fig. 1b) but also how to select which action is the correct one to perform (Fig. 1a).

Fig. 1 |. Hitting the perfect shot.

a,b, To win the point, a tennis player must choose the right shot from their repertoire (a, down-the-line pass or drop shot) and coordinate multiple joint forces (b, shoulder, elbow and wrist) to perform the chosen action correctly. Neurons in the inferior olive provide reward and error-related information that could be used to teach the cerebellum how to improve both action selection (a) and action execution (b).

Previous research has shown that neurons located in the inferior olive act as teachers during sensorimotor learning¹. These neurons, which send axons known as climbing fibers to Purkinje cells of the cerebellum, are activated when the sensory consequences of our actions do not match our own internal expectations, for example, when the tennis ball goes too far, perhaps because we hit it with too much force or because a puff of wind pushed it out. These so called sensory prediction error (sPE) signals are critical for sensorimotor learning; they alert the cerebellum that the motor command for the action just performed did not have the intended consequences and that it needs to be modified. To be useful teachers, however, climbing fibers need to do more than simply notify Purkinje cells that something didn’t go according to plan. Indeed, climbing fiber signals are thought to provide additional error-related information⁴ about the sign of the sPE (for example, was the tennis ball hit too hard or not hard enough?) and about the size of the sPE (for example, did the tennis ball miss the target by a few inches or was it not even close?). All of the information contained in the sPE signal of climbing fibers is perfectly suited to teach the cerebellum which metrics of the motor command need to be adjusted to improve the accuracy and precision of our actions (Fig. 1b).

Now, two new studies have revealed that the role of climbing fibers as teachers of the cerebellum is likely to extend far beyond their well-known contribution to improving motor performance^2,3. Mice were trained to obtain a fluid reward by performing a specific forelimb action, either a properly timed bar release² or a precise steering-wheel movement³. Climbing fibers located in forelimb-controlling areas of the cerebellar cortex were found to modulate their activity in response to the reward, and this modulation was stronger when the reward was surprising or unexpected, for example, during initial training or if the reward was delivered once in a while when the mice were not performing the task. In addition, climbing fibers fired in anticipation of the reward, when forelimb movements were being correctly executed. This pattern of climbing fiber activity resembles reward prediction error (rPE) signals recorded throughout the brain⁵, which are thought to be important for reinforcement learning, i.e., learning to select and reinforce those actions that are most likely to lead to reward (Fig. 1a). However, unlike rPE signals, which are modulated in opposite directions depending on whether the reward was better or worse than expected, climbing fibers do not distinguish between the two cases and seem to simply signal violations in reward expectation regardless of valence.

The discovery of reward-related signals in climbing fibers is a game-changer and poses a serious challenge for classical theories of sensorimotor learning that have neatly divided neural systems into those used to improve action selection and those used to improve action execution⁶. In such a bipartite view, rPE signals generated in midbrain dopamine neurons teach the basal ganglia which action has the most value and is expected to maximize reward (i.e., reinforcement learning), whereas sPE signals generated in the inferior olive teach the cerebellum how to modify motor commands and execute the selected action correctly (i.e., supervised learning). However, a growing body of evidence suggests that the functions of the basal ganglia and cerebellum may not be so easily separable and that the two systems interact via reciprocal connections during goal-directed behavior⁷, in both reinforcement as well as sensorimotor learning tasks^8–11. For example, recent work in mice has revealed a direct monosynaptic pathway from the cerebellar nuclei to midbrain dopamine neurons that plays a key role in certain types of reinforcement learning¹², and reward-related signals have been measured in granule cells of the cerebellar cortex¹³. But the finding that climbing fibers encode parameters related to reward expectation, delivery and evaluation goes much further than previous work in blurring the lines between the neural systems that support action selection and action execution, by demonstrating that the neurons that teach the cerebellum provide the type of instructive signals that are necessary for both.

As is always the case with groundbreaking discoveries, the finding of reward-related signals in climbing fibers raises a number of important questions. First and foremost, it will be important to selectively manipulate these signals and determine what role they play during goal-directed sensorimotor learning and how this role differs from that of rPE signals in other parts of the brain, including in midbrain dopamine neurons⁵. Are the reward signals of climbing fibers necessary and/or sufficient for maximizing the utility of our actions, or do they serve in an ancillary capacity to support or modulate the function of the basal ganglia? Second, there is a growing appreciation for the idea that, in addition to the well-established role of many areas of the cerebellar cortex in motor control and learning, other regions may be important for a number of social and cognitive functions¹⁴. More research is needed to assess whether the reward-related signals of climbing fibers are topographically organized and sent to specific sub-regions or broadcasted throughout the entire cerebellum. Regardless of the amount of work ahead, investigating the role of climbing fibers has never looked so rewarding.