FIG. 1. Dopamine: updating the past, invigorating the present.

Top, Circles with arrows represent states and the potential actions from those states. Arrow widths indicate learned values of performing each action. As states/actions fade into the past, they are progressively less eligible for reinforcement. Middle, a burst of dopamine occurs. The result is invigoration of actions available from the current state (red), and plasticity of the value representations for recently performed actions (purple). Bottom, as the result of plasticity, the next time these states are encountered their associated values have increased (arrow widths). Through repeated experience reinforcement learning can “carve a groove” through state space, making certain trajectories increasingly more likely. In addition to this learning role, the invigorating, performance role of dopamine seems to speed up the flow along previously-learned trajectories.