Fig. 3. Certain features of experimental results run counter to predictions of TD.

a Putative temporal basis functions (in red) observed in experiments^22,25 develop over training, shown here schematically. If, after training on a given interval between the conditions stimulus (CS) and the unconditioned stimulus (US), the interval is scaled, the basis-functions also change. Recordings in striatum²² show these basis-functions scale with the modified interval (top), while in recordings from hippocampus²⁵ (bottom), they are observed to redistribute to fill up the new interval. b According to the $TD\left(0\right)$ (temporal difference learning with no traces) theory, RPE neuron activity (blue) during learning exhibits a backward moving bump, from the time of the US to the time of the CS (left). For $TD\left(\lambda \right)$ (TD with trace decaying at a time constant $\mathrm{\lambda }$) the bump no longer appears (right). c A schematic depiction of experiments where there is no observation of a backward shifting bump^28,30. d The integral of dopamine neuron (DA) activity according to TD theory (left) should be constant over training (for $\mathrm{\gamma }=1$, dotted line) or decreasing monotonically for ($\mathrm{\gamma }<1$, blue line). Right, reanalyzed existing experimental data from a trace conditioning task in Coddington and Dudman (2018)³². The horizontal axis is the training trial, and the vertical axis is the mean activity modulation of DA neuron activity integrated over both the cue and reward periods (relative to baseline). Each blue dot represents a recording period for an individual neuron from either the ventral tegmental area (VTA) or substantia nigra compacta (SNc) (n = 96). The black line is a running average over 10 trials. A bracket with a star indicates blocks of 10 individual cell recording periods (dots) which show a significantly different modulated DA response (integrated over both the cue and reward periods) than that of the first 10 recording periods/cells (Significance with a two-sided Wilcoxon rank sum test, p < 0.05). See also Supplementary Fig. 1. Definitions: temporal difference learning (TD), dopamine neurons (DA).