A precise and adaptive neural mechanism for predictive temporal processing in the frontal cortex

Summary

The theory of predictive processing posits that the brain computes expectations to process information predictively. Empirical evidence in support of this theory however is scarce and largely limited to sensory areas. Here, we report a precise and adaptive mechanism in the frontal cortex of non-human primates consistent with predictive processing of temporal events. We found that the speed of neural dynamics is precisely adjusted according to the average time of an expected stimulus. This speed adjustment, in turn, enables neurons to encode stimuli in terms of deviations from expectation. This lawful relationship was evident across multiple experiments and held true during learning: when temporal statistics underwent covert changes, neural responses underwent predictable changes that reflected the new mean. Together, these results highlight a precise mathematical relationship between temporal statistics in the environment and neural activity in the frontal cortex that may serve as a mechanistic foundation for predictive temporal processing.

Introduction

Since the early work of Hermann Helmholtz (Peddie, 1925), the field of psychology has embraced the idea that humans rely on prior knowledge to construct a coherent interpretation of raw sensory inputs. Central to this idea is the notion of internal models that represent statistical regularities in the environment (Ito, 1970; Kawato et al., 1987; Werbos, 1987) and help us make better perceptual inferences (Kersten et al., 2004), optimize behavioral responses (Körding and Wolpert, 2004), and swiftly adapt to environmental changes (Courville et al., 2006).

Internal models establish lawful relationships between statistical regularities in the environment and neural signals in the brain (Echeveste et al., 2020; Ma et al., 2006; Vertes and Sahani, 2019). However, the inherent complexities of circuits and signals in the nervous system have made the discovery and concrete characterization of such lawful relationships extremely challenging (Berkes et al., 2011; Walker et al., 2020). One influential hypothesis inspired by information-theoretic accounts of brain function (Attneave, 1954; Barlow, 1961) is that prior statistics enable the brain to process incoming information predictively; i.e., in terms of deviations from expectations (Friston, 2005; Leinweber et al., 2017; Schwiedrzik and Freiwald, 2017). To do so, the nervous system is thought to encode the predictable components of sensory inputs, which are compared to the actual inputs once they become available (Schultz et al., 1997; Wolpert et al., 1998). Predictive processing has been touted as a canonical cortical computation and used to provide a functional account of a wide range of cortical phenomena, including the response properties of neurons in early visual cortex (Rao and Ballard, 1999; Srinivasan et al., 1982) and the logic of laminar information processing in cortical circuits (Bastos et al., 2012; Keller and Mrsic-Flogel, 2018).

The conceptual impact of this theory however has been far greater than evidence in its support. Here, we highlight three major gaps in our current understanding that the present study aims to address. First, high-resolution recordings in animal experiments have largely focused on how individual neurons in sensory areas encode sensory predictions (Berkes et al., 2011; Egner et al., 2010; Meyer and Olson, 2011; Schwiedrzik and Freiwald, 2017). For example, the foundational implications of predictive processing for efficient coding (Barlow, 1961) have only been explored in terms of tuning properties of spatially-tuned sensory neurons (Simoncelli and Olshausen, 2001). As such, evidence for predictive processing in higher-order cortical areas is still wanting (Huang and Rao, 2011). Second, the implications of predictive processing have typically been examined in terms of static firing rates of neurons and not their dynamics. Recent advances, in contrast, suggest that fundamental neural computations might be carried out through dynamic patterns of activity that emerge from interactions among populations of neurons (Chaisangmongkon et al., 2017; Churchland et al., 2012; Egger et al., 2019; Mante et al., 2013; Mastrogiuseppe and Ostojic, 2018; Parthasarathy et al., 2019; Remington et al., 2018; Sohn et al., 2019). Therefore, it is critical that we revisit predictive internal models in terms of population coding and dynamics. Finally, one strong appeal of the theory is that it makes quantitative predictions about the relationship between neural signals and environmental statistics. Experimentally however, it has proven challenging to establish such precise relationships, even for simple statistical properties such as the expected value of a stochastic variable.

Here, we begin to address these outstanding questions in a set of electrophysiology experiments in the frontal cortex of non-human primates. The dorsomedial frontal cortex (DMFC) is strongly implicated in the temporal control of behavior (Chen and Wise, 1995; Lara et al., 2018; Merchant et al., 2013; Mita et al., 2009; Shima and Tanji, 2000; Stuphorn and Schall, 2006). A robust finding across numerous experiments including many of ours is that when monkeys produce time intervals, neural signals associated with the movement initiation undergo temporal scaling, i.e., they stretch or compress in time according to the duration of the produced interval (Egger et al., 2019; Hardy et al., 2018; Mita et al., 2009; Remington et al., 2018; Renoult et al., 2006; Wang et al., 2017). Here, inspired by the temporal scaling associated with movement initiation, we ask whether measurements (as opposed to production) of time intervals also involve temporal scaling, and if so, could such scaling serve as a predictive process to measure time intervals in terms of deviations from prior expectations. Using both old and new data, we uncover a precise mathematical relationship between neural activity and environmental statistics of measured time intervals: while anticipating a future stimulus, the speed at which neural responses evolve over time is inversely proportional to the average time that the stimulus is expected to occur. Moreover, we demonstrate that the speed modulations allow individual time intervals to be encoded relative to their means, i.e., in terms of prediction errors. The lawful relationship between neural speed and temporal expectation was present across multiple independent experiments including a novel adaptation experiment wherein neural responses underwent predictable adjustments in accordance with changes in the mean of the distribution.

Results

Behavioral and neural signatures of predictive processing in the ‘Ready-Set-Go’ task

In a previous study, we trained two monkeys to perform a time interval reproduction task known as ‘Ready-Set-Go’ (referred to as the ‘reproduction task’ in the following) (Sohn et al., 2019) (Figure 1A). The reproduction task requires animals to (1) fixate a central spot, (2) measure a sample time interval (t_s) between two visual flashes (‘Ready’ followed by ‘Set’), and (3) produce a matching interval (t_p) immediately after Set by initiating a saccade (‘Go’) toward a visual target presented left or right of the fixation point. Animals performed this task in two conditions associated with two distinct t_s distributions (Figure 1A, top right). The conditions were cued explicitly by the color of the fixation spot and were interleaved across short blocks of trials (Methods). When the fixation spot was red, t_s was sampled from a ‘Short’ uniform distribution (480–800 ms). When the fixation spot was blue, t_s was sampled from a ‘Long’ distribution (800–1200 ms). The amount of reward monkeys received at the end of each trial decreased linearly with the magnitude of the relative error ((t_p-t_s)/t_s) (Figure 1A, bottom left).

Figure 1. Single neuron signatures of temporal expectations during time estimation.

(A) Ready-Set-Go task. On every trial, the monkey measures a sample interval (t_s) between the Ready and Set cue (visual annulus flashed around the fixation point). After Set, the animal produces a matching interval (t_p) by making a delayed saccade to a peripheral target (Go). Bottom left: reward function. The animal receives reward when t_p is within a fixed window around t_s. Inside this window, the amount of reward decreases linearly with the relative error (t_p-t_s)/t_s. Top right: sample interval distributions. To modulate animals’ temporal expectations about the timing of the Set cue, t_s was sampled from one of two distributions: a Short (red) or a Long (blue) distribution, respectively. The distributions were interleaved in short blocks of trials (length: 4.0 ± 4.4 trials; uniform hazard) and indicated on every trial by the color of the fixation point. (B) Firing rate of six example neurons during the measurement epoch of the task, color-coded by condition. Shaded areas denote 95% CI obtained from bootstrap resampling (N=100). Firing rates were obtained after binning (w_bin=20ms) and smoothing (sd_kernel=40ms) spike counts averaged across trials. (C) Firing rate at the time of Ready (Fr⁰) for the Long versus Short condition. Each circle represents one neuron (N=619 for monkey G, N=542 for monkey H). Fr⁰ was computed in a 20-ms window immediately following Ready. The diagonal distribution shows the difference (Fr⁰_short - Fr⁰_long) across the population of recorded neurons (paired t-test, p=0.40 for monkey G, p=0.07 for monkey H). (D) Change in instantaneous firing rate (dFr/dt) averaged within the measurement epoch for the Long versus Short condition. Each circle represents one neuron. dFr/dt was computed as the absolute difference in firing rate between consecutive 20-ms bins average over the measurement epoch, normalized by the bin size. Results were qualitatively unchanged if the duration of the averaging period in the Long condition was matched to that in the Short condition (i.e., averaging from Ready to Ready+800ms in both conditions). The diagonal distribution shows that the change in firing rate (i.e., ‘speed’) across the population of neurons is significantly higher in the Short compared to the Long condition (paired t-test, p<10⁻¹⁰ for both monkeys).

A salient feature of behavior in this task is that responses regress toward the mean of the interval distribution (Jazayeri and Shadlen, 2010). We found a robust expression of this regression-to-the-mean phenomenon (Sohn et al., 2019) causing opposite biases in the Short and Long conditions for the overlapping 800 ms interval (Figure S1). This observation suggests that animals measure time intervals predictively, i.e., relative to the mean of the temporal distribution. Accordingly, we hypothesized that the neural activity in the measurement epoch of the reproduction task (i.e., between Ready and Set) encodes the mean of the interval distribution.

To test this hypothesis, we analyzed neural activity (N=619 neurons for monkey G, N=542 for monkey H) in the dorsomedial region of the frontal cortex (DMFC; Figure S2), an area that has been strongly implicated in providing temporal control over behavior in humans (Coull et al., 2004; Cui et al., 2009; Halsband et al., 1993; Macar et al., 2006; Pfeuty et al., 2005; Rao et al., 2001), monkeys (Isoda and Tanji, 2003; Kunimatsu and Tanaka, 2012; Kurata and Wise, 1988; Merchant et al., 2011, 2013; Mita et al., 2009; Ohmae et al., 2008; Okano and Tanji, 1987; Romo and Schultz, 1992), and rodents (Kim et al., 2009, 2013; Matell et al., 2003; Murakami et al., 2014; Smith et al., 2010; Xu et al., 2014). During the measurement epoch, neurons had complex and heterogeneous response profiles that typically differed between the two conditions (Figure 1B, S3). Previous recordings in sensory and sensorimotor areas have reported a signature of prior expectations in terms of systematic changes in the baseline firing rate of individual neurons (Basso and Wurtz, 1998; Darlington et al., 2018; Erickson and Desimone, 1999; Hanks et al., 2011; Rao et al., 2012; Sakai and Miyashita, 1991; Schlack and Albright, 2007). Inspired by these previous findings, we performed a variety of analyses looking for systematic firing rate differences between the two conditions. At the level of individual neurons, differences in firing rates changed over time, and were unstructured (Figure S3) with activity sometimes stronger in the Short condition, and other times in the Long condition. Similarly, the normalized difference in firing rates across the population did not systematically differ between the two conditions (Figure S3). We also analyzed the difference between firing rates at the time of Ready, denoted Fr⁰, when the only information available to the animal was the condition type. Although many neurons had different levels of activity depending on the condition, across the population, neurons were equally likely to be more or less active in the Short compared to the Long condition (two-sided paired-sample t-test, t(617)=0.83, p=0.40 for monkey G, t(553)=−1.84, p=0.07 for monkey H, Figure 1C). Based on these results, we concluded that the expectation of the mean is not encoded by systematic firing rate differences across the two conditions.

Moving beyond static firing rates, we considered the possibility that the mean interval might be encoded in how firing rates change over time (i.e., neural dynamics). Recent work has highlighted the importance of neural dynamics in a wide array of tasks (Chaisangmongkon et al., 2017; Mante et al., 2013; Remington et al., 2018; Vyas et al., 2020). For example, it has been shown that neural responses compress or stretch in time according to previously learned temporal contingencies (Mello et al., 2015; Merchant et al., 2011; Mita et al., 2009; Shimbo et al., 2021), a phenomenon referred to as temporal scaling. Scaling has been reported during the planning of delayed movements (Emmons et al., 2017; Maimon and Assad, 2006; Matell et al., 2003; Renoult et al., 2006; Wang et al., 2017; Xu et al., 2014), but also when animals anticipate an external event (Brody et al., 2003; Damsma et al., 2020; Kim et al., 2013; Komura et al., 2001). Neural responses in the reproduction task exhibited a qualitatively similar temporal scaling between the two conditions: the activity profile in the Long condition often appeared as a stretched version of the activity profile in the Short condition (Figure 1B). To quantify this scaling effect, we estimated the average rate of change (i.e., ‘speed’) of firing rate over the measurement epoch for each neuron (Methods). Consistent with our qualitative observation, the speed across the population of neurons was on average larger in the Short compared to the Long condition (two-sided paired-sample t-test, t(617)=12.6, p<10⁻¹⁰ for monkey G, t(545)=12.9, p<10⁻¹⁰ for monkey H; Figure 1D, S3). This finding suggests that although prior expectations were not encoded by an overall modulation of firing rates, they impacted the speed at which neural responses evolved over time.

A subset of DMFC neurons temporally scale their activity patterns according to the mean interval

Although the phenomenon of temporal scaling is not new (Brody et al., 2003; Emmons et al., 2017; Gouvêa et al., 2015; Henke et al., 2020; Kim et al., 2013; Komura et al., 2001; Maimon and Assad, 2006; Matell et al., 2003; Mello et al., 2015; Merchant et al., 2011; Mita et al., 2009; Renoult et al., 2006; Shimbo et al., 2021; Wang et al., 2017; Xu et al., 2014), its functional implication is not understood. Different studies have offered an interpretation in terms of attention (Ghose and Maunsell, 2002), anticipation (Janssen and Shadlen, 2005; Miniussi et al., 1999), or reward expectation (Komura et al., 2001; Platt and Glimcher, 1999). These interpretations however are largely qualitative and do not specify the exact relationship between temporal scaling and experimentally imposed temporal statistics.

Here, we explore an alternative functional interpretation of temporal scaling that is grounded in the theory of predictive processing. According to this theory, neural signals ought to have a precise relationship to the statistics of sensory inputs: neural signals should represent what is expected (Berkes et al., 2011; Rao and Ballard, 1999; Simoncelli and Olshausen, 2001) so that incoming information can be encoded relative to that expectation (Friston, 2005; Leinweber et al., 2017; Schwiedrzik and Freiwald, 2017). Accordingly, we hypothesized that temporal scaling in the Ready-Set epoch is adjusted in proportion to the mean of the interval distribution. We refer to this hypothesis as the mean-predictive-temporal-scaling (MPTS) hypothesis. MPTS makes a specific prediction: the speed at which neural activity evolves over time must be inversely proportional to the average time that the stimulus is expected to occur (faster for earlier, slower for later). Equivalently, the ratio of speeds across different interval distributions should be exactly equal to the reciprocal ratio of the corresponding interval means.

We tested this prediction first at the level of single neurons. If patterns of activity are scaled according to the average interval, one should be able to reconstruct the firing rate of a neuron in the Short condition based on its activity in the Long condition by applying a precise scaling operation (Figure 2A). Let us call $r_{short}\left(t\right)$ and $r_{long}\left(t\right)$ the firing rates of a neuron in the Short and Long conditions, and ${\mu }_{short}$ and ${\mu }_{long}$ the corresponding mean intervals. The MPTS hypothesis predicts that $r_{short}\left(t\right)$ should be well approximated by $r_{long}\left(\lambda t\right)$, where the scaling factor $\lambda$ is equal to the ratio $\frac{{\mu }_{long}}{{\mu }_{short}}$, which in our experiment, is $\frac{1000 ms}{640 ms}\simeq 1.56$.

Figure 2. Neural dynamics encodes the statistical mean of the distribution.

(A) Schematic demonstration of the temporal scaling analysis for individual neurons. We fitted each neuron’s response profile in the Short condition ($r_{short}(t)$, red) by a temporally scaled version of the response in the Long condition ($r_{long}(t)$, blue). We predicted that the optimal scaling factor, $\lambda$, of the fitted response ($r_{long}(\lambda t)$, grey) would be equal to the ratio of the means of the two distributions, ${\mu }_{long}/{\mu }_{short}$. (B) Distribution of scaling factors across neurons in both monkeys (black: all neurons; grey: only neurons whose responses accounted for more than 0.1% of the total variance). Both the black and gray distributions had two peaks, one near unity (black dashed line) and one near the predicted value ${\mu }_{long}/{\mu }_{short}$ (red line). We fitted a Gaussian mixture model to the black distribution (black line) to estimate the location of the two peaks (Methods). The first peak (M1, black triangle) was near unity (1.11±0.33 for monkey G, and 1.07±0.27 for monkey H; mean±sd), and the second peak (M2, red triangle) was near the predicted value (1.60±0.14 for monkey G, and 1.58±0.15 for monkey H). (C) Population dynamics during the measurement epoch. We applied principal component analysis (PCA) to visualize neural trajectories associated with both conditions in the subspace spanned by the top 3 PCs (~75% of total variance explained). Population dynamics described two parallel trajectories evolving at different speeds, as shown qualitatively by the spacing between consecutive states (bin size: 20 ms) along the trajectories (faster for Short, in red, compared to Long, in blue). (D) Schematic demonstration of the temporal scaling analysis across the population. We estimated speed differences between the two conditions by computing the time needed to travel the same arbitrary distance $d$ (black segment) along the Short and Long trajectory ($t_{short}$ and $t_{long}$, respectively), starting from a common reference state (black square). Since immediately after Ready, responses were dominated by a non-specific, likely visually-evoked transient (Figure S3), we chose 400 ms after Ready as the reference state. However, the results were robust to the choice of reference state (Figure S4). (E) For any distance $d$ traveled along the trajectories, we have by definition $v_{short}=d/t_{short}$ and $v_{long}=d/t_{long}$. The ratio of speeds must therefore verify the relationship $v_{short}/v_{long}=t_{long}/t_{short}$. If the ratio of speeds also abides by $v_{short}/v_{long}={\mu }_{long}/{\mu }_{short}$ predicted by the MPTS hypothesis, then it follows that $t_{long}/t_{short}={\mu }_{long}/{\mu }_{short}$, or equivalently, $t_{long}=({\mu }_{long}/{\mu }_{short}{) t}_{short}$. Therefore, MPTS predicts that the mapping between $t_{long}$ and $t_{short}$ should be linear, and the slope of this mapping should be precisely equal to the predicted scaling factor, $\lambda ={\mu }_{long}/{\mu }_{short}$. (F) We plotted the mapping between $t_{long}$ and $t_{short}$ to estimate the empirical scaling factor relating the two neural trajectories. The mapping was linear and diverged from the unity line with a slope close to the predicted value ${\mu }_{long}/{\mu }_{short}\simeq 1.56$ (95% CI for scaling factor: [1.39 1.58] for monkey G, [1.50 1.64] for monkey H). Overlaid with the mapping, the red line shows the prediction constrained to have a slope equal to the value ${\mu }_{long}/{\mu }_{short}$; the intercept is chosen for visualization to minimize the root-mean-squared-error between the prediction and the data. Inset: the slope of a regression model on the empirical mapping (black distribution for bootstrapped values) matches the predicted value (red line) and differs significantly from the null value obtained by randomly shuffling conditions separately for each neuron (grey distribution).

We fitted each neuron’s activity profile in the Short condition with a scaled version of its activity profile in the Long condition (see Methods and Figure S3) and analyzed the fit to $\lambda$ across the population. We found that the distribution of scaling factors across the population was bimodal, with one peak near $\lambda =1$(no scaling), and another peak near the value predicted by MPTS, $\lambda =\frac{{\mu }_{long}}{{\mu }_{short}}$ (Gaussian mixture model, mean±sd, M₁=1.11±0.33, M₂=1.60±0.14 for monkey G; M₁=1.07±0.27, M₂=1.58±0.15 for monkey H; Figure 2B). This bimodality was notable and suggested that the population comprised a mixture of two ensembles, a scaling and a non-scaling ensemble. Scaling neurons adjust their patterns of activity based on the mean interval in each condition, while non-scaling neurons display similar patterns of activity across conditions.

To better understand the observed dichotomy across DMFC neurons, we performed two additional analyses. First, we considered the possibility that the bimodality might reflect the fact that only a subpopulation of DMFC neurons is modulated by the task. That is, task-modulated neurons could contribute to the peak centered at the predicted value, while non-modulated neurons could contribute to the peak centered at one. To test for this possibility, we repeated the scaling analysis albeit only on a subset of neurons which had the largest firing rate modulations during the measurement epoch. Specifically, we sorted neurons by the amount of variance in their firing rate across time points in the measurement epoch and kept neurons which contributed more than 0.1% of the total variance (150/741 neurons for monkey G, 164/617 for monkey H; see Methods). The resulting distribution of scaling factors for this reduced population exhibited the same bimodality as the original distribution (M₁=1.03±0.41, M₂=1.58±0.18 for monkey G; M₁=1.17±0.36, M₂=1.59±0.16 for monkey H; Figure 2B), indicating that the bimodality was not a byproduct of task modulation.

Second, we examined the possibility that scaling and non-scaling neurons might serve distinct computational roles in the different task epochs (i.e., measurement and production epoch). One possibility is that scaling neurons signal the mean interval during the measurement epoch, while non-scaling neurons selectively engage in motor planning during the production epoch. To test this possibility, we categorized neurons into scaling and non-scaling neurons based on their dynamics during the measurement epoch and analyzed their dynamics separately during the production epoch. Previous studies have shown that during the production epoch, the main feature of dynamics is the modulation of speed based on the interval to produce (Renault et al., 2006; Mita et al., 2009; Mello et al., 2015; Wang et al., 2017). We thus set out to determine whether speed modulations during the production epoch were present in both scaling and non-scaling populations. We found that the speed of dynamics in both populations strongly correlated with the interval (Figure S3). This result indicated that scaling and non-scaling neurons did not play distinctive roles across task epochs.

Together, these results provided evidence that, during the measurement epoch, the activity profile of a subpopulation of neurons in DMFC was scaled according to the mean of the interval distribution. Moreover, this result could not be explained in terms of systematic differences in task modulation, or distinct functional roles between task epochs across neurons.

The speed of neural dynamics lawfully reflects the mean of experienced temporal distributions

Because the amount of scaling varied from neuron to neuron, we next sought to quantify the scaling effect at the population level. Following recent practices for high-dimensional neural datasets, we considered neural activity across the entire population as a state evolving in a high-dimensional state space where each dimension represents the activity of one neuron (Cunningham and Yu, 2014; Vyas et al., 2020). We first applied principal component analysis (PCA) to visualize how the dynamics in the two conditions unfolded between Ready and Set (Methods). Qualitatively, results were consistent with MPTS: the speed of neural trajectories appeared faster for the Short compared to the Long condition (Figure 2C).

To test whether the speeds were quantitatively consistent with MPTS, we devised an analysis to measure speed differences between conditions (Figure 2D; see Methods). Briefly, we computed the time ($t_{short}$ and $t_{long}$) necessary to travel the same distance along the Short and Long trajectory, starting from a fixed reference state. If the speed is faster in the Short compared to the Long condition, we expect to have $t_{long}>t_{short}$ for any arbitrary distance. Moreover, if the speed exactly scales with the average interval in each condition, we expect a linear relationship between $t_{long}$ and $t_{short}$, with a slope equal to the predicted scaling factor, $\frac{{\mu }_{long}}{{\mu }_{short}}$ (Figure 2E).

To ensure that our speed estimates were not biased by dimensionality reduction, we performed this analysis in the full state space, i.e., including all neurons. The analysis was particularly suited to assess speed differences at the population level despite the bimodality in the single-neuron scaling factors (Figure 2B). Indeed, if we assume the full state space is divided into two subspaces, one made of the neurons that scale with the mean interval (scaling subspace) and one made of the neurons that do not scale (non-scaling subspace), then by definition, the distances traveled in the non-scaling subspace for the Short and Long conditions are equal, and therefore do not contribute to the final speed differences.

When we applied the speed analysis to the neural data, we found that the empirical scaling factor accurately matched the predicted value (95% CI contained the predicted scaling factor: [1.39 1.58] for monkey G, [1.50 1.64] for monkey H; Figure 2F), both at the level of individual sessions and across animals (Figure S4). As a control, we generated a null distribution for the scaling factor after randomly reassigning the condition type (Short vs Long) for each neuron. In that case, we expected and observed no significant speed differences (95% CI for scaling factor: [0.78 1.26] for monkey G, [0.88 1.14] for monkey H; Figure 2F, inset). Note that because speed differences across conditions emerged ~300 ms after Ready (Figure S4), we focused our analysis on the second half of the measurement epoch. However, results were robust when the analysis was extended over the entire measurement epoch (Figure S4). Moreover, we verified that the relationship between the neural speed and the mean interval held for other values of the mean ratio (Figure S5). Finally, we ensured that the speed differences across conditions were present even on the very first trial following a condition switch (Figure S4). This result indicated that the mere presence of the context cue suffices to modulate the speed of dynamics based on temporal expectations.

Together, these results provide compelling evidence that the speed of neural population dynamics within DMFC is adjusted according to the mean expected interval within each condition and reveal a precise mathematical relationship between temporal scaling and experienced temporal statistics in accordance with the MPTS hypothesis.

DMFC encodes time intervals as deviations away from the mean to serve motor planning

So far, we have shown that the speed of dynamics in DMFC during Ready-Set quantitatively reflects animals’ expectation, i.e., the mean interval of the time distribution. This observation is consistent with the presence of a predictive signal within DMFC but does not explain the nature of time interval representation during the Ready-Set epoch. Consistent with the notion of predictive processing, we hypothesized that time intervals within each distribution are encoded in terms of deviations away from their respective mean (H1) and sought to test H1 against an alternative in which time intervals are not represented relative to their mean, but in an absolute manner (H2).

According to H1, neural responses associated with individual intervals should be organized as deviations away from an invariant neural state representing the prior mean (‘mean state’). To test this prediction, we analyzed the geometry of neural responses in the state space in the vicinity of the mean states associated with the two conditions (see Methods). Specifically, we computed the local tangent at the mean state of one trajectory (reference tangent) and calculated the angle between this reference tangent and the local tangents computed at every state along the other trajectory (test tangent). This cross-condition angular analysis makes qualitatively different predictions for H1 and H2. Under H1, the angle between the reference tangent and the test tangent should follow a U-shape profile: the angle should be smallest when the test tangent is computed at the mean state, and progressively increase when computed at neural states away from the mean (Figure 3A, left). Under H2, in contrast, the angle should monotonically increase as the time difference between the two tangents increases (Figure 3A, right). We performed this analysis twice, using either the Short or Long trajectory as the reference (Figure 3A, red and blue). In both cases, results were unequivocally consistent with H1, and not H2: the angular difference was minimum when tangents were computed precisely at the mean state of each condition (Figure 3A, right).

Figure 3. Encoding of elapsed time relative to expectations.

We used two complementary analyses (A,B) to test whether DMFC dynamics encode time intervals within each condition relative to the corresponding mean (H1), or in terms of absolute interval duration, irrespective of the prior condition (H2). (A) Organization of tangent vectors along the neural trajectories relative to the mean state (the neural state associated with the mean interval). For each trajectory, we computed the tangent at the mean state (dark red and blue arrows in the right panel) and calculated the angle between this tangent and the local tangents computed at every state along the other trajectory (pale pink and purple arrows). For any given state, the local tangent was computed by connecting the states immediately preceding and following that state (bin size: 20 ms). According to H1, the angle between the mean tangent of one trajectory and the local tangents of the other trajectory should display a U-shape profile; that is, the angle should be minimum when the tangents are computed at their respective mean states (left panel). By contrast, H2 predicts that the angle should display a monotonic profile, i.e., increase as the time difference between the two tangents increases (middle panel). The neural data (right panel) were consistent with H1: the angle was minimum when the tangents were computed near the mean states. (B) Organization of neural states associated with each interval relative to the mean states. We defined a hyperplane (grey plane) which contains the average of the two mean states and whose normal vector (black arrow) is the average of the two mean tangents. We then calculated the distance to this hyperplane for every neural state along each trajectory (by convention, the distance was designated as positive when the normal vector pointed in the direction of the state). Finally, we plotted the distance from the hyperplane (ordinate) as a function of elapsed time relative to the mean interval (abscissa: ‘Deviation from mean’) in each condition separately. According to H1, distances should match across conditions (left panel). In contrast, H2 predicts that distances should match for the intervals with the same actual duration. The neural data (right panel) were consistent with H1: distances were similarly organized across conditions. (C) We analyzed the speed of neural dynamics during the production epoch to test whether DMFC encoded the error-from-the-mean (H3) or the actual interval (H4) during motor planning. To do so, we computed the speed of neural dynamics between Set and Go for each interval, and each condition separately. According to H3, speeds corresponding to intervals equally far from their respective mean in different conditions should match (left panel). By contrast, H4 predicts that speeds corresponding to intervals that coincide in terms of absolute times across conditions should match (middle panel). The neural data (right panel) were consistent with H4: DMFC dynamics during Set-Go reflected the actual interval, and not the error-from-the-mean.

Having established that the two neural trajectories are self-similar around their respective mean states, we next examined whether the neural states associated with individual intervals were organized relative to the invariant mean state. To do so, we defined a hyperplane which contained the average of the two mean states and whose normal vector was the average of the two tangents computed at the mean states. We then used this hyperplane to examine how states associated with individual intervals were organized relative to the hyperplane. According to H1, if two intervals from different conditions are equally distant from their respective means, then the associated neural states should also be equally distant from the hyperplane (Figure 3B, left). In contrast, according to H2, states that are equidistant from the hyperplane should correspond to intervals that are equal across conditions (Figure 3B, middle). We tested the predictions of H1 and H2 by computing the distance to the hyperplane of every state along both trajectories. We found that the organization of neural representations at the time of Set were consistent with H1, but not H2: states associated with individual intervals were encoded as deviations away from the mean state (Figure 3B, right). Together, these results indicated that DMFC provides a suitable error signal to encode time intervals relative to their mean, consistent with predictive processing.

Finally, we sought to understand how the error-from-the-mean signal is used during the production epoch (between Set and Go) to serve motor planning. For the animal to produce the desired interval in the production epoch, neural activity in DMFC (or downstream areas involved in motor planning) should ultimately carry information about the actual interval, not just the error-from-the-mean. One possibility is that DMFC combines the error-from-the-mean signal with an explicit representation of the mean itself (encoded along the dimension that separates the two trajectories during the measurement epoch) to rapidly compute the actual interval for the production epoch. Alternatively, DMFC dynamics during the production epoch might only reflect the error-from-the-mean, and not the actual interval to produce. In that case, the computation of the actual interval might happen in downstream areas responsible for initiating the movement at the appropriate time.

To distinguish between these two possibilities, we analyzed neural dynamics during the production epoch. Previous studies have shown that the most notable feature in the production epoch is that the speed of dynamics is faster for shorter intervals, and slower for longer intervals (Renault et al., 2006; Mita et al., 2009; Mello et al., 2015; Wang et al., 2017; Remington el al., 2018). However, this result alone cannot differentiate between the possibility that the speed covaries with the error-from-the-mean (H3) or the actual interval (H4). To distinguish between H3 or H4, we compared speeds across conditions: H3 predicts that the trajectories associated with intervals that are equally far from their respective means in different conditions should evolve at equal speeds (Figure 3C, left). In contrast, H4 predicts that the trajectories associated with intervals that are equal in different conditions should evolve at equal speeds (Figure 3C, middle). A direct analysis of speeds across intervals and conditions rejected H3 and supported H4: the speed of neural trajectories prior to Go encoded the actual intervals, and not their corresponding error-from-the-mean (Figure 3C, right).

Together, these results highlight a relatively simple logic by which DMFC dynamics might support the animals’ timing behavior during the reproduction task: 1) between Ready and Set, the speed of dynamics is adjusted based on the mean, 2) at the time of Set, the speed modulation allows the neural state to encode the measured interval in terms of an error-from-the-mean, 3) from Set to Go, the speed of dynamics is adjusted based on the actual interval to reflect the time of movement initiation.

Predictive temporal scaling is not explained by motor preparation

So far, we have proposed that temporal scaling in the measurement epoch reflects a predictive process encoding the mean t_s. However, since the reproduction task requires t_p to match t_s, one plausible alternative explanation is that scaling is related to the average t_p, and not the average t_s. That is, scaling could reflect the fact that the animal is preparing to produce a short or a long interval (on average), depending on the condition. Indeed, many studies have found that when animals prepare a delayed motor response, the speed of dynamics during the motor preparation period scales with the instructed delay (Maimon and Assad, 2006; Matell et al., 2003; Mello et al., 2015; Merchant et al., 2011; Mita et al., 2009; Murakami et al., 2014; Renoult et al., 2006; Wang et al., 2017; Xu et al., 2014). It is therefore possible that the speed modulations in the measurement epoch are part of a global scaling that spans both Ready-Set and Set-Go epochs and is directly associated with the final motor preparation.

To test this alternative explanation, we analyzed neural recordings in DMFC in a variant of the reproduction task, which we refer to as the ‘gain task’, in which the t_s and t_p distributions were dissociated (Remington et al., 2018). The gain task had the same basic skeleton as the reproduction task: animals had to measure t_s between Ready and Set, and produce t_p between Set and Go. However, the key aspect that differentiated the gain task from the reproduction task was that the two experimental conditions had the same t_s distribution but different t_p distributions. Specifically, in one condition, t_p had to match t_s (same as the reproduction task), while in a second condition, t_p had to be equal to the measured t_s multiplied by 1.5 (gain=1 or 1.5, respectively) (Figure 4A). Accordingly, the gain task offered an ideal opportunity to verify that speed scaling during the measurement epoch was due to the distribution of t_s, and not t_p.

Figure 4. Gain task to rule out motor preparation as an alternative explanation for temporal scaling.

(A) To verify that the neural speed during the measurement epoch of the reproduction task reflected the underlying statistics of t_s, and not that of t_p, we used a gain task to dissociate the t_s and t_p distributions. In contrast to the reproduction task (top), in the gain task (bottom) animals were exposed to two identical sample interval distributions but had to produce either 1 (gain=1, red) or 1.5 (gain=1.5, blue) times the measured interval (Remington et al., 2018). Similar to the reproduction task, the condition was indicated to the animal by the color of the fixation point. (B) Firing rate of four example neurons during the measurement epoch of the gain task (between Ready and Set), color coded by condition. Shaded areas denote 95% CI obtained by standard bootstrapping (N=100). Note the absence of temporal scaling compared to Figure 1B. (C) Scaling analysis at the single neuron level. We performed the same analysis as in Figure 2B on the gain task dataset (N=138 neurons for monkey C, N=201 for monkey J). When we fitted the firing rate of each neuron in the g=1 condition based on a scaled version of its firing rate in the g=1.5 condition, the distribution of scaling factors showed a single peak at one (black dashed line). This indicates that there were no systematic speed differences across conditions at the single neuron level. (D) Population dynamics in the gain task. We plotted neural trajectories associated with both gain conditions in the space spanned by the top 3 PCs (~90% of total variance explained). In contrast to the reproduction task (Figure 2C), there is no apparent speed difference along the trajectories. (E) Empirical scaling factor. Similar to Figure 2F, we measured speed differences along the trajectories by computing the mapping between t_g=1 and t_g=1.5, i.e., the time necessary to travel an arbitrary distance $d$ on both trajectories. We estimated the empirical scaling factor by finding the slope of the mapping and confirmed that both conditions evolved at the same speeds (data lies on the unity line, in agreement with the prediction; 95% CI for scaling factor: [0.87 1.12] for monkey C; [0.94 1.19] for monkey J). Inset: The regression slope of the mapping was not different from the null value (unity) obtained by shuffling conditions across neurons (unpaired t-test on bootstrapped distributions, p>0.05 for both monkeys).

We applied the same set of analyses to DMFC activity (138 neurons in monkey C, 201 in monkey J) in the gain task. At the level of single neurons, activity during the measurement epoch did not exhibit any scaling effect between the two conditions (scaling factor across individual neurons, mean±sd, $\lambda$=0.96±0.14 for monkey C; $\lambda$=1.00±0.24 for monkey J; Figure 4B–C). At the population level, neural trajectories were separated between the two gain conditions (Figure 4D), similar to what we observed between the Short and Long conditions of the reproduction task (Figure 2C). When we computed the speed along the trajectories, however, we found no speed differences between the two gain conditions (95% CI for scaling factor: [0.87 1.12] for monkey C; [0.94 1.19] for monkey J; Figure 4E, S5). This result is noteworthy, given that the only difference between the reproduction task and the gain task was whether or not the distribution of t_s was the same in the two conditions. The presence of precise speed scaling in the reproduction task and lack thereof in the gain task provides clear evidence that speed differences in the measurement epoch reflect the expected value of t_s, and not t_p.

Behavioral adaptation to new temporal statistics

As a causal test for the MPTS hypothesis, we finally sought to assess whether the speed of neural dynamics would change if the animals had to adapt to new temporal statistics. Indeed, one of the strong predictions of the theory is that predictive processing mechanisms should be adjusted flexibly to remain tuned to the current environmental statistics (Benucci et al., 2013; Fairhall et al., 2001; Liu et al., 2016; Rasmussen et al., 2017; Rustichini et al., 2017; Weber and Fairhall, 2019). To test whether neural dynamics would adapt to changes in the statistics of time intervals, we conceived a variant of the reproduction task, which we refer to as the ‘adaptation task’. In the adaptation task, we first sampled t_s from a fixed (‘pre’) distribution (t_s between 660–1020 ms, mean: 840 ms), and then covertly switched to a single (‘post’) interval (t_s = 1020 ms; Figure 5A, inset). Importantly, contrary to the original reproduction task, the change in temporal statistics in the adaptation experiment was not cued in any way. This design guaranteed that neural changes would only reflect changes in animals’ expectation based on the recent history of experienced intervals. Moreover, we chose the post interval to be at one extremum of the pre distribution (i.e., shortest or longest interval) to leverage the bias in t_p to track adaptation behaviorally. Specifically, we predicted that adaptation to a single t_s at either end of the pre distribution would be accompanied by a gradual removal of the bias associated with the reproduction of that t_s.

Figure 5. Behavioral and neural adaptation to changes in temporal statistics.

(A–B) To assess how behavioral and neural responses would be affected by a covert change in temporal statistics, we challenged animals with a variant of the reproduction task (referred to as the adaptation task). In the adaptation task (top), we first sampled t_s from a ‘pre’ distribution (660–1020 ms), and then covertly switched (in this case) to the longest interval of the pre distribution (‘post’). The plot shows the time series of t_p produced by the animal pre and post switch. Each dot represents one trial, and t_p are shown in chronological order, color-coded according to the associated t_s. After the switch (vertical black arrow), t_p gradually adjusted over thousands of trials until the initial bias toward the mean of the pre distribution disappeared. The red line shows the running mean of t_p post switch (window: 300 trials). (C) Following the switch in interval statistics, we hypothesized that neural activity would adjust to reflect the new average interval. Because the mean of the post distribution (${\mu }_{post}$= 1020 ms) was longer than the mean of the pre distribution (${\mu }_{pre}$= 840 ms), the MPTS hypothesis predicted that the speed should slow down by a scaling factor of $\lambda ={\mu }_{post}/{\mu }_{pre}\simeq 1.21$. The plot shows the firing rate of six example neurons pre switch (blue) and early post switch (dark pink). The number of trials pre and early post was matched (~400 trials immediately before and after the switch). Changes in neural responses were qualitatively in line with our prediction: many of the neurons appeared to stretch their activity profiles post compared to pre switch. (D–E) Empirical scaling factor early post switch. Similar to Figure 2F, we measured speed differences between pre and early post activity by computing the mapping between t_pre and t_{post early}, i.e., the time necessary to travel an arbitrary distance $d$ on both trajectories. The resulting mapping diverged from the unity line with a slope close to the predicted value ${\mu }_{post}/{\mu }_{pre}$ (95% CI for scaling factor: [1.09 1.28] for monkey G, [1.11 1.38] for monkey H). Inset: distribution of scaling factors of bootstrapped data (black) and the corresponding null distribution (grey) obtained after randomly shuffling the condition (pre vs post) for each neuron.

We performed this manipulation in the same two monkeys used in the reproduction task. Under the pre distribution, animals showed the characteristic biases toward the mean (Figure 5A–B). When we switched to the post interval, animals’ responses immediately after the switch were indistinguishable from pre switch (Wilcoxon rank sum test, Z=0.99, p=0.32 for monkey G, Z=−1.22, p=0.22 for monkey H), but then gradually adjusted over thousands of trials until responses were no longer biased. This was quantitatively confirmed by comparing the distribution of produced intervals early and late post switch (two-sided unpaired t-test on t_p distributions of the first and last 400 trials after the switch, t(1012)=−9.31, p<10⁻¹⁰ for monkey G, t(778)=−5.92, p<10⁻⁸ for monkey H). This adaptation was robust across animals and across experiments using either the shortest or the longest t_s as the post interval (Figure S1). One potential concern with using a single interval is that animals may choose to ignore the measurement epoch altogether and produce the desired t_p from memory. We performed additional adaptation experiments that included catch trials to verify that animals continued to measure the single interval in the post condition (Figure S1).

Neural adaptation to new temporal statistics

We recorded DMFC activity (92 neurons in monkey G, 50 in monkey H) during a single session where animals experienced a change in interval statistics. Tracking changes of neural activity during adaptation is challenging because the recordings need to be stable throughout the adaptation process and be high-yield so that we can estimate neural states based on small numbers of trials during adaptation. Therefore, as a prerequisite, we developed a custom recording approach that ensured the required stability and high-yield (see Methods and Figure S2).

To examine the neural correlates of adaptation, we first compared neural activity before and after the change in temporal statistics to test whether the relationship between pre/post activity was consistent with the predictions of the MPTS hypothesis. Specifically, we first focused on neural activity in a 400-trial window immediately before (‘pre’) and after the switch (‘post’).

Similar to what we had done for the reproduction task (Figure 1C), we tested if the new temporal statistics had any consistent effect on the baseline firing rate of individual neurons. For the majority of neurons, mean firing rates over the entire measurement epoch had changed as a result of adaptation (72% (66/92) for monkey G, 58% (29/50) for monkey H, Figure 5C). However, similar to the reproduction task, the direction of change was not systematic across neurons: on average, neurons were equally likely to increase or decrease their baseline firing rate between pre and post (two-sided paired t-test, t(91)=0.21, p=0.83 for monkey G, t(59)=−2.56, p=0.01 for monkey H; Figure S6). This result indicated that adaptation modulated the firing rates but the nature of the effect across neurons was not consistent or systematic.

Next, we sought to directly assess whether the activity changes between pre and post were consistent with the temporal scaling of neural responses in proportion to the new mean interval (MPTS hypothesis). Given the mean of the post and pre distributions (${\mu }_{post}$= 1020 ms and ${\mu }_{pre}$= 840 ms), MPTS predicts that the speed should slow down by a factor of $\lambda =\frac{1020 ms}{840 ms}\simeq 1.21$. The slowing down of dynamics was already evident at the level of single neurons (Figure 5C), many of which appeared to stretch in time after the switch (note for instance the peak firing rate of unit #H_3105 in Figure 5C which has shifted to a longer time). To test the prediction of MPTS quantitatively, we estimated speed differences at the population level between pre and post. Using the same speed analysis used in the reproduction task (Figure 2F), we found that the speed was adjusted to its predicted value robustly across the two monkeys (95% CI for scaling factor: [1.09 1.28] for monkey G, [1.11 1.38] for monkey H, Figure 5D–E). The data also rejected the alternative hypotheses according to which the speed might reflect either the shortest or the longest interval of the distribution. Indeed, the latter predicts that the speed should not have changed between pre and post ($\lambda =\frac{1020 ms}{1020 ms}=1$) while the former predicts a scaling factor of $\lambda =\frac{1020 ms}{660 ms}\simeq 1.55$. Both values were not included in the 95% confidence interval of the empirical scaling factor. Neural changes were thus fully consistent with animals adjusting their temporal expectations to the new distribution mean. This finding strengthens the validity of MPTS as it establishes a causal link between the interval mean and the speed of neural dynamics.

Neural changes precede behavioral changes during adaptation to new interval statistics

At this point, we have established a tight mathematical relationship between changes in temporal statistics and neural adaptation: when the interval distribution changes, the speed of dynamics in DMFC is adjusted to quantitatively reflect the new mean interval. One important remaining question is whether the neural changes are caused by the new interval statistics or whether they reflect changes in the behavior, for example, through reafference (Figure 6A). If behavioral adaptation were to precede speed adjustments, we would not be able to reject the possibility that the neural changes in DMFC are a consequence of behavioral adaptation. In contrast, if speed adjustments were to precede behavioral adaptation, we would be able to conclude that neural changes are caused by the introduction of the new temporal statistics. To distinguish between these two possibilities, we examined the timescale of speed adjustments in DMFC to that of behavioral adaptation.

Figure 6. Timescale of neural and behavioral adaptation.

(A) In the adaptation experiment, we found both behavioral and neural changes. One possibility (correlational path, in grey) is that neural changes were the indirect result of behavioral changes. For instance, changes in animals’ responses during adaptation to the new temporal statistics could affect neural activity through reafference and accumulate over trials to lead to persistent neural changes. Alternatively (causal path, in black), changes in statistics may have directly caused the observed neural changes, independent of behavioral adaptation. To dissociate between these two possibilities, we compared the timescale of neural and behavioral changes. Neural changes preceding behavioral changes would be consistent with the causal path, while behavioral changes preceding or co-occurring with neural changes would favor the correlational path. (B–C) To assess the timescale of neural adaptation following the switch in interval statistics, we examined how the empirical scaling factor changed in a 100-trial window (50% overlap) running throughout the session. To perform this analysis, we first computed a reference trajectory (traj_ref) obtained by combining all trials (pre and post switch). We then computed the neural trajectory associated with each 100-trial block (traj_block). To estimate the scaling factor associated with a particular block, we plotted the mapping between t_block and t_ref, corresponding to the time needed to travel an equal distance $d$ along traj_block and traj_ref, respectively. We calculated the slope of this mapping (obtained via linear regression) for each block and normalized it by the average slope across all pre switch blocks. Pre switch slopes were thus expected to lie around unity, and subsequently reach their predicted value ($\lambda ={\mu }_{post}/{\mu }_{pre}$; red line). The data was in agreement with this prediction and showed that the speed rapidly adjusted to its predicted value. Shaded areas denote 95% CI obtained from bootstrapping. (D–E) A direct comparison of scaling factors (left) and produced intervals (right) pre switch (blue arrow in panel B and C) versus post switch (orange arrow) show that neural changes preceded behavioral changes. Roughly 200 trials following the switch, the neural speed had already adjusted (95% CI for scaling factor in the trial window, [1.13 1.44] for monkey G, [1.04 1.46] for monkey H), while animals’ behavior was virtually unchanged compared to pre switch (two-sided unpaired t-test on tp distributions, t(167)=0.97, p=0.33 for monkey G, t(162)=−0.38, p=0.71 for monkey H).

Our initial analyses already provided evidence that neural activity had already changed ~400 trials after the switch (Figure 5C). To quantify the timescale of neural changes rigorously, we computed how the scaling factor changed throughout adaptation using a 100-trial window running over the entire session (Methods). With this finer-grained analysis, we were able to confirm that ~200 trials post switch, the population speed was significantly different from pre switch and not significantly different from its final predicted value (95% CI for scaling factor, [1.13 1.44] for monkey G, [1.04 1.46] for monkey H; Figure 6B–E). By comparison, animals’ behavior in the same post switch trial window was statistically indistinguishable from pre switch (two-sided unpaired t-test on t_p distributions, t(167)=0.97, p=0.33 for monkey G, t(162)=−0.38, p=0.71 for monkey H, Figure 6D–E). This observation rules out the possibility that neural changes were purely driven by behavioral changes, and further supports the notion that changes in the neural speed reflected the dynamic adjustments of animals’ temporal expectations.

Discussion

Predictive processing is among the most influential theories in neurobiology. Critical to this theory is the notion that the nervous system represents statistical regularities in the environment in the form of an internal model (Ito, 1970; Kawato et al., 1987; Werbos, 1987). The internal model encodes the predictable component of sensory inputs such that only residual errors from a priori predictions undergo further processing (Bastos et al., 2012; Keller and Mrsic-Flogel, 2018; Rao and Ballard, 1999; Srinivasan et al., 1982). However, the degree to which this theory can be applied broadly to explain neural mechanisms is still an active area of research (Huang and Rao, 2011). Our work extends and strengthens the case for this theory in a few important directions. First, current accounts of predictive processing are typically limited to modulations of static firing rates and thus cannot be straightforwardly extended to scenarios in which the information processing involves dynamics (Basso and Wurtz, 1998; Darlington et al., 2018; Erickson and Desimone, 1999; Hanks et al., 2011; Rao et al., 2012; Rasmussen et al., 2017; Sakai and Miyashita, 1991; Schlack and Albright, 2007; Schultz et al., 1997). Our work identifies a signature of predictive processing directly based on adjustments of neural dynamics. Second, evidence for predictive processing has been largely limited to early sensory areas (Berkes et al., 2011; Egner et al., 2010; Meyer and Olson, 2011). Our work serves as an example of how this theory may explain neural observations in higher-order brain areas. Finally, and perhaps most importantly, our work provides a precise mathematical formulation of how neural signals precisely reflect the mean of temporal statistics.

The idea that the nervous system adjusts to the statistics of sensory inputs is not new (Attneave, 1954; Barlow, 1961; Brenner et al., 2000; Młynarski and Hermundstad, 2021). In line with this idea, a variety of adaptation phenomena have been robustly observed across sensory domains and species (Dean et al., 2005; Díaz-Quesada and Maravall, 2008; Fairhall et al., 2001; Kim and Rieke, 2001; Maravall et al., 2007; Nagel and Doupe, 2006; Smirnakis et al., 1997). In the temporal domain, in particular, previous studies have demonstrated that neural dynamics is systematically adjusted to the timescale of inputs (Fairhall et al., 2001; Hosoya et al., 2005; Lundstrom et al., 2008). Further, neural changes following input statistics have been linked to predictive functions of the nervous system (Millman et al., 2020; Schwartz and Berry, 2008). Our results add to the existing body of literature by revealing a signature of predictive processing through the adjustment of neural dynamics based on experienced temporal statistics. In particular, we hypothesized and found compelling evidence that when awaiting a future event that can occur at different times, neural responses speed up or slow down in accordance with the statistical mean of the expected times. Converging evidence from several experiments made a strong case for this interpretation. First, the distribution of temporal scaling factors across single neurons in DMFC revealed a sharp and distinct peak at the value predicted by our hypothesis. Second, modulations of the speed of dynamics estimated by an unbiased analysis of neural trajectories across the population of neurons in individual animals and experimental sessions quantitatively matched the predictions of our hypothesis. Third, a control experiment verified that the modulation of speed was specifically related to the anticipated distribution of sensory events and not the ensuing motor response. Finally, a novel adaptation experiment confirmed that DMFC dynamics is adjusted when environmental statistics change, and that the associated neural changes can be quantitatively predicted. Importantly, the adaptation experiment revealed a causal relationship between the temporal statistics of the environment and the speed of anticipatory neural dynamics.

Speed changes between the two prior conditions in the original reproduction task can, in principle, be driven by two mechanisms. First, these changes may result from rapid reconfigurations of neural dynamics in response to context-dependent external inputs (prior-dependent color of the fixation point), as we and others have previously reported (Mante et al., 2013; Remington et al., 2018; Sohn et al., 2020). Alternatively, speed changes could be due to an adaptive process that rapidly infers the prior condition (and its corresponding mean) based on observed sample intervals, similar to the meta-adaptation that occurs after repeated exposures to alternating environments (Robinson et al., 2016). Our analysis of speed following condition switches revealed that speed changes occurred immediately after the switch (Figure S4). This result indicates that speed changes were set by the external cue, and not through a meta-adaptive process. However, it is likely that speed underwent additional trial-by-trial refinements in accordance with the local fluctuations of the animals’ internal estimate of the mean. Moreover, these fluctuations likely contributed to the overall variability in the animal’s behavior, even though the dominant source of variability in this task is the scalar noise associated with production of time intervals (Jazayeri and Shadlen, 2010, 2015; Wang et al., 2017, 2020). Indeed, the gradual refinements of the animals’ estimate of the mean was a key signature of behavior during the adaptation experiment and was associated with predictable changes in the speed of neural dynamics. Comparing the reproduction and adaptation tasks, it is also remarkable that behavioral adjustment across two widely different timescales, one involving rapidly changing contextual cues, and another involving slow adaptation, would rely on the same predictive speed-control mechanism. This observation suggests a shared neural basis for experience-dependent and context-dependent adaptive sensorimotor behavior.

Like most discoveries, ours raises more questions than it answers. Here, we highlight some of the most pressing ones. First and foremost, what does the lawful relationship between neural speed and mean interval imply as a computational algorithm? This relationship establishes a form of invariance in the neural space: it guarantees that when the brain measures elapsed time, the neural trajectory may reach a fixed desired state at the expected mean interval irrespective of the underlying distribution. Consistent with this idea, we found evidence in our data that neural population activity reaches an invariant state across conditions at the mean interval and that a linear readout could decode deviations around this state to measure elapsed time relative to the mean (Figure 3A–B). Invariances of this kind are important as they could be the manifestation of a latent coding space that confers flexibility and generalization (Behrens et al., 2018; Shepard, 1987). In the context of our finding, the brain may leverage this invariant distance metric to make relational inferences that generalize across contexts. For example, this organization would cause an interval that is one standard deviation away from the mean to be mapped onto the same neural distance to the mean irrespective of the distribution. Under suitable assumptions about noise, this scaling effect may provide a natural explanation for the scalar variability in time interval judgments (Rakitin et al., 1998). Similarly, this invariance may allow a criterion for categorical judgments to readily generalize to new stimuli (Sheahan et al., 2021). In our own work, the rapid adjustments of the neural speed in the adaptation task may have benefited from this representational invariance to perform rapid directed explorations in the neural space (e.g. meta-learning) (Gershman and Niv, 2010; Kumaran et al., 2016; Sohn et al., 2020; Tenenbaum et al., 2011; Wang et al., 2019; Wilson et al., 2014). Specifically, modulations of speed based on the mean may serve to map time distributions onto a fixed range of desired states (Atick, 1992; Fairhall et al., 2001; Padoa-Schioppa, 2009; Rabinowitz et al., 2011; Rasmussen et al., 2017; Rustichini et al., 2017), such that changes in the mean would result in systematic prediction errors away from the desired states which in turn can drive efficient adaptive changes through directed exploration (Friston, 2005). Building on these observations, future work could examine whether this invariance is present and how it may facilitate rapid adaptation in other domains such as speed-accuracy tradeoff (Palmer et al., 2005; Simen et al., 2016) or the temporal control of attention (Cadena-Valencia et al., 2018; Miniussi et al., 1999).

Juxtaposing our finding relating temporal scaling to the mean of an interval distribution with previous work reporting temporal scaling during time interval discrimination (Mello et al., 2015; Mendoza et al., 2018; Xu et al., 2014) and production (Damsma et al., 2020; Hardy et al., 2018; Henke et al., 2020; Merchant et al., 2011; Mita et al., 2009; Murakami et al., 2014; Remington et al., 2018; Sohn et al., 2019; Tanaka, 2007; Wang et al., 2017) raises the question of whether these phenomena are computationally related. We propose that all these observations may be unified under the theory of predictive processing, and the only thing that differentiate between them is the nature of what is being predicted. In our work, animals had to estimate a time interval, and thus it is natural for the system to predict the expected value of the distribution. In time interval discrimination task, scaling may accommodate the measurement of test intervals relative to the criterion. Finally, in motor timing tasks, scaling may serve as a prediction for the time of reward, which might be essential for reward-based learning and incorporating delay discounting information into reward prediction errors (Kim et al., 2008; Kobayashi and Schultz, 2008). Scaling during motor planning may also play a role in predicting the sensory consequences of actions, which is thought to be an integral part of motor control (Wolpert and Miall, 1996). Our findings thus point to a unifying functional explanation of the phenomenon of temporal scaling commonly observed in timing tasks across species and brain areas (Henke et al., 2020; Maimon and Assad, 2006; Mello et al., 2015; Merchant et al., 2013; Mita et al., 2009; Renoult et al., 2006; Shimbo et al., 2021; Wang et al., 2017; Xu et al., 2014).

For the adaptation experiment, we set the post-adaptation distribution to a single value at the extrema of the pre-adaptation distribution. This choice enables us to quantify behavioral adaptation more easily and examine its neural underpinnings within a single session and under a fixed sensory input. However, a potential caveat of this choice is that the pre and post adaptation distributions differed both in terms of mean and variance. This raises the possibility that certain aspects of our findings in the adaptation experiment were caused by an adaptation to variance. While we cannot rule out this possibility, we note that the relationship we found between the speed of neural dynamics and the mean was replicated across a range of experimental manipulations including cued prior switches in the original reproduction task with multiple pairs of distributions (Figure 2), cued gain changes in the gain experiment (Figure 4), as well as the adaptation experiment (Figure 5). We therefore are confident that our conclusion about the relationship between mean and speed holds regardless of how variance might impact neural responses. We also note that previous work on a task similar to ours has shown that behavioral adaptation to variance and mean have different timescales (Sohn and Lee, 2013). This difference was also evident in a pilot experiment where we quantified the time course of adaptation to variance. We collected behavioral data in an experiment in which the prior changed covertly from a uniform distribution with 5 intervals to a single interval at the middle of that distribution. Critically, this manipulation changes the variance of the distribution without changing its mean and can thus reveal the behavioral changes associated with adaptation to a new variance. We found that adaptation to variance was significantly slower than adaptation to the mean; it took several days for the animal to adapt to the new variance (Figure S6). Together, these findings suggest that adaptation to variance might rely on other slower mechanisms that may or may not be predictive.

However, many important questions about the relationship between neural dynamics and prior distributions remain unanswered. For example, there is great interest to understand how neural responses encode higher order features of the distribution such as multimodality, variance, and skewness. Addressing these questions requires comparison of neural responses between sample distributions that are carefully tailored to the question of interest. Currently, there is no definitive answer for how neurons encode variance although recent findings have suggested a functional role for nonlinear representations (Sohn et al., 2019; Vertes and Sahani, 2019). Beyond the variance, it is also not known how the brain represents skew in a distribution even though humans can be exquisitely sensitive to it (Motoyoshi et al., 2007). For skewed distributions, the scaling may continue to reflect the mean or may instead represent another statistic such as the mode of the distribution (Stocker and Simoncelli, 2006). For multimodal distributions, the mean interval may no longer be a good predictor. For example, for a bimodal distribution with two distinct modes, the system might adjust the speed to predict the two modes sequentially (Janssen and Shadlen, 2005; Zimnik and Churchland, 2021) or use different groups of neurons (or subspaces) to generate predictions in parallel (Glaser et al., 2018; Kang et al., 2020; Lorteije et al., 2015). Finally, it is important to extend our findings to more elaborate sensorimotor tasks that involve both spatial and temporal uncertainty such as interception tasks (Chang and Jazayeri, 2018; Kwon and Knill, 2013; Merchant et al., 2004), intuitive physics tasks (Battaglia et al., 2013; McIntyre et al., 2001), or more complex agent-based pursuit tasks (Gulli et al., 2020; Kuhrt et al., 2020; Yoo et al., 2020). These extensions will be critical for developing a comprehensive theory of how the nervous system learns spatiotemporal error distributions (Herzfeld et al., 2014) and use that information to update internal predictions.

Finally, our adaptation experiment revealed that the nervous system quickly updates internal predictions following environmental changes, but lags in its ability to respond to these changes. A common and implicit assumption in sensorimotor adaptation experiments is that learning-dependent changes in the neural activity must occur in register with changes in behavior. With this assumption in mind, the difference in timescales of learning between the neural and behavioral data is surprising. However, this assumption may not hold true in general. For example, while playing squash, one might notice that the ball is too soft after only a few shots. Nonetheless, an hour of practice might be needed to adjust one’s shots to the new ball. In other words, certain aspects of learning may advance long before they can be incorporated into behavior. This two-stage learning process is a notable feature of control systems that rely on forward and inverse models (Jordan and Rumelhart, 1992), and has been documented in various motor adaptation experiments (Flanagan et al., 2003; Pierella et al., 2019).

Accordingly, we sketch a speculative hypothesis for how adaptation in our experiment might unfold in two stages. In the first stage, following the change in the prior distribution, fast neural changes adjust the predictive process that encodes the new mean. In the second stage, the system incorporates information about the new mean to gradually update the ensuing motor command responsible for interval production. Indeed, we found that the firing rate of some neurons in DMFC changed more slowly and in keeping with behavioral changes (Figure S6). The observation that some neurons change rapidly while others change more slowly and in register with behavior provide tantalizing evidence in support of our speculative hypothesis. However, our current experiment and data are not sufficient to definitively test the main idea of two-stage adaptation whereby a fast predictive process is thought to serve a teaching role in adjusting the slower process that controls behavior (Jordan and Rumelhart, 1992). Additional experiments will be needed to test this hypothesis. Our work nevertheless sets the stage for a direct experimental test of this hypothesis and provides a platform for understanding the role of predictive processing in sensorimotor adaptation.

STAR Methods

CONTACT FOR REAGENT AND RESOURCE SHARING

Further information and requests for resources and reagents should be directed to and will be fulfilled by the Lead Contact, Mehrdad Jazayeri (mjaz@mit.edu).

EXPERIMENTAL MODEL AND SUBJECT DETAILS

All experimental procedures conformed to the guidelines of the National Institutes of Health and were approved by the Committee of Animal Care at the Massachusetts Institute of Technology. Experiments involved four awake behaving monkeys (macaca mulatta; ID: G and H for the reproduction task and the adaptation task; weight: ~8 kg; age: 5 years old; ID: C and J for the gain task). Animals were head-restrained and seated comfortably in a dark and quiet room and viewed stimuli on a 23-inch monitor (refresh rate: 60 Hz). Eye movements were registered by an infrared camera and sampled at 1kHz (Eyelink 1000, SR Research Ltd, Ontario, Canada). The MWorks software package (https://mworks-project.org) was used to present stimuli and to register eye position. Neurophysiology recordings were made by 1 to 3 24-channel laminar probes (V-probe, Plexon Inc., TX) through a bio-compatible cranial implant whose position was determined based on stereotaxic coordinates and structural MRI scan of the animals. Analyses of both behavioral and electrophysiological data were performed using custom MATLAB code (Mathworks, MA).

METHOD DETAILS

Behavioral task

RSG trial structure

Monkeys performed a time interval reproduction task known as the ‘Ready-Set-Go’ task (Jazayeri and Shadlen, 2015; Sohn et al., 2019). Each trial began with animals maintaining their gaze on a central fixation point (white circle: diameter 0.5 deg; fixation window: radius 3.5 deg) presented on a black screen. Upon successful fixation, and after a random delay (uniform hazard; mean: 750 ms, min: 500 ms), a peripheral target (white circle: diameter 0.5 deg) was presented 10 degrees left or right of the fixation point and stayed on throughout the trial. After another random delay (uniform hazard; mean: 500 ms, min: 250 ms), the Ready and Set cues (white annulus: outer diameter 2.2 deg; thickness: 0.1 deg; duration: 100 ms) were sequentially flashed around the fixation point. Following Set, the animal had to make a proactive saccade (self-initiated Go) toward the peripheral target so that the produced interval (t_p, between Set and Go) matched the sample interval (t_s, between Ready and Set). The trial was rewarded if the relative error, |t_p-t_s|/t_s, was smaller than 0.2. If the trial was rewarded, the color of the target turned green, and the amount of juice delivered decreased linearly with the magnitude of the error. Otherwise, the color of the target turned red, and no juice was delivered. The trial was aborted if the animal broke fixation prematurely before Set or did not acquire the target within 3t_s after Set. After a fixed inter-trial interval, the fixation point was presented again to indicate the start of a new trial. To compensate for lower expected reward rate in the Long condition due to longer duration trials (i.e., longer t_s values), we set the inter-trial intervals of the Short and Long conditions to 1220 ms and 500 ms, respectively.

Reproduction task

In the reproduction task (Figure 1–3), the sample interval, t_s, was sampled from one of two discrete uniform distributions, with 5 values each between 480–800 ms for Short, and 800–1200 ms for Long. The distributions alternated in short blocks of trials (min of 5 trials for G, 3 trials for H, plus a random sample from a geometric distribution with mean 3, capped at 25 trials for G, 20 trials for H), and was indicated to the animal by the color of the fixation point (red for Short, blue for Long). On half the trials, and only in this first experiment, monkeys were required to acquire the peripheral target (at Go) using a hand-held joystick (‘hand trials’) instead of a saccade (‘eye trials’). However, for consistency with the other experiments (control and adaptation tasks) which did not include hand trials, we only analyzed eye trials in this study.

Gain task

To verify that neural dynamics during the measurement epoch of the reproduction task did not depend on the intervals produced by the animals, we analyzed neural recordings of two other monkeys (monkey C and J) performing a variant of the reproduction task (‘gain task’; Figure 4). Full details of the task and experimental setup can be found in (Remington et al., 2018). Briefly, in the gain experiment, t_s was always sampled from the same discrete uniform distribution (7 values between 500–1000 ms) but animals had to apply a multiplicative gain (gain=1 or 1.5) to the measured interval before reproducing it. That is, animals had to produce either 1 or 1.5 times t_s, depending on the condition. Similar to the reproduction task, the condition type alternated in blocks of trials, and the desired gain value was indicated by the color of the fixation point.

Adaptation task

In the adaptation task (Figure 5 and 6), t_s was first sampled from a pre distribution (5 values between 660–1020 ms), and then became equal to a single interval, either the shortest or the longest value of the pre distribution. The switch was not cued in any way and occurred unpredictably during the session (typically between 500 and 800 trials after the start of the session to guarantee that there were enough trials both before and after the switch for physiology and for behavioral adaptation to happen with a single session). As a control, on some behavioral sessions, we introduced catch trials meant to probe animals on a different interval post switch and verify that they were still measuring time even when exposed to a single interval (Figure S1). The catch trials were rare (~6%) and unpredictable. Prior to recordings, and during behavioral training, animals experienced only a few of these adaptation sessions (12 out of 132 sessions over 239 days for monkey G, 17 out of 121 sessions over 274 days for monkey H), which were also interleaved with a series of ‘wash-out’ sessions with no switch. This was done to prevent overtraining the animals on these switches.

Electrophysiology

Recording procedure

In the interval and adaptation tasks, we used 2 to 3 laminar V-probes to record neural activity acutely in the dorsomedial frontal cortex (DMFC), comprising the supplementary eye field (SEF), the presupplementary motor area (pre-SMA), and the dorsal portion of the supplementary motor area (SMA). Figure S2 shows the exact electrode penetration sites on an MRI reconstruction of each animal’s brain. In the gain task, recordings were done using 1 to 2 laminar V-probes inserted in the same area within DMFC (Remington et al., 2018). Signals were amplified, bandpass filtered, sampled at 30 kHz, and saved using the CerePlex data acquisition system (Blackrock Microsystems, UT). Spikes from single- and multi-units were sorted offline using the Kilosort software suite (Pachitariu et al., 2016).

Recording stability

In the interval and gain tasks, experimental conditions (Short vs Long, gain=1 vs 1.5) were interleaved over relatively short blocks of trials (<100 trials); recording stability in these experiments was therefore not a concern when comparing neural activity across conditions (Remington et al., 2018; Sohn et al., 2019). In the adaptation task, however, we had to track neural activity over the course of an entire recording session (~2-3 hours). We imposed a 1h settling time between electrode insertion and the onset of recordings to minimize electrode drift during the session. Figure S2 shows one example raster plot demonstrating stable spiking activity across trials. Recording stability was further confirmed by a more detailed analysis of spike waveforms (see Figure S2 for raw waveform traces) based on the approach used in (Hill et al., 2011). For every cluster extracted from spike sorting, we considered each spike waveform as a time series summarized as a point in a 60-dimensional space and computed the Mahalanobis distance between each waveform and the average waveform (across all spikes). This resulted in a distribution of distances, which we fitted to a chi-squared distribution following (Hill et al., 2011). We calculated the likelihood of each spike belonging to this fitted distribution, and any spike whose likelihood was lower than a fixed threshold was considered as an outlier. The threshold was defined as the inverse of the total number of spikes for that cluster. This analysis provided us with a single number (percent of outlier spikes) for every cluster and allowed us to assess recording stability. In the analyzed adaptation sessions, we found that only 1% for monkey G, and 0% for monkey H of identified clusters had a percent of outlier spikes greater than 10%.

As an additional safeguard against instabilities, we systematically discarded clusters whose firing rate (averaged over the entire measurement epoch of the task) dropped below 1 spike per second. This guaranteed that neurons which disappeared over the course of the session (or were coming from noise) were not included in the analyses. The number of clusters (single or multi-unit activity) respectively for the interval and adaptation tasks was 741 and 92 for monkey G, and 617 and 50 for monkey H. For the gain task, the number of single/multi-units was 138 for monkey C, 201 for monkey J.

QUANTIFICATION AND STATISTICAL ANALYSIS

Behavioral analyses

For behavioral analysis, we used a probabilistic mixture model to identify and reject outlier trials. Specifically, we calculated the likelihood of each t_p (corresponding to a given t_s) coming from either a task-relevant Gaussian distribution, or from a lapse distribution which we modelled as a uniform distribution extending from the time of Set up to 3t_s. Any trial which was more likely to come from the lapse distribution was considered as an outlier (<5% trials) and discarded before further analyses.

Single neuron analyses

Firing rates were obtained by binning spiking data (20-ms bins for interval and gain task, 30-ms bins for the adaptation task), and smoothing using a Gaussian kernel (standard deviation of 40 ms). In addition, bootstrapped firing rates were generated via resampling trials with replacement (100 repeats). The number of trials where the visual target randomly appeared left or right of the fixation point were always matched before averaging across trials.

Estimating the rate of change of neural activity

To verify that the rate of change of neural activity (i.e., ‘speed’) was faster in the Short compared to the Long condition (Figure 1D), we computed the absolute difference in firing rates between consecutive 20-ms time bins and averaged these differences over the entire measurement epoch. We verified that the resulting speed difference across conditions was not due to the longer duration of the measurement epoch in the Long condition. When we matched the measurement epoch for the Short and the Long condition (i.e., restricting the analysis between Ready and the longest t_s of the Short condition), speed differences were qualitatively unchanged.

Scaling analysis at the single neuron level

To assess how much single neuron activity profiles ‘stretched’ or ‘compressed’ in time across conditions (Figure 2A), we devised an analysis to reconstruct patterns of activity in the Short condition based on a scaled version of the pattern of activity in the Long condition. If we call $r_{short}\left(t\right)$ and $r_{long}\left(t\right)$ the pattern of activity of one neuron in the Short and the Long condition, respectively, our analysis searched for the set of parameters that minimized the difference ${[({\gamma r}_{long}(\lambda t)+\delta )-r_{short}(t)]}^2$, where $\lambda$ was the scaling factor, and $\gamma$ and $\delta$ allowed for gain and baseline modulations across conditions. Example fits as well as distributions of fitted parameters are shown in Figure S3.

To better understand the source of the bimodality in the distribution of scaling factors across neurons (Figure 2B), we sub-selected neurons which had the largest firing rate modulations during the measurement epoch. We sorted neurons by the amount of variance in their firing rate across time points in the measurement epoch, and kept neurons which contributed more than 0.1% of the total variance (150/741 for monkey G, 164/617 for monkey H).

Gaussian mixture model

To quantify the bimodality of the distribution of scaling factors across the population of neurons, we fitted a simplified Gaussian mixture model to the distribution. To do so, we first estimated the probability density of the empirical distribution (Gaussian kernel density estimator, ksdensity in Matlab). We then ran an optimization procedure (fminsearch) to minimize the root-mean-squared-error (RMSE) between the estimated probability density and the sum of two Gaussian probability density functions (pdf), each parametrized by their mean, variance and a gain factor multiplying the entire pdf.

Population-level analysis

With the exception of the neural trajectories shown in the space of principal components (Figure 2C, 4D), all population-level analyses were performed using all neurons (i.e., no dimensionality reduction was applied).

Temporal mapping analysis

To measure speed differences between neural trajectories (e.g., Short versus Long), we relied on a new ‘temporal mapping’ method inspired from the KiNeT analysis originally introduced in (Remington et al., 2018). For every state on the Short trajectory, we first computed the time elapsed ($t_{short}$) and the distance traveled ($d_{short}$) from a fixed reference state on the trajectory defined at $t_{ref}$ (relative to Ready). $d_{short}$ was calculated as the sum of the Euclidean distance between consecutive states along the trajectory between $t_{ref}$ and $t_{short}$. Next, we found the state on the Long trajectory whose distance ($d_{long}$) from the reference state was closest to $d_{short}$, and marked the corresponding time ($t_{long}$). Finally, we plotted $t_{long}$ as a function of $t_{short}$, which we refer to as the temporal mapping, and computed the slope of this mapping to estimate the scaling factor related to speed differences across conditions.

Mathematically, if we call $r_j^{short}\left(i\right)$ and $r_j^{long}\left(i\right)$ the firing rate of neuron $j$ at time $i$ respectively in the Short and Long condition, then for every $t_{short}>t_{ref}$, we calculated $t_{long}$ as follows:

$$t_{long}\left(t_{short}\right)=mi{n}_t\left\{abs\left|log\left(\frac{\sum _{i=t_{ref}}^{t_{short}}\sqrt{\sum _{j=1}^N{\left(r_j^{short}\left(i+1\right)-r_j^{short}\left(i\right)\right)}^2}}{\sum _{i=t_{ref}}^t\sqrt{\sum _{j=1}^N{\left(r_j^{long}\left(i+1\right)-r_j^{long}\left(i\right)\right)}^2}}\right)\right|\right\}$$

For the interval and adaptation tasks (Figure 2F, 5D–E), the reference state was defined at 400 ms post Ready. This choice was motivated by the observation that, early after Ready, neural responses were largely dominated by a non-specific visual transient with equal speeds across conditions (Figure S3). We however ensured that our results were unaffected if the reference state was defined earlier in the measurement epoch (Figure S4). For the gain task (Figure 4E, S5), where the speeds were hypothesized to be equal throughout the measurement epoch, the reference state was defined at Ready.

To compute the scaling factor associated with the two trajectories (i.e., slope of the temporal mapping), we proceeded in three steps. First, we identified the time point when the mapping started to diverge from the unity line (i.e., when $t_{short}$ becomes different from $t_{long}$). We call this time point $t_{onset}$. Second, we identified the time point when $t_{long}$ became equal to $max(t_{short})$. We call this time point $t_{offset}$. Finally, we used linear regression to estimate the scaling factor as the slope of the temporal mapping between $t_{onset}$ and $t_{offset}$.

Running-window speed analysis

For the running-window speed analysis in the adaptation experiment (Figure 6B–C), we adjusted the analysis to ensure better statistical power. We first computed a global neural trajectory obtained by averaging population activity across all trials (pre and post switch combined). We then computed the block-specific trajectory by averaging population activity in a 100-trial window (which we ran over the entire session; overlap between windows: 50%). A temporal mapping and the associated scaling factor (slope) was computed between the global trajectory and each block trajectories. Because the number of trials that went into each block trajectory was relatively small, individual temporal mappings were noisy. We therefore increased the number of points used in the regression analysis to compute the slope by placing the reference state at the time of Ready. Finally, we normalized the scaling factor in each block by the average scaling factor computed across all blocks pre switch. This normalization guaranteed that (1) the scaling factors post switch could be directly tested against the predicted value ${\mu }_{post}/{\mu }_{pre}$, and (2) the potential bias introduced by including the early portion of the measurement epoch to compute the mapping slope was canceled out.

In one monkey (monkey H), the running-window speed analysis displayed large fluctuations toward the end of the session, i.e., after the neural changes had already converged to their predicted value. For this monkey and this analysis only, we relied on the waveform stability criterion to reject neurons which had more than 1% of outlier spikes (see Recording stability section above). This led to a decrease in the number of neurons included in the analysis (18 out 50), but was sufficient to resorb the large fluctuations.

Analysis of geometry of dynamics

To examine the nature of the representation of time intervals in the reproduction task, we analyzed the geometry of neural trajectories associated with each condition. In the first analysis (Figure 3A), we looked at the organization of tangent vectors along the neural trajectories. For each trajectory, we computed the tangent at the mean state (reference tangent) and calculated the angle between the reference tangent and the local tangents computed at every state along the other trajectory (test tangent). For any given state, the local tangent was computed by connecting the states immediately preceding and following that state (bin size: 20 ms). In the second analysis (Figure 3B), we looked at the organization of neural states associated with each interval relative to the mean states. We defined the unique hyperplane which contained the average of the two mean states and whose normal vector was the average of the two tangents computed at the means. We then calculated the Euclidean distance to this hyperplane for every neural state along each trajectory. By convention, the distance was defined as positive when the normal vector pointed in the direction of the state.

Principal component analysis

We performed principal component analysis (PCA) to plot neural trajectories in the reproduction and gain tasks (Figure 2C, 4D). Note however that PCA was not used for any of the quantitative population-level analyses; for these analyses, all neurons were included. PC trajectories were obtained by gathering smoothed firing rates in a 2D data matrix where each column corresponded to a neuron, each row corresponded to a given time point in the measurement epoch. To obtain a common set of principal components for different conditions (e.g., Short vs Long, or gain = 1 vs 1.5), we concatenated PSTHs for the different conditions along the time dimension. We then applied principal component analysis and projected the original data onto the top 3 PCs, which explained about 75% of total variance in the reproduction task, and 85% in the gain task.

Supplementary Material

Supplementary Material

Figure S1 (Related to Figure 1 and 5). Monkey behavior in the reproduction and adaptation task. (A) Monkey behavior in the reproduction task. When exposed to two distributions of sample intervals (red for Short, blue for Long), animals systematically bias their responses toward the mean of each distribution (black arrows). Dots represent single trials, and open circles show the average t_p per each individual t_s. (B) Monkey behavior in the adaptation task with catch trials. To ensure animals were still measuring the sample time interval even when exposed to a single interval, we introduced rare (~6%) ‘catch’ trials in which t_s was different from the post (‘delta’) interval. In sessions where the delta interval was chosen to be the longest of the pre distribution, the catch interval was chosen as the second longest interval of the pre distribution. Bottom: t_p distributions associated with delta (dark blue) and catch (light blue) trials were significantly different (two-sided unpaired-sample t-test, p<10⁻¹⁰ for both monkeys), indicating that animals continued to measure time while adapting to the new distribution (dark and light green distributions confirm the adaptation between early and late post switch on the delta interval). (C) Monkey behavior in the adaptation task for different post intervals. To test the robustness of behavioral adaptation, we ran sessions in which the post interval was the shortest interval of the pre distribution. Comparison of t_p distributions early (dark green) and late (light green) post switch confirmed that the animals successfully adapted (p<10⁻⁵ for both monkeys). Introduction of catch trials in these sessions further confirmed that animals continued to measure time post switch.

Figure S2 (Related to Figure 1 and 5). Neural recordings in the reproduction and adaptation task. (A) Recording sites for the reproduction task and the adaptation task. MRI surface reconstruction showing individual recording sites for both monkeys. Each dot represents one recording site – red for the reproduction task, green for the adaptation task. AS: arcuate sulcus; CS: central sulcus; PS: principal sulcus. (B) Recording stability. To verify the stability of our recordings, we plotted the waveforms at different stages of the recording session during the adaptation experiment (blue for pre switch; light, intermediate, dark orange for early, middle, late post switch). For visualization, waveforms are spatially arranged to match their recording sites on the three V-probes (superficial to deep from top to bottom). (C) Example raster plot during the adaptation task (see Figure 5C for the PSTH associated with that unit). Spikes are aligned to Ready (red); orange dots show Set, green dots show Go. For clarity, trials pre switch have been sorted by t_s, however in the experiment, t_s was randomly sampled. Trials post switch (above horizontal black line showing the transition) are shown in chronological order.

Figure S3 (Related to Figure 1 and 2). Neural dynamics during the measurement and production epoch of the reproduction task. (A) Single neuron activity during the measurement epoch of the reproduction task. Red for Short, blue for Long, black for scaled version of Long fitted to Short (see Methods for details about the scaling analysis). Lambda is the fitted scaling factor. Shaded areas denote 95% CI obtained via standard bootstrapping. (B) Parameter fits for temporal scaling. Top: each neuron’s activity profile in the Short condition ($r_{short}(t)$) was fitted with a scaled version of its activity in the Long condition ($r_{long}(t)$). The fitting procedure minimized the reconstruction error between $r_{short}(t)$ and ${\gamma r}_{long}(\lambda t)+\delta$, where $\lambda$ is the scaling factor, and $\gamma$ and $\delta$ allow for gain and baseline modulations across conditions. Bottom: distribution of fitted parameters across neurons in both monkeys. The red line on the $\lambda$ distributions shows the predicted value $\lambda =\frac{{\mu }_{long}}{{\mu }_{short}}$, where ${\mu }_{short}$ and ${\mu }_{long}$ is the average time interval in the Short and Long condition, respectively. (C) Differences in firing rate and speed across conditions of the reproduction task. Top: Difference in firing rates between the Short and Long condition throughout the measurement epoch of the task. Each black line represents one neuron; the red line represents the average difference across neurons. This difference fluctuated around zero and was unstructured across time points. Middle: Normalized differences in firing rates between the Short and Long condition throughout the measurement epoch of the task. Each black line represents one neuron; the red line represents the average difference across neurons. Similar to top, the normalized difference fluctuated around zero and was unstructured across time points. Bottom: Normalized differences in the rate of change of firing rates between the Short and Long condition. Black line shows average across neurons; shaded area denotes 95% CI. (D) Dynamics of scaling and non-scaling neurons in the production epoch. We analyzed the dynamics in the production epoch of scaling and non-scaling neurons separately. We found that the speed of dynamics in both populations strongly correlates with the interval (Pearson correlation r=−0.98, 95% C.I. [−0.99 −0.96] for Short condition, r=−0.98 [−0.99 −0.97] for Long condition in scaling population of monkey H; r=−0.98 [−0.996 −0.96] for Short, r=−0.96 [−0.98 −0.93] for Long in non-scaling population of monkey H; r=−0.978 [−0.987 −0.968] for Short, r=−0.976 [−0.985 −0.96] for Long in scaling population of monkey G; r=−0.98 [−0.99 −0.93] for Short, r=−0.979 [−0.99 −0.95] for Long in non-scaling population of monkey G.

Figure S4 (Related to Figure 2). Speed modulations in the reproduction task. (A) Temporal mapping across sessions in the reproduction task. Left: each thin line represents one session (N=12 for monkey G; N=17 for monkey H), the thick line represents the average across sessions, the dashed line is the unity. Right: scaling factor (mapping slope) for monkey G (squares) and monkey H (circles) as a function of recorded neurons in each session. Red line shows the predicted value $\lambda =\frac{{\mu }_{long}}{{\mu }_{short}}$. (B) Early versus late dynamics during the measurement epoch of the reproduction task. Top: during the Ready-Set epoch, neural responses had qualitatively different dynamics early (Ready – 400 ms) compared to late (400 ms – Set). Early dynamics tended to be non-condition specific, while late dynamics showed robust differences in speed across conditions (see right panel for an example neuron; as well as Figure S3). To confirm this observation rigorously, we performed a cross-temporal principal component analysis (PCA) quantifying the amount of subspace overlap between the activity in the early vs late window of both Short and Long conditions. Results are summarized in a heatmap shown in the middle row: each cell (i, j) shows the percent of total variance explained when projecting the data of condition/window (i) onto the top 3 PCs computed from condition/window (j). Consistent with our qualitative observations, the overlap between early and late in each condition was smaller than the overlap across conditions for a given window. Bottom: This result prompted us to define the reference state in our speed analysis (Figure 2F) at Ready+400ms to focus on the late part of the dynamics. Defining the reference state at Ready instead of Ready+400ms leads to a slight underestimation of the scaling factor (bottom row; grey distribution deviates from the predicted value $=\frac{{\mu }_{long}}{{\mu }_{short}}\simeq 1.56$; 95% CI [1.33 1.53] for monkey G, [1.39 1.48] for monkey H). This reflects the fact that early during the measurement epoch, dynamics is dominated by non-condition specific patterns of activity which bias the scaling factor towards unity. However, if we project the data onto the subspace associated with the late period of the measurement epoch (variance explained >85%), the resulting scaling factor matches the prediction even when the reference state is defined at Ready (black distribution overlaps with the predicted value; 95% CI [1.42 1.61] for monkey G, [1.48 1.61] for monkey H). (C) Speed adjustments immediately following a condition switch in the reproduction task. We used the temporal mapping analysis (see Methods) to compute speed differences between the Short and Long conditions on the first 5 trials following a block switch. The plot shows that on the very first trial of a block, the speed difference is already at the predicted value (ratio of the two mean intervals; red line) and is not different from the value computed using all trials of the block. This result indicates that the presence of the context cue suffices to modulate the speed of dynamics based on temporal expectations, and rejects the alternative hypothesis that speed is adjusted based only on the experienced t_s.

Figure S5 (Related to Figure 2 and 4). Neural dynamics in the reproduction and gain task. (A) Relationship between neural speed and mean interval for other values of mean ratio in the reproduction task. Top: in the original reproduction task, animals were exposed to two alternating distributions between 480–800 and 800–1200 ms. In this setting, the neural speed difference between the two conditions (as quantified by the scaling factor $\lambda$) was close to the predicted value equal to the ratio between the two means ($\frac{{\mu }_{long}}{{\mu }_{short}}\simeq 1.56$). Bottom: to ensure that this result did not depend on the specific mean ratio value, we collected data in an alternative design where the distributions were between 560–720 and 720–920 ms. As predicted, the neural speed difference between these two conditions was reduced to the new mean ratio ($\frac{{\mu }_{long}}{{\mu }_{short}}\simeq 1.28$). (B) Temporal mapping and instantaneous speed in the gain task. Top: absolute instantaneous speed was computed as the Euclidean distance between consecutive states separated by 20-ms bins. Throughout the measurement epoch, speeds did not diverge across conditions (red for g=1, blue for g=1.5). Bottom: temporal mappings for both monkeys lie along the unity (dashed) line, indicating no relative speed differences across conditions. Histograms show distributions of scaling factors for the bootstrapped data (dark grey) and a null distribution obtained by randomly shuffling conditions across neurons (light grey).

Figure S6 (Related to Figure 5 and 6). Behavioral and neural changes during adaptation. (A) Differences in firing rate before and during adaptation in the adaptation task. We plotted the baseline activity of each neuron at the time of Ready (20-ms window following Ready) in the pre (abscissa) and early post (ordinate) condition. Pre and early post had the same number of trials (~400 trials immediately before and after the switch). Each dot represents one neuron. The diagonal distribution shows that there were no systematic differences in firing rates across the population (p>0.01 for both monkeys). (B) Fast and slow neural changes in the adaptation task. In the adaptation experiment, we found that neurons changed their speeds (i.e., stretched their activity pattern) at seemingly different timescales. To quantify this effect, we computed how the scaling factor of individual neurons changed across blocks of trials (block size = 100 trials, overlap = 50%). For all neurons, we expected the scaling factor to start from its baseline value of 1 (black dashed line) and subsequently, following the switch in distribution (vertical dotted line), to reach the predicted value equal to the ratio of the means post/pre (red dashed line). We found fast-adapting neurons (top) and slow-adapting neurons (middle) that were consistent with our prediction. The timescale of the slow-adapting neurons appeared to match that of behavior (bottom). (C) Behavioral adaptation to changes in variance. In one pilot session, we exposed the animal to a pre distribution (t_s between 660–1020 ms) and then covertly switched to a single post interval at the middle of the pre distribution (t_s = 840 ms). This manipulation allowed us to examine the behavioral adaptation to changes in the variance, while the mean is unchanged. The plot shows single trial responses following the collapse of the interval distribution onto its mean. The variance in the animals’ responses (running mean +/− 2.5 standard deviation shown in black; window size = 1200 trials) gradually decreases over multiple days.

DATA AND CODE AVAILABILITY DETAILS

The published article includes all datasets generated or analyzed during this study. The code supporting the current study is available from the corresponding author on request.