Predictive coding of multisensory timing

Abstract

The sense of time is foundational for perception and action, yet it frequently departs significantly from physical time. In the paper we review recent progress on temporal contextual effects, multisensory temporal integration, temporal recalibration, and related computational models. We suggest that subjective time arises from minimizing prediction errors and adaptive recalibration, which can be unified in the framework of predictive coding, a framework rooted in Helmholtz’s ‘perception as inference’.

Introduction

The sense of time, unlike other senses, is not generated by a specific sensory organ. Rather, all events that stimulate the brain, regardless of sensory modality, contain temporal cues. Because of heterogeneous processing of sensory events, subjective time may differ significantly for a given duration across modalities. For example, an auditory event is often perceived longer than a visual event of the same physical interval [1]. Subjective time is also susceptible to temporal context, voluntary actions, attention, arousal and emotional states, all of which can bias it away from physical time [2,3,4••,5,6]. Over the past several decades, researchers have advanced our understanding of how we perceive and integrate multisensory and sensorimotor timing, with examples such as the ‘central-tendency’ effect [7••,8••,9], the time shrinking illusion [10], and sensorimotor temporal recalibration [11,12••]. In this article we examine a few selected duration-related temporal phenomena and related computational models, and show how those phenomena can be parsimoniously explained within the predictive coding framework [13,14,16•]. We propose that subjective time is an outcome of adaptive processes of the brain that minimize the overall estimation error to boost the reliability of estimation of external temporal structures.

Subjective time as inference

One and a half centuries ago Hermann von Helmholtz famously suggested that perception can be understood as a process of unconscious inference: “The connection between the sensation and external object can never be expressed without anticipating it already in the designation of the sensation… This is because inductive reasoning is the result of an unconscious and involuntary activity of memory” [17]. Time perception is also the result of unconscious inference. Subjective time can be easily influenced by internal expectation, as suggested by Karl Vierordt [18] around the same time as von Helmholtz. He observed that subjective judgment of duration is attracted to an ‘indifference point’, which is close to the central mean of all the durations experienced [9,18]. That is, short durations tend to be overestimated and long durations underestimated. Hollingworth later coined this phenomenon of gravitation toward the expected mean magnitude as the ‘central tendency’ effect [19].

The recent surge of interest in the central tendency effect [7••,8••,20•,21,22] has taken this topic to a new level within Bayesian inference framework. This development has been motivated by the fact that, across a wide variety of tasks, the fundamental problem encountered by the brain is coping with uncertainty [15]. To minimize uncertainty, the brain needs to maximally utilize the available information, combining not only sensory input but also top-down ‘prior belief’ in a weighted average manner. In Bayesian terms, perception emerges from probabilistic inference, including the likelihood associated with the sensory evidence and prior belief (see Box 1). While this type of weighted average is clearly beneficial when the external environment is relatively stable, combining multiple sources of information in the brain would engender perceptual and cognitive biases when the environment changes.

Box 1. Dynamic updating of priors in time perception.

Various types of contextual calibration of time perception can be explained in the framework of Bayesian inference(See Figure I) [4••]. The central idea of Bayesian inference is that the brain uses all available temporal information to minimize the prediction error. One source of temporal information comes directly from sensory inputs, which depends on sensory measures and signal quality. For example, for a given duration D, the sensory measure is S. This cue can be expressed in Bayesian term as the likelihood function P(S|D). Another cue that the brain often uses is internal expectation based on the prior knowledge. In Bayesian term it is the prior function P(D). According to Bayes’ rule, the probability of a duration being D, given the sensory measure S is the product of the prior probability and the likelihood, normalized by the probability of the sensory measures:

$$\mathit{P}\left(\mathit{D}\left|\right\}\mathbf{\text{S}}\right)=\frac{\text{P}\left(\mathbf{\text{S}}\left|\right\}\mathbf{\text{D}}\right)\mathbf{\text{P}}\left(\mathbf{\text{D}}\right)}{\mathbf{\text{P}}\left(\text{S}\right)}$$

The probability distribution P(S|D) is known as the posterior probability. When both the likelihood and prior are independent Gaussians, the optimal duration estimate can be predicted by

$$\mathit{D}=\mathit{w}{\mathit{D}}_{\mathit{s}}+(1-\mathit{w}){\mathit{D}}_{\mathit{p}}$$

where D_s and D_p are the expected mean of the likelihood and prior, and the weight $\text{w}=\left(1/{\sigma }_{\text{s}}^2\right)/\left(1/{\sigma }_{\text{s}}^2+1/{\sigma }_{\text{p}}^2\right)$ is proportional to its reliability, in which $1/{\sigma }_s^2\hspace{0.17em}\text{and}\hspace{0.17em}1/{\sigma }_p^2$ are the reliability of the likelihood and prior. The variance of this optimal estimate is $\left({\sigma }_s^2{\sigma }_p^2\right)/\left({\sigma }_s^2+{\sigma }_p^2\right),$ which is the minimum variance among all possible linear weighted combinations between the sensory estimate and the prior. When there are two conditional independent likelihoods (e.g., one from the auditory modality and another from the visual modality), and the prior is not the focus factor, the optimal estimate is very similar:

$$\mathit{D}={\mathit{w}}_{\mathit{a}}{\mathit{D}}_{\mathit{a}}+(1-{\mathit{w}}_{\mathit{a}}){\mathit{D}}_{\mathit{b}}$$

where D_a and D_b are the mean of two individual sensory estimates, and the weight w_a is proportional to its reliability.

With predictive coding, the internal predictive prior is not fixed, but is dynamically adjusted from the prediction errors. The top-down predictions are delivered through the backward connections. So long as this successfully predicts the lower level activity, all is well, and no further action needs to ensue. But where there is mismatch, a ‘prediction error’ occurs and the ensuing (error-indicating) activity is propagated to the higher level. This automatically adjusts probabilistic representations at the higher level so that top-down predictions cancel prediction errors at the lower level, yielding rapid perceptual inference [13–15,16•]. In a simple case, this predictive processing can be described by Kalman filter [4••,46] — a dynamic optimal prior updating process when noises are Gaussian:

$${\mathit{P}}_n=(1-\mathit{g}){\mathit{P}}_{n-1}+\mathit{g}\Delta$$

where P_n and P_n−1 are the priors at time n and n − 1, g is the Kalman gain, which is optimally determined by the variances of the internal prior and the prediction error. As shown by a developmental study on the temporal recalibration [12••], Kalman gain is larger in the adult group compared to the young groups (see text).

Figure I.

Schematic illustration of Bayesian inference of duration. The red curve denotes the likelihood P(S|D) for a given duration signal, the blue curve the prior at time n, and the dashed blue curve the updated prior at time n + 1. The dark green curve is the posterior based on Bayesian inference. There are two updating processes: the posterior updating based on the cues and the prior is for reliable sensory estimates, and the prior updating based on error correction is for minimizing forthcoming prediction errors.

Jazayeri and Shadlen [7••] recently reinvestigated the central tendency effect in duration reproduction using a Bayesian approach, and confirmed that the fundamental principle of central tendency is a strategy to minimize the overall temporal reproduction errors by combining both sensory likelihood and prior knowledge (e.g., the statistical distribution) of the to-be-estimated duration. Their approach is illustrated in Figure 1. When asked to reproduce temporal intervals, people tend to underestimate long intervals and overestimate short intervals, always ‘regressing toward the mean’. Importantly, the mean is set dynamically, for the specific range being tested in that session. This is brought out most clearly for reproductions at 850 ms: the bias can be either toward shorter or longer intervals, depending on the range of intervals sampled in that particular session (lower panels).

Figure 1.

Regression to the mean in a time-reproduction task. Upper graphs: average reproduction intervals as a function of physical reproduction for measurements made in three different durations: short (squares, 494–847 ms), intermediate (circles, 671–1024 ms), and long (triangle, 847–1200 ms). The graphs at left refer to control subjects with no musical training, those in center to trained drummers, at right to trained string-instrumentalists. All the non-drummers show a strong regression to the mean of that particular interval: drummers respond veridically. Lower graphs: response distributions for interval 850 ms, taken from the three different conditions (short, intermediate, and long; symbols as before). The direction of the bias depends on the interval, tending to under-estimation for the short interval and overestimation for the long interval.

Adapted from [8••].

Note that the internal prior may not equate to the rigid physical distribution of the stimuli, but rather be better captured as a smoothed approximation of the distributions up to third-order moments [8••,20•]. As sensory precision may vary among different groups of individuals as well as across different modalities [8••,21,23], central tendency effects vary according to the weighted average strategy of Bayesian inference. By testing subjects with various levels of musical expertise, Cicchini and colleagues [8••] have demonstrated the variation of tendency effects is closely related to Bayesian optimal encoding. Non-percussionists, who had large variability of visual duration reproduction, showed a standard central tendency effect, while expert drummers responded veridically owing to their high precision of reproduction (Figure 1). Similar variations of central tendency has been shown in patients with Parkinson’s disease [23]. Patients are less prone to the central bias with their dopaminergic medication than without medication, as patients have higher sensory reliability in their medication state. Those findings [8••,21,23] suggest the brain represents recent statistics of event duration, and this information is incorporated in on-going perception, thus, producing biases such as regression to the mean; however, the degree of tendency biases depends crucially on the sensory reliability [8••].

The internal prior that influences perceptual judgments sometimes can be built up quickly, for example, by preceding or following a test interval with a distractor interval. One clear example of such ‘quick’ contextual effect is the ‘time-shrinking’ illusion [10]. A short distractor interval presented before or after a test interval shortens the perceived duration of the test interval [10,24]. Recently Burr et al. [25••] showed that these quick temporal contextual effects depend on the duration of the events: for short events, the contextual effects were very strong (Figure 2a), while for long events it is negligible (Figure 2b). They further showed that the existence of contextual effects was strongly associated with the precision of interval judgments (measured by Weber fraction), which were larger for short than for long intervals (Weber’s law was not observed). This relationship is consistent with Bayesian predictions. Assume the interval prior that represents the ensemble mean of the sequence of intervals keeps at a constant Weber fraction. According to Bayesian inference, the weight of the prior decreases when the reliability of the duration estimate increases, leading to a decrease of contextual effect when the reference duration increases (Figure 2c,d).

Figure 2.

(a and b) PSEs for judging the duration of auditory stimuli, as a function of the duration of the distractor, for base duration of 100 and 1000 ms. For the base duration of 100 ms, distractors shifted the PSE, in a way consistent with assimilation (short distractors cause underestimation and vice versa), while there is no effect for base duration of 1000 ms. (c and d) Illustration of how the Bayesian prior interacts with the likelihood estimate of duration within the Bayesian model, for 100 ms and 1000 ms base durations. The prior (in green) is assumed to be the same normalized width in the two conditions, corresponding to a Weber fraction of 0.2. The Weber fraction of the likelihood (red curves) vary with duration, broad at short durations, narrow at long durations. The prior dominates in determining the posterior (black curve) at 100 ms base duration, the likelihood dominates at 1000 ms base duration.

Adapted from [25].

Multisensory duration and optimal cue integration

In natural environments, real objects stimulate not just one but many senses: a roaring Ferrari is both seen and heard. Integrating the different sensory cues arising from the same object can clearly be advantageous for perception. On the other hand, integrating cues from different objects would be detrimental. In the spatial domain there is ample evidence showing that the brain can integrate separate spatial cues in an optimal manner [26,27], which can be predicted by maximum likelihood estimation (MLE). That is, multiple cues are weighted averaged according to their precisions (Box 1).

Several recent studies have focused on whether the brain integrates temporal cues in an optimal manner. Using an audiovisual apparent motion paradigm, Shi and colleagues [28] demonstrated that implicit estimation of audiovisual interval is in good agreement with the MLE prediction. Incorporating both onset and offset redundant information of multisensory durations, Hartcher-O’Brien et al. [29] have also confirmed that perceived duration is close to the optimal weighting process predicted by the MLE model. However, several other studies have shown multisensory temporal integration can be suboptimal [30,31]. For example, Burr and colleagues [30] found that pure MLE model fits only roughly with the trend in a temporal bisection task for audiovisual interval discrimination. The empirical weight of the auditory cue is far higher that the model predicted. Note, though, those studies that applied the MLE model did not consider potential influence of prior and hyper-prior information in their modeling, which may explain the sub-optimality. One important prediction of optimal integration is that the precision of the integrated signal is finer than that for any individual cues (see Box 1). Hartcher-O’Brien et al. [29] showed that Weber fractions of audiovisual duration estimates were smaller than Weber fractions of auditory or visual alone duration estimates, in agreement with the prediction of the MLE model. Similar improvement of precision with multiple timers has also been found in temporal control and motor coordination [32•,33,34]. For example, Ivry and Richardson [34] showed that in-phase bimanual tapping, compared with unimodal tapping, greatly reduced temporal variability, and the reduction of the variability can be best accounted by a model in which two timing signals are averaged. This account could be a special case of the MLE model when two timing signals have similar variability.

Another study conducted by Burr and colleagues [35] directly measured precision of duration discrimination, and they found an interesting phenomenon: dips in discrimination thresholds (i.e., peaks in sensitivity) near the border between subjective simultaneity and asynchrony for auditory, visual and audiovisual intervals. They suggested that the enhancement of temporal discrimination may result from the boundary between two perceptual categories: synchrony and asynchrony, which the brain may use as an additional discrimination cue. Here we apply Bayesian inference to their original data, assuming the sensitivity of interval discrimination depends on the reliability of two cues: one is the discrimination threshold of the base interval, which depends on interval size (we assume Weber law for simplicity); the other is the likelihood of two compared intervals falling into two separate categories, either simultaneous or asynchronous. Figure 3 shows that the MLE model predicts well this dip effect — enhancement of interval discriminations near the border of the simultaneity window (Figure 3).

Figure 3.

(a) Illustration of how the MLE model combines two interval-discrimination cues. Cue 1 is based on the difference between the base interval (τ) and the comparison interval (τ + Δ), which we assume follows Weber’s law: σ₁ = w_fτ + c. Cue 2 comes from synchrony/asynchrony categorization: when one interval falls within the simultaneity window and the other does not, this is a strong cue as to which is longer. The probability of this occurring peaks near the boundary of the simultaneity. We assume the discriminability function is normal distributed in logarithmic. According to the MLE model (see Box 1), the discrimination threshold of the combined cues is ${\sigma }_1{\sigma }_2/\sqrt{{\sigma }_1^2+{\sigma }_2^2}.$ (b) Interval discrimination thresholds for subject CL (dots) as a function the base interval, separated for audiovisual, and visual-auditory conditions (adapted from [35]), and the prediction of the MLE model (curves).

Being able to detect simultaneity between signals of different modality is important, as it is a strong cue that they arise from the same object, and should therefore be integrated. It is therefore reasonable that the brain dedicates neural mechanisms to this task, and that psychophysical experiments can reveal the action as a separate cue. However, this approach still begs the question of how simultaneity is calculated. A promising approach is bases on cross-correlation between the signals of different modality [35,36], and an approach that has been applied successfully in stereopsis [37].

Temporal recalibration and predictive coding

As the external environment changes constantly, the brain needs to recalibrate its internal representation to be consistent with the external world. Action, which is strongly coupled with perception, is a powerful calibrator of time perception. Action can either compress or expand the perceived duration depending on the temporal relation between actions and events [38–43]. Morrone et al. [38], for example, have demonstrated that the interval of an event delineated by two successive visual flashes in the retinal periphery around a pre-saccadic period is often compressed, similar to the space compression induced by saccadic eye movements. Voluntary actions can also compress time through ‘intentional binding’ [39,44], whereby actions are perceived as shifted toward their effects when they are performed volitionally. This temporal recalibration is so strong that, after a sequence of response-delay-effect adaptation, even the order of the action and its sensory consequence can be subjectively reversed, so that the consequence seems to precede the action [11]. Those studies demonstrate that our brain constantly recalibrates the timing of actions and outcomes to fit the prior expectation that these events are contiguous [45].

It is important to note that an ideal temporal recalibrator should consider the statistical properties of sensory inputs from past history. When sensory inputs are noisy, they would be poor signals to use to calibrate the internal prior, which itself may be more reliable than the sensory signals. However, when the sensory inputs are reliable, and deviate from the prediction, they should update the prediction to fit better the external world [46]. Vercillo et al. [12••] recently demonstrated that this does occur, with a developmental study on temporal recalibration, where they compared temporal recalibration among different age groups. They found that adults exhibited strong recalibration effect, similar to Stetson et al. [11], but the recalibration effect did not occur in the younger age groups (Figure 4a). They also showed that the precision thresholds for the temporal-order task improved with age (Figure 4b) and, most interestingly, the magnitude of recalibration correlated negatively with individual temporal precision (Figure 4c). This suggests that the perceptual systems of the young age group is acting efficiently in not incorporating response-effect delays into their prior belief, as these sensory measures are noisy. They confirmed this idea by measuring adult performance with added noise, and showed that noisy signals are indeed ineffective for recalibration. This kind of precision-based temporal recalibration can be explained by dynamic Bayesian inference, such as a Kalman filter (Box 1) [4••,46].

Figure 4.

How sensory-motor recalibration is affected by precision. (a) Average recalibration as a function of age. (b) Average thresholds as a function of age. (c) The recalibration effect as a function of thresholds: the correlation is strong (R² = 0.25, p < 0.001, slope = −0.9).

Adapted from [12••].

Another type of action-induced time distortion is duration-expansion of a subsequent sensory event. For instance, when a person shifts the focus of gaze to a clock, the second hand of a clock appears to, momentarily, stand still, the stopped clock illusion (‘chronostasis’) [40]. Other manual actions [41,43,47], action preparation [48], or even irrelevant action context [49], can also expand subjective duration expansion. The original explanation for the ‘chronostasis’-type duration expansion is that the brain likely backdates the onset event to the action onset due to the uncertainty of the event induced by voluntary action [40]. This account is in line with recent predictive coding suggestions — action preparation and action alter the prediction of ongoing streams of sensory results [15].

Note that subjective time compression and expansion do not exclude each other, but may rather reveal two different effects on different events caused by voluntary action. The time between an action and its effect, and the duration of events occurring before and during the action are likely to be compressed; subsequent sensory events are prone to be dilated. Both effects reflect dynamic updating about the temporal relationship between outgoing actions and incoming sensations [11,50]. The brain needs to explain away prediction errors between the actual current sensory timing and the predicted one. By updating internal temporal prior with prediction errors based on Bayesian inference, top-down predictions could minimize temporal prediction errors at the lower level, ‘explaining away’ those conflicts [13,14,15,16•].

Concluding remarks

In this article we have discussed some recent research on contextual effects in duration perception, multisensory temporal integration and temporal recalibration. Application of the predictive coding framework offers a new view on subjective time perception. Various types of time distortions and multisensory temporal integration can be regarded as the outcome of perceptual inference by maximally using available information to minimize overall temporal prediction errors. Temporal assimilation effects, such as central tendency and recalibration depend much on the precision of the available temporal cues, which can be predicted by computational principles within the predictive coding framework. It is important to note, however, the literature reviewed above is a small part of large body of multisensory timing studies. Future empirical work and continue application of predictive coding framework will expand our understanding of multisensory timing and time perception.