Predicting the Future With a Scale-Invariant Temporal Memory for the Past

Abstract

In recent years, it has become clear that the brain maintains a temporal memory of recent events stretching far into the past. This letter presents a neurally inspired algorithm to use a scale-invariant temporal representation of the past to predict a scale-invariant future. The result is a scale-invariant estimate of future events as a function of the time at which they are expected to occur. The algorithm is time-local, with credit assigned to the present event by observing how it affects the prediction of the future. To illustrate the potential utility of this approach, we test the model on simultaneous renewal processes with different timescales. The algorithm scales well on these problems despite the fact that the number of states needed to describe them as a Markov process grows exponentially.

1. Using Memory to Predict the Future

Reinforcement learning (RL) models that are designed for Markov processes (Watkins & Dayan, 1992; Sutton, 1988) have been extraordinarily successful in accounting for reward systems in the brain (Schultz, Dayan, & Montague, 1997; Waelti, Dickinson, & Schultz, 2001) and have led to remarkable achievements in artificial intelligence (Mnih et al., 2015; Silver et al., 2018). Despite the success of RL, its affinity for Markov statistics may be a serious limitation. The real world contains many distinct causes that predict their effects at a range of timescales, presenting a challenge for learners optimized for Markov statistics. Of course, random processes with memory can be turned into Markov processes at the cost of defining additional states. However, the cost in terms of memory, and time to learn transition probabilities among an exponentially growing number of states, may be excessive in some settings.

It has been proposed that a primary function of the mammalian brain is to predict future events to enable adaptive behavior (Clark, 2013; Friston, 2010). Evidence from neuroscience has made clear that the brain contains robust memory for the identity and time of recent events extending well into the past. For instance, sequentially activated time cells in the hippocampus, prefrontal cortex, and striatum (MacDonald, Lepage, Eden, & Eichenbaum, 2011; Tiganj, Cromer, Roy, Miller, & Howard, 2018; Mello, Soares, & Paton, 2015) maintain information about the time at which recent events were experienced over at least tens of seconds, and perhaps much longer. Experimental presentation of distinct stimuli triggers different sequences of time cells (Tiganj et al., 2018; Taxidis et al., 2020; Cruzado, Tiganj, Brincat, Miller, & Howard, 2020) so that these populations maintain information about what happened when. In addition to sequentially activated time cells, neurons in the entorhinal cortex (Tsao et al., 2018; Bright et al., 2020) and other cortical regions (Bernacchia, Seo, Lee, & Wang, 2011; Murray et al., 2017) carry temporal information via populations of neurons that respond with a spectrum of characteristic timescales, in some cases up to at least minutes (Tsao et al., 2018). This letter, inspired by work arguing that conditioning results from an attempt to learn temporal contingencies between stimuli (Balsam & Gallistel, 2009; Gallistel, Craig, & Shahan, 2019), presents a formal model that learns to predict the future given a temporal record of the past. This proposed mechanism is computable given a temporal history that can be translated in time and proposes a solution for how to estimate the future from a past that includes information about many past events.

This letter proceeds as follows. In the rest of this section, we review a model for retaining a record of past events and associations between event pairs. In section 2, we present the model for predicting the future given a temporal record of the past. In section 3, we discuss its computational complexity, timescale invariance, and several other properties. In section 4, we present a numerical demonstration of the efficacy of this algorithm. Finally, in section 5, we compare this algorithm to traditional RL algorithms and point out its connections to neuroscience.

1.1. A Formal Model for Temporal Record of the Past

We start with an agent capable of observing and remembering several types of events, such as the onset of a 440-Hz tone or the appearance of an image of an apple. In this section, we describe a model for its capabilities. We will see that the agent maintains a fuzzy time line of past events, which it uses to make pairwise associations between events. Neurobiological justification for this model is outlined in section A.1 of the appendix.

1.1.1. Events in Continuous Time

We assume that the world provides a series of discrete events that occur in continuous time. For simplicity, without loss of generality, suppose there are three types of events, which we call x, y, and z, respectively. Whenever we need to avoid confusion, we use event type to refer to type of event and event episode to refer to an individual occurrence of an event. We encode the occurrence of the event type x as a signal $f^{\mathrm{X}}\left(t\right)$, which is the sum of Dirac delta functions centered at the occurrence times of episodes of x (see Figure 1a). (We discuss quantities in relation to x; such statements hold analogously for y and z.) We call $t$, the argument for the signal $f^{\mathrm{X}}\left(t\right)$, real time or external time, to emphasize that this time axis is a feature of the world instead of being constructed by the observer. We denote the collection of all three signals as $\mathbf{f}\left(t\right)$ and analogously for the quantities to follow. At every instant in (external) time $t_0$, the agent has direct access to $\mathbf{f}\left(t_0\right)$ (which is zero precisely unless the event of interest occurs at $t_0$), but not $\mathbf{f}$ at any other time value. Signals are shown in Figure 1a in the case where x, y, and z occur at times 0, 1, and 2, respectively.

Figure 1:

Memory is a fuzzy representation of the signal up to the present. (a) Signal as a function of external time, for three event types, x, y, and z. This is the scenario considered in Figures 2 through 6. (b) Memory for a recent event as a function of internal past time, at varying (external) times since the event occurred. As a function of internal past time, peaks in the memory are present at approximately the time interval since the event.

1.1.2. Temporal Memory

At every instant in time $t_0$, the agent's memory for x, denoted ${\tilde{f}}^{\mathrm{X}}(\stackrel{\ast }{\tau };t_0)$, is a fuzzy representation of the signal up to the present, $f^{\mathrm{X}}(t_0-\stackrel{\ast }{\tau })$. From the agent's perspective, the internal past time, $\stackrel{\ast }{\tau }>0$, indexes how long ago events in memory might have occurred. The degree of fuzziness of the memory varies inversely with a sharpness parameter $k$, which is typically a small even integer; throughout this letter, it is fixed at 8.

Consider an event that exactly happens at a particular time, ${\tau }_0$. At time ${\tau }_0+t$, the memory element for that event is given by $\tilde{f}(\stackrel{\ast }{\tau };{\tau }_0+t)={\Phi }_k(t/\stackrel{\ast }{\tau })/\stackrel{\ast }{\tau }$, where the fuzziness, ${\Phi }_k(·)$, is given by the dimensionless equation,

$${\Phi }_k\left(x\right)=u\left(x\right){\kappa }_0{x}^k{e}^{-kx},$$ (1.1)

${\kappa }_0=k^{k+1}/k!$ is a normalizing constant and $u$ is the unit step function. Memories for a recent event are shown in Figure 1b for various values of $t$. For an arbitrary signal $\mathbf{f}$, the associated memory up to time $t$ is

$$\tilde{\mathbf{f}}(\stackrel{\ast }{\tau };t)=\frac{1}{\stackrel{\ast }{\tau }}{\int }_{-\infty }^t\mathbf{f}\left(\tau \right){\Phi }_k\left(\frac{t-\tau }{\stackrel{\ast }{\tau }}\right)d\tau .$$ (1.2)

(The origin for equations 1.1 and 1.2 is outlined in section A.1) In other words, the memory for an event type is the sum of the memory elements associated with each episode of that event type. On its face, equation 1.2 appears to assume that the agent has access to the infinite past of $\mathbf{f}\left(t\right)$. However, previous work has shown that $\tilde{\mathbf{f}}(\stackrel{\ast }{\tau };t)$ can be efficiently and time-locally constructed from a set of leaky integrators with a spectrum of time constants (see section A.1; Shankar & Howard, 2013). Using this approach, the number of leaky integrators necessary to remember the past to some bound $T$ goes up like $logT$. Previous papers (e.g., Shankar & Howard, 2013) using this formalism have made explicit use of this property in choosing to sample values of $\stackrel{\ast }{\tau }$ on a logarithmic scale and logarithmically compress the $\stackrel{\ast }{\tau }$ axis in integrals. In this letter, we do not logarithmically compress the $\stackrel{\ast }{\tau }$ axis in integrals. However, one may adopt an alternative interpretation, consistent with this letter, as follows. Within this alternative interpretation, the expressions for integrals over $\stackrel{\ast }{\tau }$ used below are logarithmically compressed (i.e., a factor of $1/\stackrel{\ast }{\tau }$ is added to the integrands). At the same time, the prefactor of $1/\stackrel{\ast }{\tau }$ is removed from equation 1.2. Neurally, this would amount to a statement that the peak firing rate of time cells triggered by a delta function is constant as a function of $\stackrel{\ast }{\tau }$.

The signal $\mathbf{f}$ up to any given external time $t_0$ fixes the event occurrence history. However, due to the agent's fuzzy memory, the agent is only able to form a fuzzy subjective belief distribution about the event occurrence history leading up to the present. We may interpret the memory for x as the agent's subjective estimate of the instantaneous rate of occurrence of x at time $t-\stackrel{\ast }{\tau }$. In other words, we have, for an infinitesimal time element $d\stackrel{\ast }{\tau }$,

$${\tilde{f}}^{\mathrm{X}}(\stackrel{\ast }{\tau };t)d\stackrel{\ast }{\tau }\approx P\left(\text{x}@t-\stackrel{\ast }{\tau }\left(d\stackrel{\ast }{\tau }\right)\right),$$ (1.3)

where $P(·)$, the probability of an event, is used in the subjective Bayesian sense to describe the agent's belief and “$\text{x}@t-\stackrel{\ast }{\tau }\left(d\stackrel{\ast }{\tau }\right)$” stands for “an episode of event x occurred within the infinitesimal time interval between $t-\stackrel{\ast }{\tau }$ and $t-\stackrel{\ast }{\tau }+d\stackrel{\ast }{\tau }$.” Since $\tilde{\mathbf{f}}$ allows the agent access to the identity of and approximate time at which past events might have happened, we describe $\tilde{\mathbf{f}}\left(\stackrel{\ast }{\tau }\right)$ to be a time line of the past.

At each instant in time $t$, the agent is also able to compute the state of the memory a time interval $\delta$ into the future, assuming that no events of interest occur during that interval. For an arbitrary signal $\mathbf{f}$, this quantity is given by

$${\tilde{\mathbf{f}}}_{\delta }(\stackrel{\ast }{\tau };t)=\frac{1}{\stackrel{\ast }{\tau }}{\int }_{-\infty }^t\mathbf{f}\left(\tau \right){\Phi }_k\left(\frac{t+\delta -\tau }{\stackrel{\ast }{\tau }}\right)d\tau .$$ (1.4)

Translation can be efficiently implemented based on the set of leaky integrators. Prior work has shown that this can be done in a neurobiologically reasonable way (see section A.2; Shankar, Singh, & Howard, 2016).

1.1.3. Estimating Pairwise Time-Lagged Statistics

Many models of memory make use of associations between the temporal context describing the recent past and the currently available stimulus. The agent described here builds pairwise associations from x (the cue) to y (the outcome) as the average state of memory for x whenever y occurs, and analogously for other event pairs:

$$\Delta M_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)\propto {\tilde{f}}^{\mathrm{X}}(\stackrel{\ast }{\tau };t)f^{\mathrm{Y}}\left(t\right).$$ (1.5)

We denote the collection of pairwise associations between event pairs as $\mathbf{M}\left(\stackrel{\ast }{\tau }\right)$, which may be thought of as an $n\times n$ matrix at every $\stackrel{\ast }{\tau }$, where $n$ is the number of possible events. We denote the collection of pairwise associations with x as the cue as ${\mathbf{M}}_{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)$, which may be thought of as a vector with $n$ elements, one for each possible outcome, at every $\stackrel{\ast }{\tau }$.

Note that as a neural network, equation 1.5 simply requires Hebbian learning. At the end of learning, we normalize ${\mathbf{M}}_{\mathrm{X}}$ by the number of episodes of the cue x, $\int f_{\mathrm{X}}\left(t\right)dt$.

For example, suppose that x always precedes y by a time interval ${\tau }_{\mathrm{X}\mathrm{Y}}$. Then, by the end of learning, we would have the pairwise association

$$M_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)={\Phi }_k({\tau }_{\mathrm{X}\mathrm{Y}}/\stackrel{\ast }{\tau })/\stackrel{\ast }{\tau }.$$ (1.6)

Figure 2 shows the pairwise associations between two pairs of events, occurring 1 and 2 time units apart, respectively.

Figure 2:

Pairwise associations fuzzily represent the rate of finding two events occurring a certain time interval apart. Other event types are ignored in computing the association between two event types. The associations shown here are based on the signals in Figure 1a. As a function of internal time, the associations peak at around the time interval separating the event pairs.

We may view $M_{\mathrm{X}}^{\mathrm{Y}}$ from two complementary perspectives. First, given the occurrence of y in the present, the agent could use $M_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)$ as a subjective estimate, based on an average over occurrences of y, of the instantaneous rate of occurrence of x at time $\stackrel{\ast }{\tau }$ in the past, that is,

$$M_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\approx P\left(\text{x}@t_{\mathrm{Y}}-\stackrel{\ast }{\tau }\left(d\stackrel{\ast }{\tau }\right)\right).$$ (1.7)

Second, given the occurrence of x in the present, the agent may use $M_{\mathrm{X}}^{\mathrm{Y}}\left(\delta \right)$ as a subjective estimate of the instantaneous rate of occurrence of y at time $\delta$ in the future, that is,

$$M_{\mathrm{X}}^{\mathrm{Y}}\left(\delta \right)d\delta \approx P\left(\text{y}@t_{\mathrm{X}}+\delta \left(d\delta \right)\right).$$ (1.8)

We use $\stackrel{\ast }{\tau }$ or $\delta$ as the time argument for $\mathbf{M}$ according to the interpretation that applies.

A limitation of directly selecting an element of the pairwise association $\mathbf{M}$ to predict the future is that the prediction can only be based on a single cue in the present (i.e., the cue corresponding to that element). To overcome this, the agent constructs the pairwise prediction $\mathbf{m}$, which integrates cue—outcome pairing information (encoded by $\mathbf{M}$) from multiple simultaneous cues in the present to predict the future. We estimate the rate of future occurrences of y based on the events in the present (time $t$) as

$$m^{\mathrm{Y}}(\delta ;t)d\delta \approx P\left(\text{y}@t+\delta \left(d\delta \right)\right),$$ (1.9)

where

$$\mathbf{m}(\delta ;t)={\kappa }_1{e}^{\left[\mathcal{M}{\mathbf{f}}_{\delta }\right]\left(t\right)}$$ (1.10)

is the collection of rates of outcomes (a vector with $n$ elements, one for each possible outcome, at every $\delta$) and ${\kappa }_1$ is a normalization constant whose form is given in section A.3. The exponential applies element-wise. The operator $\mathcal{M}$ is defined by

$$\left[\mathcal{M}{\mathbf{f}}_{\delta }\right]\left(t\right)=\frac{1}{\left|{\mathcal{E}}_t\right|}\sum _{\alpha ∊{\mathcal{E}}_t}\int f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)log{\mathbf{M}}_{\alpha }\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau },$$ (1.11)

where ${\mathcal{E}}_t$ is the set of events occurring at time $t$, $\left|{\mathcal{E}}_t\right|$ is the number of events occurring at time $t$, ${\mathbf{M}}_{\alpha }$ is the collection of pairwise associations with $\alpha$ as the cue (a vector with $n$ elements, one for each possible outcome in correspondence with $\mathbf{m}$, at every $\delta$), and

$$f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)={\Phi }_k(\delta /\stackrel{\ast }{\tau })/\stackrel{\ast }{\tau }$$ (1.12)

denotes the future state of the memory element associated with the currently occurring episode of $\alpha$. (Contrast this with ${\tilde{f}}_{\delta }^{\alpha }$ (see equation 1.4), which denotes the future memory state induced by all past occurrences of $\alpha$.) The logarithm in equation 1.11 applies element-wise. The operator $\mathcal{M}$ may be thought of as operating on the precomputed future memory state of the current events (see equation 1.4) to generate a prediction for the future. In general, equations 1.8 and 1.9 provide similar estimates for the future. The normalization constant ${\kappa }_1$ is such that precisely when x is the only cue for y and the time delay between them is fixed, equations 1.8 and 1.9 provide exactly the same estimate for y. Mathematical details are in section A.3.

Intuitively, the integral that is the second term on the right-hand side of equation 1.11 takes the product of $log{\mathbf{M}}_{\mathrm{X}}$ with the future state of the memory element $f_{\delta }^{\mathrm{X}}$. Let us consider y to be the outcome of interest and consider the case where the time interval between x and y, ${\tau }_{\mathrm{X}\mathrm{Y}}$, is constant. In this case, the integral (and thus, $m^{\mathrm{Y}}$) attains a maximum as a function of $\delta$ when $\delta$ coincides with the peak of $log{M}_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)$, which is approximately ${\tau }_{\mathrm{X}\mathrm{Y}}$. Fittingly, this is behavior we expect of $m^{\mathrm{Y}}$.

Both $M_{\mathrm{X}}^{\mathrm{Y}}$ and the corresponding equation 1.11 integral are smooth functions that peak around the delay interval between the two events. This may prompt the question of why the integral is used instead of $M_{\mathrm{X}}^{\mathrm{Y}}$ in its place (Tiganj et al., 2019). The strength of the present formulation is that equations 1.10 to 1.12 closely parallel equations 2.2 to 2.3 in the next section, for which the integral is necessary. Using equations of similar form is more neurobiologically realistic because it suggests that analogous neural architecture supports the computation for both pairwise prediction $\mathbf{m}$ and prediction $\mathbf{p}$, to be introduced later.

2. Predicting the Future with a Scale-Invariant Past

It would be straightforward to build a prediction for the future based on a single event (e.g., the most recent event) using the pairwise associations $\mathbf{M}$. The challenge is to build a prediction that is based on multiple events in the recent past. One difficulty arises when associations overlap. For example, we associate the sound of rain (x) with a chance of hearing thunder (z). We also associate the sight of wet ground (y) with a chance of hearing thunder (z). Having heard the sound of rain, the prediction for thunder should not be increased by the sight of wet ground when we step outdoors. This example illustrates one of the pitfalls of simply adding the predictions suggested by the pairwise associations.

To address double-counting, in addition to pairwise associations, we construct credit associations between event pairs, which is the key for this algorithm to generate a time line for the future. In section 2.1, we explain how the agent constructs a time line of the future by integrating over a time line of the past, weighted by credit associations between cues and outcomes. In section 2.2, we show how the agent learns credit associations between cues and outcomes based on comparing predictions prior to the cue with predictions due to the cue.

2.1. Generating Predictions from Credit Associations

In addition to the pairwise associations $\mathbf{M}$, we build the credit associations $\mathbf{C}$ between each pair of events (a cue and an outcome) as a function of internal time $\delta$ since the cue. The credit associations $\mathbf{C}\left(\delta \right)$ may be thought of as an $n\times n$ matrix at every $\delta$. We denote the collection of credit associations with x as the cue as ${\mathbf{C}}_{\mathrm{X}}\left(\delta \right)$, which may be thought of as a vector with $n$ elements, one for each possible outcome, at every $\delta$. We interpret $C_{\mathrm{X}}^{\mathrm{Y}}\left(\delta \right)$ as the logarithm of the factor by which an agent adjusts its subjective estimate of the instantaneous rate of occurrence of y at time $\delta$ in the future, having just observed x. Denoting ${\mathbf{p}}^{-}\left(\delta \right)$ as the agent's prior estimate (just before observing x), we have

$${\left[p^{-}\left(\delta \right)\right]}^{\mathrm{Y}}exp{C}_{\mathrm{X}}^{\mathrm{Y}}\left(\delta \right)d\delta \approx P\left(\text{y}@t_{\mathrm{X}}+\delta \left(d\delta \right)\right).$$ (2.1)

equation 2.1 relates to the observation of one cue at one time (the present). For cues in the past, the further in the past they occur, the more imminent outcomes should seem. For example, if x has credit for y peaking at $\delta =5$ and x occurred three time units ago, y should be expected in two time units. Accounting for multiple cues over the past, we find that at time $t$, the agent's internal time line for a time $\delta$ into the future, is

$$\mathbf{p}(\delta ;t)=\mathbf{\Lambda }\odot e^{\left[\mathcal{C}{\tilde{\mathbf{f}}}_{\delta }\right]\left(t\right)},$$ (2.2)

where $\mathbf{p}$ stands for prediction and is a vector over event types, $\mathbf{\Lambda }$ consists of the long-term average of each event type, $\odot$ denotes the element-wise product, and the exponential applies element-wise. The operator $\mathcal{C}$ is defined by

$$\left[\mathcal{C}{\tilde{\mathbf{f}}}_{\delta }\right]\left(t\right)=\sum _E\int {\mathbf{C}}_E\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^E(\stackrel{\ast }{\tau };t)d\stackrel{\ast }{\tau },$$ (2.3)

where the index of summation $E$ indexes the possible cue types, and ${\mathbf{C}}_E\left(\stackrel{\ast }{\tau }\right)$ represents the collection of credit associations with $E$ as the cue (a vector with $n$ elements, one for every possible outcome, at every $\stackrel{\ast }{\tau }$). Intuitively, the integral sums products of ${\mathbf{C}}_E$ with the projected memory ${\tilde{f}}_{\delta }^E$. Let us consider y to be the outcome of interest and x the only cue, and consider the case where the time interval between x and y, ${\tau }_{\mathrm{X}\mathrm{Y}}$, is constant. In this case, the integral (and thus, $p^{\mathrm{Y}}$) attains a maximum as a function of $\delta$ at the value of $\delta$ at which the peaks of $C_{\mathrm{X}}^{\mathrm{Y}}$ and ${\tilde{f}}_{\delta }^{\mathrm{X}}$ coincide. The credit $C_{\mathrm{X}}^{\mathrm{Y}}$ peaks around ${\tau }_{\mathrm{X}\mathrm{Y}}$, the time delay between x and y. The projected memory ${\tilde{f}}_{\delta }^E$ peaks around the time ${\tau }_{\mathrm{X}}+\delta$, where ${\tau }_{\mathrm{X}}$ is the time that has elapsed since x. Therefore, the integral (and thus, $p^{\mathrm{Y}}$) peaks around $\delta ={\tau }_{\mathrm{X}\mathrm{Y}}-{\tau }_{\mathrm{X}}$, which is the time remaining to y, the outcome of interest. In other words, the agent's expectation for y would be the highest at a time when y is, in fact, due. Mathematical details are in section A.4.

We interpret $p^{\mathrm{Y}}(\delta ;t)$ as the agent's subjective estimate, made at time $t$, of the instantaneous rate of occurrence of y at time $\delta$ in the future, that is,

$$p^{\mathrm{Y}}(\delta ;t)d\delta \approx P\left(\text{y}@t+\delta \left(d\delta \right)\right).$$ (2.4)

Unlike equations 1.8 and 2.1, this estimate takes into account all of the events that have occurred in the recent past. A schematic distinguishing the utility of the pairwise associations $\mathbf{M}$ and the credit associations $\mathbf{C}$ in making predictions is shown in Figure 3. Just as we consider $\tilde{\mathbf{f}}\left(\stackrel{\ast }{\tau }\right)$ a time line of the past, we consider $\mathbf{p}\left(\delta \right)$ a time line of the future. Note that $p^{\mathrm{Y}}(\delta =0;t)$ would correspond to the agent's internal model for, in the language of point process theory, the conditional intensity function of y (see Rasmussen, 2018).

Figure 3:

Predictions can be made using credit associations $\mathbf{C}$ based on memories of the multiple events in the recent past. The horizontal axis shows events occurring in real time. The event signal for this scenario is shown in Figure 1a. Associated with each point in real time is an agent's internal time axis, shown here diagonally at $t=1.0$, which indexes memories of the past (bottom half) and predictions for the future (top half). The agent may make a prediction for the future with $\mathbf{M}$ (see equation 1.8) based on the currently observed event (here, y). As a better alternative, the agent may make a prediction for the future with $\mathbf{C}$ (see equation 2.2) based on multiple events in the present and the recent past.

As an illustration, consider again the scenario where events x, y, and z always occur consecutively, once on each trial, at relative times 0, 1, and 2, respectively, with a very long gap between trials. Once x occurs, the proposed algorithm (explained in the following sections) generates predictions for y and z that become more and more imminent as time elapses (see Figure 4). As a function of $\delta$, the predictions peak at approximately the time when the events are, in fact, due.

Figure 4:

Predictions for events peak at about the right time and become more imminent with time. The events x, y, and z occur on each trial at times 0, 1, and 2, respectively, as with previous figures. (a) A schematic of the state of memory and prediction as a function of time. The axes have the same interpretation as in Figure 3. At real time 0.00, x is observed, leading to a prediction for y and z, depicted along the diagonal internal time axis. As real time passes, the memory of x recedes into the past, and the predictions for y and z become more imminent, depicted by the events' downward movement along the internal time axis. (b) Prediction for y and z generated by simulation using the proposed algorithm as the memory for x recedes into the past, depicted at four time points. The peak times for the prediction for y and z correspond roughly to when the events are in fact due and move toward zero as time passes. For example, in the top-most plot, right after x occurs, y and z are to occur in 1 and 2 time units, respectively. Indeed, the generated predictions for y and z peak at approximately $\delta =$ 1 and 2, respectively.

2.2. Computing Credit Associations

Loosely speaking, we assign credit for an outcome to an event according to how much the event's occurrence would revise the prediction for that future outcome. In our example, wet ground would be assigned little to no predictive value, since the chance of thunder has already been predicted by the sound of rain. During training, we update the credit assigned to an event when that event occurs. In this section, we describe the update that happens when x occurs with no loss of generality.

Formally, as we have stated, $exp{C}_{\mathrm{X}}^{\mathrm{Y}}\left(\delta \right)$ is the factor by which we should adjust the prediction for y at time $\delta$ in the future, having just observed x. Therefore, to compute $exp{C}_{\mathrm{X}}^{\mathrm{Y}}\left(\delta \right)$, whenever x is observed, we will first compute the prediction for y before and due to the observation of x, and analogously for other possible outcomes.

2.2.1. Prediction prior to Event Observation

Prior to event observation at time $t$, the prediction associated with internal future time $\delta$ is given simply by

$${\mathbf{p}}^{-}(\delta ;t)=\underset{t^{\text{'}}\to t^{-}}{lim}\mathbf{p}(\delta ;t^{\text{'}}).$$ (2.5)

This prediction arises from the memory of cues in the past and specifically excludes the effects of what occurs at time $t$.

Consider the scenario in Figure 1, where x, y, and z occur consistently at trial times 0, 1, and 2, respectively. When x occurs, ${\left(p^{-}\right)}^{\mathrm{Y}}={\Lambda }_{\mathrm{Y}}$, the long-term average rate of y, for all $\delta$. This is because ${\mathbf{p}}^{-}$ is computed based on memory of events occurring before x, of which there are none (see Figure 5c). In contrast, when y occurs, ${\left(p^{-}\right)}^{\mathrm{Z}}$ shows a peak at $\delta =1$, based on memory of events occurring before y (i.e., x), and the credit association between x and z (see Figure 5f).

Figure 5:

Observed events receive less credit for future events, which have already been predicted based on past events. As with all previous figures, the events x, y, and z occur on each trial at times 0, 1, and 2, respectively. (a) A schematic of the state of memory and prediction as a function of time, as in Figure 4(a). At real time 0.00, x is observed and the memory is empty. (b) An illustration of an agent's inferences at the time x occurs. No memory of past events exists to suggest a prediction, whereas the currently observed event x suggests that y occurs soon. (c) Plots of ${\mathbf{p}}^{+}$, ${\mathbf{p}}^{-}$ (top) and $e^{C_{\mathrm{X}}^{\mathrm{Y}}}$ (bottom) as a function of internal future time, $\delta$, at the time x occurs, for the prediction of y. The quantity ${\mathbf{p}}^{+}$ (red) is the pairwise association between x and y, while ${\mathbf{p}}^{-}$ (purple) is flat as a function of $\delta$ as there is no memory of events. The quantity $e^{C_{\mathrm{X}}^{\mathrm{Y}}}={\mathbf{p}}^{+}/{\mathbf{p}}^{-}$ (orange). (d) Same as panel a, but at real time 1.00. y is observed, and x is in memory. (e) An illustration of an agent's inferences at the time y occurs. The agent remembers x, prompting a prior prediction of z. The currently observed event y suggests the same, but the agent does not gain much information from y, and hence assigns y less credit. (f) Same as panel c, but at the time y occurs, for the prediction of z. The quantity ${\mathbf{p}}^{+}$ is the pairwise association between y and z, which is the same as that between x and y. However, ${\mathbf{p}}^{-}$ reflects the prior prediction for z based on the memory for x. (This is $p^{\mathrm{C}}$ from the bottom-most plot in Figure 4b.) Thus, $e^{C_{\mathrm{X}}^{\mathrm{Y}}}$ is diminished.

2.2.2. Prediction due to Event Observation

For the prediction due to the observed event x itself, we use the pairwise prediction in accordance with equation 1.9,

$${\mathbf{p}}^{+}(\delta ;t)=\mathbf{m}(\delta ;t).$$ (2.6)

For the scenario in Figure 1, when x occurs, ${\left(p^{+}\right)}^{\mathrm{Y}}=m^{\mathrm{Y}}=M_{\mathrm{X}}^{\mathrm{Y}}$ (see Figure 5c), and when y occurs, ${\left(p^{+}\right)}^{\mathrm{Z}}=m^{\mathrm{Z}}=M_{\mathrm{Y}}^{\mathrm{Z}}$ (see Figure 5f). Both of these have the same form, peaking sharply at $\delta =1$, since the time intervals between x and y and between y and z are fixed and equal.

2.2.3. Updating $\mathbf{C}$

When x is observed at time $t$, we update ${\mathbf{C}}_{\mathrm{X}}$ in the following manner:

$$\Delta exp{\mathbf{C}}_{\mathrm{X}}\left(\delta \right)\propto \frac{{\mathbf{p}}^{+}(\delta ;t)}{{\mathbf{p}}^{-}(\delta ;t)}-exp{\mathbf{C}}_{\mathrm{X}}\left(\delta \right).$$ (2.7)

The division and exponentiation are performed element-wise for each possible outcome. This update depends on the previous state of $\mathbf{C}$ (through equations 2.5 and 2.2). During training, as events occur, we update respective components of $\mathbf{C}$, which in turn enhances the agent's predictions of the future as training proceeds. This update rule squares with the intuition that events be assigned credit in accordance with their association with outcomes that are not previously predicted. As training proceeds, $exp{\mathbf{C}}_{\mathrm{X}}\left(\delta \right)$ approaches ${\mathbf{p}}^{+}(\delta ;t)/{\mathbf{p}}^{-}(\delta ;t)$ in expectation, up to the variability of event occurrence history in recent episodes of x during training. Since we assume stationary statistics, a small learning rate (i.e., constant of proportionality in equation 2.7) should be used to minimize the effects of such variability.

For the scenario in Figure 1, as noted, the observation of x generates a prior prediction for z that is present when y occurs. Thus, via equation 2.7, y receives less credit for z than x for y at each $\delta$ (see Figure 5), even though the x–y and y–z pairwise associations are the same (see Figure 6). As a practical matter, since the learning of $\mathbf{C}$ depends on the accurate learning of $\mathbf{M}$, for best results, $\mathbf{C}$ should be learned only after $\mathbf{M}$ stabilizes during training.

Figure 6:

For event pairs, credit density can differ despite having the same pairwise associations. A summary of the event pair associations (top) and credit densities (bottom) for all nontrivial event pairs for the scenario in all previous figures, where the events x, y, and z occur on each trial at times 0, 1, and 2, respectively. The pairwise associations $M_{\mathrm{X}}^{\mathrm{Y}}$ and $M_{\mathrm{Y}}^{\mathrm{Z}}$, overlapping perfectly, are slightly displaced for clarity. However, $C_{\mathrm{X}}^{\mathrm{Y}}$ is greater than $C_{\mathrm{Y}}^{\mathrm{Z}}$ due to the memory of x allowing a prior prediction for z when y occurs, as shown in Figure 5(f).

2.3. Summary

The agent's memory $\tilde{\mathbf{f}}$ encodes a time line of past events (see equation 1.2). Using Hebbian association, the agent makes pairwise associations $\mathbf{M}$ between each pair of event types as a function of internal time (see equation 1.5). This lets the agent form a pairwise prediction $\mathbf{m}$ for the future whenever events occur, but only based on the pairwise correlations associated with those events as cues. To predict future events based on past events, the agent learns credit associations $\mathbf{C}$ between each pair of event types as a function of internal time. The agent uses $\mathbf{C}$ and $\tilde{\mathbf{f}}$ to generate a time line of future events (see equation 2.2). While the agent learns, each time an event occurs, we step $exp{C}_{\alpha }^{\beta }$ (where $\alpha$ is the event that occurred) toward the ratio of the prediction for $\beta$ due to $\alpha$ (based on $\mathbf{M}$), to the prediction for $\beta$ prior to $\alpha$ (based on $\mathbf{C}$) (see equation 2.7). This design curbs double-counting of correlations for an outcome associated with multiple cues at different points in the past. Through learning, we expect the agent to produce better and better predictions for events in its future.

3. Properties of the Prediction Algorithm

The algorithm described above has interesting computational properties. We will discuss how it scales with the number of event types that can be distinguished and the timescales over which prospection is implemented. It can be shown that the model is optimal for pairwise predictions modulo the uncertainty that comes from finite temporal resolution of memory. Moreover, the model is invariant to rescaling of time, which may be useful in applications where the relevant time scale is not known a priori.

3.1. Scaling Properties

As with traditional associative models, the computational time and space required for this algorithm vary quadratically with the number of event types considered. In typical RL models, each state $s$ must be defined to include all of the information that could affect the transition to the next state in order to fit into the Markov structure. If transitions depend on the indefinite past, the number of possible states would become unwieldy. In contrast, the event types used here are economically defined to be those events that occupy a single point in time (e.g., x, y), which are much smaller in number.

In addition, this algorithm runs in time and space polynomial in the number of $\stackrel{\ast }{\tau }$ time points considered in $\tilde{\mathbf{f}}\left(\stackrel{\ast }{\tau }\right)$ and $\delta$ time points considered in $\mathbf{p}\left(\delta \right)$. For example, in equation 2.3, for each $\delta$, the numerical integral is computed in time linear in the number of $\stackrel{\ast }{\tau }$, the variable of integration, corresponding to how far in the past memories are considered. The full prediction, over all $\delta$ that the agent considers, is computed in time linear in the number of $\delta$. Translation to different values of $\delta$ can be implemented serially, consistent with neural considerations (Shankar et al., 2016), or be parallelized in silico. The quick performance comes at the cost of the ability to directly handle some forms of joint statistics among cues. We discuss this shortcoming in section 5.2.

The longest timescale over which predictions are based and are made increases exponentially with computational demands. Although the integral form in equation 1.2 would seem to require memory for the entire history up to the present, $\tilde{\mathbf{f}}$ can be generated from leaky integrators with a number of time constants (Shankar & Howard, 2013). The scale invariance of ${\Phi }_k$ allows us to choose the distribution of time constants as a geometric series, resulting in a logarithmic relationship between the number of integrators and the longest timescale that can be represented.

3.2. Equivalence of Fuzzy Memory and Input Temporal Uncertainty

Even when the time interval between events is fixed, fuzzy memory (finite $k$) leads to temporal fuzziness in both the pairwise association $\mathbf{M}$ and prediction $\mathbf{p}\left(\delta \right)$. At every instant in time, this induced fuzziness is equivalent, in its effect on the prediction, to fuzziness due to intrinsic temporal uncertainty in the signal $f$ faced by an agent with perfect memory (infinite $k$).

As an example, consider an agent with fuzzy memory encountering x, followed by y after a fixed time interval $\tau$. Precisely at the time x occurs, the agent's prediction for y is given by

$$p^{\mathrm{Y}}(\delta ;t_{\mathrm{X}})=\frac{{\kappa }_1^{-1}}{\delta }{\Phi }_k\left(\frac{\tau }{\delta }\right),$$ (3.1)

where ${\kappa }_1$ is as given in section A.3. Another agent with perfect memory encountering x, followed by y after a random time interval $\tau$, whose probability density function is given by $q_{\tau }\left(t\right)={\Phi }_k(t/\delta )/\delta$, makes an optimal prediction following x equivalent to equation 3.1. The derivation of equation 3.1 is given in section A.5.1.

Although the fuzzy memory agent's prediction for y some time after encountering x is different from equation 3.1, this equivalence property still holds: at every instant in time, there exists a perfect memory agent, with observations subject to some density function of $\tau$, with an equivalent optimal prediction.

3.3. Timescale Invariance

The prediction algorithm inherits the timescale invariance of the temporal record of the past. If the input signals are time-dilated, the resulting predictions would be time-dilated, rescaled in magnitude and otherwise unchanged (see Figure 7). Therefore, the prediction algorithm, with an appropriate range of $\stackrel{\ast }{\tau }$ and $\delta$, supports chains of events that happen over any timescale.

Figure 7:

Predictions are timescale–invariant. Top: Credit density and, as an example, the prediction after x occurs for y and z, as a function of internal future time, $\delta$, for the scenario in all previous figures. Middle, bottom: When the scenario is time-dilated, shown here by 10 and 100 times, the model output is unchanged as a function of dilated internal time. In the case of predictions, the magnitude rescales to preserve the area under the curve. This suggests that the proposed algorithm supports chains of events that occur over any timescale.

Formally, for any constant $\lambda$, the estimated probability of event occurrence within a small duration $d\delta$, $p\left(\delta ;t\right)d\delta$, is invariant under the transformation

$$\begin{array}{ccc}\hfill t& \to & \lambda t,\hfill \\ \hfill \stackrel{\ast }{\tau }& \to & \lambda \stackrel{\ast }{\tau },\hfill \\ \hfill \delta & \to & \lambda \delta .\hfill \end{array}$$

This means that within the limits of a computational implementation, that is, far from the smallest and largest values of $\stackrel{\ast }{\tau }$ and $\delta$ (which grow exponentially with the resources committed to representing time), the model provides the same relative temporal resolution.

One may wonder whether, as an alternative to computing $\mathbf{C}$, one can generate a future time line $\mathbf{p}\left(\delta \right)$ and directly update it using ${\mathbf{p}}^{+}$ and ${\mathbf{p}}^{-}$ whenever an event occurs. A difficulty with this approach is that a timescale would have to be chosen for the evolution of $\mathbf{p}\left(\delta \right)$ between events, violating the timescale invariance property that we desire.

3.4. With Fuzzy Memory, Credit Is Assigned Based on Temporal Proximity

Consider the scenario where x occurs, then y, then z, always with the same time delays. In the limit of perfect memory, y would receive no credit for z. This is because the occurrence of x would allow the time of occurrence of z to be predicted perfectly at all times. The occurrence of y would not improve the (already perfect) prediction. When memory is fuzzy, the x–z pairwise association would have a larger temporal uncertainty than the y–z pairwise association, since y and z are closer in time than x and z (see equation 1.5). Therefore, the occurrence of y would improve the prediction for z. The closer y occurs to z, the more y sharpens the prediction for z and the more credit is assigned to y for z. Figure 8 illustrates this effect, and supporting equations are worked out in section A.5.2.

Figure 8:

Temporal proximity promotes credit assignment. Events x, y, and z occur at times 0, $2-t_{\mathrm{Y}\mathrm{Z}}$, and 2, respectively. (a) A schematic of the state of memory and prediction at the time y occurs, for six values of $t_{\mathrm{Y}\mathrm{Z}}$. The axes have the same interpretation as in Figure 3. (b) Credit assigned to y for z is shown here for the six values of $t_{\mathrm{Y}\mathrm{Z}}$, as a function of internal future time, $\delta$. In other words, each line represents different amounts of temporal proximity between y and z, while the interval between x and z remains fixed. For $t_{\mathrm{Y}\mathrm{Z}}=1.9$, y is much closer in time to x than to z. In this case, the credit is almost flat, as the prediction for z due to x is still fresh. The case $t_{\mathrm{Y}\mathrm{Z}}=1$ is the scenario in Figure 1 through 6. For lower and lower values of $t_{\mathrm{Y}\mathrm{Z}}$, credit density is more and more sharply peaked. The prediction for z due to x has flattened out, allowing the effect of the pairwise association between x and z to dominate. The analytic form of the lines plotted is worked out in section A.5.2.

4. Demonstration: Event Streams with Memory and Multiple Characteristic Timescales

We have seen that the algorithm described here is able to predict the future based on a temporally extended record of the past containing multiple possible cues. In addition, this prediction does not require selection of a preferred timescale, allowing for generalization across an exponentially large range of times. As a consequence of these two properties, this approach is well suited to applications where the relevant timescale is not known a priori or to situations where there are multiple processes at different characteristic timescales that must be simultaneously learned. To illustrate these properties, we demonstrate learning of the algorithm on a time series of discrete events generated from multiple Markov renewal processes (MRP).

In principle, the algorithm we describe is capable of handling multiple cues with additive effects (but see section 5.2) stretching into the indefinite past. However, for simplicity, we generate a scenario such that each event has exactly one cue. This cue is mostly found at most 15 time steps before the event. For comparison, most consecutive events have an intervening time of between 0.1 and 15 time units. Crucially, the cue is not usually the immediately preceding event, but one of the several preceding events. Thus, one cannot merely predict the future based on the most recent event. To add realism, we introduce a small amount of variability in the event type of the outcome, as well as a small amount of gaussian variability in the time of the outcome.

The way we generated a scenario with such properties is to superpose several MRPs, each with three base event types, u, v, w. MRPs have the property that the type of each event is the sole determiner of the probability distribution of the type and time of the next event. In other words, each event has a single cue. Superposing MRPs destroys the guarantee that the cue immediately precedes its outcome. We generated the scenario using two approaches, mainly differing in the way event types are determined in the superposed process. For the first approach, event types in the superposed process are determined according to the base type of the event and the MRP of origin. For example, for a superposition of 7 MRPs, there would be $3\times 7=21$ event types ($1\text{u},1\text{v},1\text{w},\dots ,7\text{w}$). An example of such a scenario with two MRPs superposed is shown in Figure 9a. Figure 9b shows the corresponding mean transition times for each type of transition. The drawback of this approach is that as the number of MRPs increases, the number of event types increases, making the prediction task inherently harder. For the second approach, event types in the superposed process are determined only according to the base types of the events, even if they originate from different MRPs. This way, for the prediction of the type of an event, there are always two wrong answers and one correct answer, for a fair comparison regardless of the number of MRPs superposed.

Figure 9:

The algorithm provides good predictions for cues at multiple timescales. (a) The top panel shows the first few events in a superposed process. The bottom two panels show the corresponding events from the two component MRPs, which are composed of events $\{1\text{u},1\text{v},1\text{w}\}$ and $\{2\text{u},2\text{v},2\text{w}\}$, respectively. Note that successive events in the superposed process (e.g., the first two events in the top-most panel) may be from different MRPs, and thus the earlier event not predictive of the later event. (b) Graph depicting mean transition times between event types within each component MRP. Weights are associated with the arrowhead closest to them. The variances of the normally distributed transition times are not shown here. Note, for example, how the $1\text{v}\to 1\text{u}$ transition takes place at the scale of about 1.5 time units, while the $1\text{u}\to 1\text{w}$ transition takes place at a different scale of about 10 time units. The two MRPs depicted here are two of the component MRPs in the simulation used to generate panel c. (c) We superpose MRPs such that event types from different MRPs are deemed different event types in the superposed process. (d) We superpose MRPs such that event types from different MRPs are identified by base types (u, v, w) irrespective of their MRP of origin, resulting in exactly three event types in the superposed process. For panels c and d, each point represents an average accuracy computed by repeating the training and testing procedures six times for each choice of number of MRPs superposed. Regardless of method of superposing MRPs, the algorithm (labeled $\mathcal{C}$) performs well above chance, showing that it provides good predictions for cues at multiple timescales. See text for a comparison of the $\mathcal{C}$- and $\mathcal{M}$-based predictions.

The algorithm we describe can be used to predict both the time and type of likely events in the future. However, for simplicity, we evaluate the algorithm on its average accuracy of predicting the type of the next event, given the time to the next event, whenever an event occurs. We generate this prediction via ${\text{argmax}}_i{p}^i(\delta =t_{n+1}-t_n;t=t_n)$, where $t_n$ is the time of the $n$th event. We call this the $\mathcal{C}$-based prediction. As a comparison, we generate an $\mathcal{M}$-based prediction via ${\text{argmax}}_i{m}^i(\delta =t_{n+1}-t_n;t=t_n)$, where $j$ is the type of the event at $t_n$, and evaluate its average accuracy. Notice that the $\mathcal{M}$-based prediction only invokes pairwise associations with event $j$ as the cue, whereas the $\mathcal{C}$-based prediction invokes credit associations with current and past events as cues. Finally, we compare these to a baseline of always predicting the most frequent event type. Our method is described in detail in appendix A.6.

The average accuracies of the prediction methods are shown in Figures 9c and 9d, as a function of the number of MRPs superposed, for the first and second approach of scenario generation, respectively. The $\mathcal{C}$- and $\mathcal{M}$-based predictions generally outperform the baseline model. Across both figures, the results are qualitatively similar. The accuracies of $\mathcal{C}$- and $\mathcal{M}$-based predictions are comparable for a single MRP. This is expected since for an MRP, the cue and its outcome are neighbors. Whenever an event occurs, the $\mathcal{M}$-based predictor uses the pairwise associations between that event and its possible outcomes to predict the type of the next event. However, as more and more MRPs are superposed, the $\mathcal{C}$-based algorithm outperforms.

What drives the difference in performance between the $\mathcal{C}$- and $\mathcal{M}$-based algorithms? Although the $\mathcal{C}$-based algorithm uses the credit associations $\mathbf{C}$ while the $\mathcal{M}$-based algorithm uses the pairwise associations $\mathbf{M}$, this difference is immaterial in this case. Since each event only has one cue, $exp{\mathbf{C}}^{\alpha }$ is proportional to ${\mathbf{M}}^{\alpha }$. Rather, the $\mathcal{M}$-based algorithm suffers when successive events originate from separate MRPs and the pairwise association between the respective event types would not be predictive. The $\mathcal{M}$-based algorithm makes predictions only based on events in the present. In contrast, the $\mathcal{C}$-based algorithm makes predictions based on events in the present and in the past, where the correct cue would be included in such situations.

This demonstration provides a proof of concept that the algorithm provides reasonable predictions for cues at timescales spanning one order of magnitude. We accomplished this without selecting any single operating scale. The demonstration gives a flavor for the advantages of the algorithm we describe over Markov models. A classic approach based on $n$th-order Markov models would entail discretizing time at some lowest-level scale (but see Kurth-Nelson & Redish, 2009; Ludvig, Sutton, & Kehoe, 2008) and sizing the memory buffer to encompass most of the longest transitions. For simplicity, we have constructed a relatively tame scenario for this demonstration, in which most event relationships only span about 1 to 15 time units, and events are sparse. In reality, the wider the range of timescales, the harder it is for standard algorithms operating at the lowest-level timescale, which fumble at timescales significantly different from their operating scale (Mozer, 1992). In scenarios where events have long-range temporal dependencies, Markov models would be significantly limited by the exponential growth in the number of states (and, thus, computational demands) with the size of the memory buffer. The algorithm we describe does not face these limitations (see section 3.1).

5. Discussion

We have proposed an algorithm that generates a scale-invariant time line of the future. This algorithm is time-local in the sense that predictions at time $t$ are derived from $\tilde{\mathbf{f}}(\stackrel{\ast }{\tau },t)$, which represents events that are, in fact, nonlocal in real time. Moreover, the translation mechanism enables event rates at future time points to be estimated. In addition to associative memory, as developed by model-free RL algorithms, this capability would let an agent construct the estimate over possible futures (McGuire & Kable, 2013).

5.1. Theoretical Properties of the Current Model

This model has properties that are quite different from traditional RL paradigms. First, this algorithm naturally runs in continuous time, which suits applications dealing with natural processes unfolding in time. This feature contrasts with basic RL algorithms, which only allow agents to move among discrete states in discrete time. In principle, this proposed algorithm can be extended such that position in higher-dimensional spaces replaces or augments time, allowing agents to navigate real and abstract spaces. Translation can be along an angle or perhaps even along a trajectory instead of being confined to a given axis (see equation 1.4).

Second, the scale invariance of the model is useful in applications where the timescale of event relationships is not known in advance. In principle, the model is indifferent toward the absolute time intervals between events. Instead, within a given scenario, it is only concerned about time intervals relative to other time intervals. In comparison, in traditional RL systems, a timescale for history dependency, if any, is set by the size of the history that the designer defines as part of the state $s$. Moreover, in many aspects of the world that we might be interested in, such as in natural language (Altmann, Cristadoro, & Esposti, 2012), network traffic (Cohen, Erez, Ben-Avraham, & Havlin, 2000), and financial markets (Cont, 2005), event dependencies exist simultaneously across a wide range of scales. This model is potentially suited for such applications, since it incorporates past events across a range of timescales, and an increase in computing resources provides an exponential increase in the length of history considered.

Third, in the context of RL, this model may be incorporated into algorithms to allow agents to naturally form a prediction of its own trajectory as a function of future time. This can be done by considering the agent's arrival at some or all states as events. In addition, by combining the predictions for future states $s$ as a function of future time, $p^s\left(\delta \right)$, with a reward function over future states, the agent can generate the predicted future reward as a function of future time, $r\left(\delta \right)$. By learning and comparing weighted integrals of $r\left(\delta \right)$ for several alternative policies, the agent can choose flexibly among these policies according to task demands. For instance, if the agent knows it only has 10 time units to complete the task, it can choose the policy with the highest ${\int }_0^{10}r\left(\delta \right)d\delta$. The model's ability to form a prediction as a function of future time stands in contrast to RL paradigms, which tend to flatten the dimension of future time. For example, a naive RL agent assigns values to states according to the expected sum of future reward starting from that state; a successor representation agent (Dayan, 1993) learns the expected future state occupancy, summed over future time, starting from each state (but see Tano, Dayan, & Pouget, 2020; Momennejad & Howard, 2018).

Finally, we note that this model provides information usually associated with model-based RL but with very different computational properties. Like model-based RL, this model provides an explicit prediction as a function of future time $\delta$. However, a constraint of model-based RL is that the time to compute an event $\delta$ in the future goes up linearly with $\delta$. In the present model, because the calculation of the prediction at a particular value of $\delta$ does not depend on the prediction at previous values of $\delta$, one could in principle compute all values of $\delta$ in parallel. Moreover, this means that it is possible to sample the $\delta$ axis in whatever way is convenient. Integrals over $\delta$ give hyperbolic discounting if the $\delta$ axis is sampled evenly as a function of $log\delta$. See also Shankar et al. (2016) for considerations related to physically instantiating translation across a population of neurons.

5.2. Theoretical Limitations of the Current Model

We highlight two limitations relating to applying this algorithm toward machine learning. First, the algorithm, as currently described, is not directly sensitive to joint statistics of two or more cues. For example, the model would be unable to capture the conditional structure “z occurs exactly if either x or y occurs, but not both.” As a consequence, the algorithm is also unable to deal appropriately with number of events. For instance, the algorithm has no basis to differentiate “x precedes y by 10 s” from “half of the time, x precedes two closely spaced occurrences of y by 10 s and the other half of the time, y does not occur.” We can mitigate this issue by perceiving events depending on context. For example, the agent can perceive the y after an x as the event xy, enabling sensitivity to joint statistics of at most two cues. In terms of computational complexity, naively implementing this would introduce a quadratic factor in the number of base events. However, we can reduce the resource complexity by finding a compressed representation of the event history while preserving information about future events—that is, dealing with the information bottleneck problem (Tishby, Pereira, & Bialek, 2000). Since existing deep neural network algorithms efficiently extract joint statistics, it would be natural to pursue research that seeks to merge this approach with deep network algorithms.

Second, this algorithm is limited in prescribing how to achieve optimal policies in the context of RL. Our focus has been on how to predict future events, not how to learn the best policy. In many contexts, it is natural to define events such that events occur depending on actions of the agent (e.g., in a spatial navigation task where events occur based on the agent's trajectory). In these cases, in effect, we presume that the agent follows an existing policy $\pi$, and the model deduces event associations and makes predictions with respect to $\pi$. The agent can certainly flexibly choose among several alternative policies, say, between $\pi$ and ${\pi }^{\text{'}}$, by comparing predictions from the start state and selecting the more rewarding alternative. However, unlike basic RL algorithms, we do not prescribe a method for learning a policy that scales in complexity with the number of states, such as a policy to navigate a grid. In the context of grid navigation, we have, in effect, avoided assigning values to coordinates on the grid, since this contradicts our design principle of allowing history to influence events (rewards). More research would be needed if one wished to pursue policy learning within the framework we describe.

5.3. Neuroscience Considerations

We now discuss two potential points of contact between the formal model presented in this letter and computational and systems neuroscience.

5.3.1. Reward Prediction Error and Dopamine

The success of RL algorithms in accounting for the firing of dopaminergic neurons in the basal forebrain (Schultz et al., 1997) is arguably the greatest achievement in computational cognitive neuroscience. The basic empirical story is well known. Dopaminergic neurons respond to unpredicted rewarding outcomes. However, with learning, as the reward becomes predicted by a neutral stimulus, the cells no longer fire to the predicted reward but instead fire to the neutral stimulus that predicts the future rewarding outcome (see Schultz, 2006, for a review of the early literature). While there are undoubtedly many details that would need to be worked out, at least the rough outline of the classical empirical story about dopamine can be mapped onto this framework.

Let us suppose that expected future value is computed at each moment by integrating over future time $\delta$, taking the projection from the vector of predicted events $\mathbf{p}\left(\delta \right)$ onto some vector $\mathbf{A}$ that describes the reward value of each possible event type in $\mathbf{p}$:

$$V\left(t\right)={\int }_0^{\infty }\mathbf{A}·\mathbf{p}(\delta ;t)g\left(\delta \right)d\delta ,$$ (5.1)

where $g\left(\delta \right)$ denotes the factor arising from compression of the $\delta$ axis. As discussed above, it is reasonable to sample $\delta$ on a logarithmic scale to implement hyperbolic discounting, in which case, $g\left(\delta \right)=1/\delta$. Let us suppose further that reward prediction error $E$ is computed as the difference between $r\left(t\right)$, the actual reward observed at time step $t$, and the change in $V$ at time step $t$:

$$E\left(t\right)=r\left(t\right)+\left[V\left(t\right)-V(t-\Delta t)\right],$$ (5.2)

where we have chosen a discrete time interval $\Delta t$ to acknowledge that the computation of value may take a substantial amount of time in the brain. For instance, Shankar et al. (2016) proposed that integrals over $\delta$ could be completed within a theta oscillation, suggesting $\Delta t$ could be as long as a few hundred milliseconds. Now, consider slowly learning an association between an inherently neutral event x and a rewarding event y, separated by a fixed delay $\tau$. Initially, y is unpredicted. When x is presented, there is no change in $V$. Similarly, $V$ is zero both before and after y is presented. Because y is rewarding, the reward prediction error is positive around the time of presentation of y. After learning, immediately after x is presented, $\mathbf{p}\left(\delta \right)$ includes the prediction for y, a time $\delta \simeq \tau$ in the future. This means that $V\left(t\right)$ changes abruptly around the time that x is presented, resulting in a positive reward prediction error. Now, after learning, consider a time $\tau$ after presentation of x. If the rewarding stimulus is omitted, negative reward prediction error is observed as the peak in $\mathbf{p}\left(\delta \right)$ corresponding to y becomes increasingly truncated. However, if y is presented at the time it is expected, then the positive reward from y is balanced by a rapid decreasing $V\left(t\right)$. Note that because y does not predict itself at a short lag, observation of y abruptly decreases the prediction of itself.

This approach aligns well with the classic understanding of reward prediction error with one very important exception. Rather than estimating expected future reward via temporal difference learning, predictions for an extended future are available at each moment. Unlike temporal difference learning algorithms, there is no sense in which value moves gradually along intermediate time points between x and y. This model thus has no difficulty accounting for the finding that value seems to rapidly “jump” between events (Pan, Schmidt, Paton, & Hyland, 2005).

5.3.2. Translation and Theta Oscillations

The algorithm described here relies on the ability to translate $\tilde{\mathbf{f}}$ toward the past. Shankar et al. (2016) suggested that hippocampal theta (4–12 Hz) oscillations could provide a mechanism for translation of temporal representations. The basic conjecture of that model for translation is that different values of $\delta$ map onto different phases of theta oscillations. If the time line $\delta$ maps onto different phases of the theta oscillation, this places a lower limit on the order of 100 ms on the time lines indexed by $\stackrel{\ast }{\tau }$ and $\delta$. Theoretical and neurobiological considerations led Shankar et al. (2016) to the conclusion that $\delta$ ought to accelerate exponentially with the theta cycle, resulting in a logarithmic sampling of the $\delta$ axis.

This conjecture made sense of several neurophysiological findings, including the gradual ramping of firing in striatal neurons accompanied by phase precession with respect to theta recorded in the hippocampus, a brain structure that is relatively distant from the striatum (van der Meer & Redish, 2011). The fact that spikes in the striatum are organized by hippocampal theta suggests that theta oscillations reflect a computation that is extended over a significant part of the brain. The learning rule presented here, equation 2.7, describes changes in the strength of connections in $\mathbf{C}$ by noting the difference between ${\mathbf{p}}^{+}$ and ${\mathbf{p}}^{-}$ at each value of $\delta$. This suggests convergent connections between axons communicating $\mathbf{M}$ and $\mathbf{C}$ arriving at target neurons representing predicted future outcomes. Perhaps the coordination implied by the involvement of theta oscillations in prediction could lead to a difference in the timing of spikes communicated via $\mathbf{M}$ and $\mathbf{C}$. Coupled with spike-timing-dependent plasticity, perhaps this could lead to the learning rule in equation 2.7.

Appendix

A.1. A Formal Model for Temporal Record of the Past

Let multiple types of discrete events occur in continuous time. For each event type, we denote the signal by $f\left(t\right)$, where each event is represented by a Dirac delta function at the instant it occurs. For each event type, an array of leaky integrators, $F$, with a range of decay rates $s$, receives the signal as input:

$$\frac{\partial }{\partial t}F(s;t)=-sF(s;t)+f\left(t\right).$$ (A.1)

The array of leaky integrators $F(s;t)$ encodes the real Laplace transform of the signal up to time $t$, where $s$ is the Laplace domain variable. For each event type, an array of time cells $\tilde{f}\left(\stackrel{\ast }{\tau }\right)$ approximately inverts the Laplace transform (see Post, 1930). This yields an estimate of the signal up to time $t$, at time offsets $\stackrel{\ast }{\tau }$ prior:

$$\tilde{f}(\stackrel{\ast }{\tau };t)=\tilde{f}(k/s;t)=\frac{{(-1)}^k}{k!}{s}^{k+1}\frac{{\partial }^k}{\partial s^k}F(s;t)={\mathbf{L}}_k^{-1}F(s;t).$$ (A.2)

The constant $k$ is a sharpness parameter. As $k\to \infty$, the estimate $\tilde{f}\left(\stackrel{\ast }{\tau }\right)$ becomes precise, at the cost of infinite resources to implement the model. As stated in equation 1.2, for an arbitrary signal $\mathbf{f}$,

$$\tilde{\mathbf{f}}(\stackrel{\ast }{\tau };t)=\frac{1}{\stackrel{\ast }{\tau }}{\int }_{-\infty }^t\mathbf{f}\left(\tau \right){\Phi }_k\left(\frac{t-\tau }{\stackrel{\ast }{\tau }}\right)d\tau .$$ (A.3)

In other words, for a given $\stackrel{\ast }{\tau }$, $\tilde{\mathbf{f}}(\stackrel{\ast }{\tau };t)$ is proportional to a causal convolution of the signal $\mathbf{f}$ with a kernel ${\Phi }_k$ that describes the smearing.

A.2. Time Translation to Estimate the Future State of the Past

The future state of the memory (see equation 1.4) can be readily computed through translation in the Laplace domain:

$${\tilde{\mathbf{f}}}_{\delta }(\stackrel{\ast }{\tau };t)\equiv {\mathbf{L}}_k^{-1}{\mathbf{R}}^{\delta }\mathbf{F}(s;t)\equiv {\mathbf{L}}_k^{-1}\left\{e^{-s\delta }\mathbf{F}(s;t)\right\}.$$ (A.4)

Building a translation operator out of realistic neurons and synapses is a nontrivial but tractable problem. It has been proposed that the brain implements translation to various amounts $\delta$ by mapping $\delta$ on to different phases of theta oscillations (Shankar et al., 2016). Previous work has long argued that theta oscillations, a prominent 4–12 Hz oscillation in the local field potential, have long been believed to be crucial in the neurobiology of memory (Buzsáki, 2002; Hasselmo, Bodelón, & Wyble, 2002; Kahana, Seelig, & Madsen, 2001). Requiring scale invariance and also consideration of the problem from the perspective of the individual neurons requires the sweep through $\delta$ to accelerate exponentially through the theta cycle.

A.3. Pairwise Association and Pairwise Prediction

The agent makes pairwise associations $\mathbf{M}$ between each pair of event types using Hebbian learning. As the agent experiences the world, the pairwise prediction $\mathbf{m}$ allows the agent to generate predictions for the future based on pairwise associations with the currently occurring events as cues. The pairwise prediction is a building block for the learning of the credit associations $exp\mathbf{C}$, from which the prediction $\mathbf{p}$ is derived. This section consists of two subsections. The first subsection motivates the form of the pairwise prediction $\mathbf{m}$ (see equations 1.10 to 1.12; in particular, the form of the integral in equation 1.11). The second subsection highlights and proves the numerical coincidence between $\mathbf{M}$ and $\mathbf{m}$ in a simple case, from which the normalization for $\mathbf{m}$ derives.

A.3.1. Equation for Pairwise Prediction

In this subsection, we motivate the equations for the pairwise prediction. We do this by showing that when memory is perfect, $\mathbf{m}$ reduces to the geometric mean of the elements of $\mathbf{M}$ associated with the events at time $t$, as desired.

Hebbian learning can be used to make pairwise associations $\mathbf{M}$ between events (see equation 1.5). The parameters of $\mathbf{M}$ are the possible cue, the possible outcome, and the internal time. The pairwise prediction $\mathbf{m}$ uses the pairwise associations $\mathbf{M}$ to make a prediction about future events based on possibly multiple currently occurring events. In other words, $\mathbf{m}$ serves to integrate cue–outcome pairwise information from multiple simultaneous cues in the present. $\mathbf{M}$, in the definition for $\mathbf{m}$, plays the role that $exp\mathbf{C}$ does in the definition for $\mathbf{p}$. In the algorithm, $\mathbf{m}$ serves the function of ${\mathbf{p}}^{+}$ (see equation 2.6).

The pairwise prediction $\mathbf{m}$ is computed when a set of events ${\mathcal{E}}_t$ occur at time $t$, as follows:

$$m^{\beta }\left(\delta ;t\right)={\kappa }_{1}exp\left(\frac{1}{\left|{\mathcal{E}}_t\right|}\sum _{\alpha ∊{\mathcal{E}}_t}\int f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)log{M}_{\alpha }^{\beta }\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\right),\delta >0,$$ (A.5)

where the constant ${\kappa }_1={\left[k{e}^{-\psi \left(k\right)}\right]}^{k+1}$, $\psi \left(k\right)$ is the digamma function, and

$$f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)={\Phi }_k\left(\delta /\stackrel{\ast }{\tau }\right)/\stackrel{\ast }{\tau }$$

denotes the future state of the memory element associated with the currently occurring episode of $\alpha$. (For $k=2$, ${\kappa }_1\sim 2.3$; for $k=8$, ${\kappa }_1\sim 1.8$; as $k\to \infty ,$ ${\kappa }_1\to \sqrt{e}\sim 1.65$.) The notation $\left|{\mathcal{E}}_t\right|$ denotes the number of elements in ${\mathcal{E}}_t$, that is, the number of events co-occurring at time $t$. The function $f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)$ is a gaussian-like function that peaks around $\stackrel{\ast }{\tau }=\delta$ and reflects the fact that at a time $\delta$ in the future, $\alpha$ would have occurred $\stackrel{\ast }{\tau }$ in the past and the memory ${\tilde{f}}^{\alpha }$ would reflect this.

We can motivate the form of the pairwise prediction $\mathbf{m}$ as follows. Let us imagine that memory were perfect such that events were localized in time exactly, so ${\varphi }_{\delta }^{\alpha }\left(\stackrel{\ast }{\tau }\right)=\delta (\stackrel{\ast }{\tau }-\delta )I(\alpha ∊{\mathcal{E}}_t),$ where ${\varphi }_{\delta }^{\alpha }$ is the analog of $f_{\delta }^{\alpha }$ when memory is perfect (i.e., $k\to \infty$), $\delta (·)$ is the Dirac delta function, and $I(·)$ is the indicator function. We would then have

$$\begin{array}{ccc}\hfill \int {\varphi }_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)log{M}_{\alpha }^{\beta }\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }& =& I(\alpha ∊{\mathcal{E}}_t)\int \delta (\stackrel{\ast }{\tau }-\delta )log{M}_{\alpha }^{\beta }\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\hfill \\ & =& I(\alpha ∊{\mathcal{E}}_t)log{M}_{\alpha }^{\beta }\left(\delta \right).\hfill \end{array}$$

The integral on the right-hand side is a convolution of $log{M}_{\alpha }^{\beta }$ with the delta function, which returns the former unchanged. In the case where exactly $x$ occurs at time $t$,

$$\begin{array}{ccc}\hfill m^{\beta }\left(\delta ;t\right)& \propto & exp\int f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)log{M}_{\alpha }^{\beta }\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\hfill \\ & =& explog{M}_{\mathrm{X}}^{\beta }\left(\delta \right)\hfill \\ & =& M_{\mathrm{X}}^{\beta }\left(\delta \right),\hfill \end{array}$$ (A.6)

so $\mathbf{m}$ would be proportional to the appropriate elements of $\mathbf{M}$. In the case where exactly $x$ and $y$ occur at time $t$,

$$\begin{array}{ccc}\hfill m^{\beta }\left(\delta ;t\right)& \propto & exp\left[\frac{1}{\left|{\mathcal{E}}_t\right|}\sum _{\alpha ∊{\mathcal{E}}_t}\int f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)log{M}_{\alpha }^{\beta }\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\right]\hfill \\ & =& exp\left\{\frac{1}{2}\left[log{M}_{\mathrm{X}}^{\beta }\left(\delta \right)+log{M}_{\mathrm{Y}}^{\beta }\left(\delta \right)\right]\right\}\hfill \\ & =& \sqrt{M_{\mathrm{X}}^{\beta }\left(\delta \right)M_{\mathrm{Y}}^{\beta }\left(\delta \right)},\hfill \end{array}$$

so $\mathbf{m}$ would be proportional to the geometric mean of the elements of $\mathbf{M}$ associated with the events at time $t$.

In the model with fuzzy memory, $f_{\delta }$ approximates ${\varphi }_{\delta }$, so the above relationships hold only approximately. In other words, in general, according to equations 1.10 to 1.12, $m^{\beta }\left(\delta ;t\right)$ is not always exactly $M_{\mathrm{X}}^{\beta }\left(\delta \right)$ (when $x$ occurs at time $t$) or $\sqrt{M_{\mathrm{X}}^{\beta }\left(\delta \right)M_{\mathrm{Y}}^{\beta }\left(\delta \right)}$ (when $x$ and $y$ occur at time $t$), and so on. Instead of using equations 1.10 to 1.12, which involves an integral, one could have defined $m^{\beta }\left(\delta ;t\right)$ directly as the geometric mean of the relevant elements of $M^{\beta }\left(\delta \right)$. However, we do not do so. We use equations 1.10 to 1.12 because they closely parallel equations 2.2 to 2.3 for the prediction, for which the integral is necessary (see section A.4). Using equations of similar form is more neurobiologically realistic, because it suggests that analogous neural architecture supports the computation for both pairwise prediction $\mathbf{m}$ and prediction $\mathbf{p}$. Integrals in time are straightforward to implement with neural networks.

In summary, we have shown that when memory is perfect, $\mathbf{m}$ reduces to the geometric mean of the elements of $\mathbf{M}$ associated with the events at time $t$, as desired. The strength of the equations underlying $\mathbf{m}$ is that they closely parallel those for $\mathbf{p}$.

A.3.2. Normalization for Pairwise Prediction

In the bulk of section A.3.1, we imagined that memory were perfect such that events were localized in time exactly. In the actual formulation, memory is fuzzy, and this is reflected in the form of ${\mathbf{f}}_{\delta }$. Therefore, the above proportionality relationships do not hold exactly in general. However, equation A.6 holds at the time of occurrence of $x$ in the case where $x$ and $y$ occur at a constant time interval $\tau$, and no event co-occurs with $x$, even in the case of fuzzy memory. We will prove this result (in lemma 3), which suggests the value that the normalization constant ${\kappa }_1$ should take. Lemmas 1 and 2 are integrals that are used to prove lemma 3.

Lemma 1.

If the constant $a$ is positive and $k$ is a nonnegative integer,

$${\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+2}}dx=\frac{k!}{a^{k+1}}.$$

Proof.

The integral is

$$I={\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+2}}dx=\frac{1}{a}{\int }_0^{\infty }\frac{a{e}^{-a/x}}{x^2}{x}^{-k}dx.$$

Noting that $\frac{d}{dx}\left(e^{-a/x}\right)=\frac{a}{x^2}{e}^{-a/x}$, we integrate by parts:

$$\begin{array}{ccc}\hfill u& =& x^{-k}dv=\frac{a{e}^{-a/x}}{x^2}\hfill \\ \hfill du& =& \left(-k\right)x^{-k-1}v=e^{-a/x},\hfill \end{array}$$

so our integral is now

$$I={\left[\frac{1}{a}{x}^{-k}{e}^{-a/x}\right]}_0^{\infty }-\frac{-k}{a}{\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+1}}dx=\frac{k}{a}\frac{1}{a}{\int }_0^{\infty }\frac{a{e}^{-a/x}}{x^2}{x}^{-\acute{\left(k-1\right)}}dx.$$

If we were to integrate by parts again, we would have

$$\begin{array}{ccc}\hfill u& =& x^{-\left(k-1\right)}dv=\frac{a{e}^{-a/x}}{x^2}\hfill \\ \hfill du& =& -\left(k-1\right)x^{-k-1}v=e^{-a/x}.\hfill \end{array}$$

Each $i$th iteration reduces the exponent on $x$ in the denominator of the integrand by 1 and introduces a factor of $\left(k-i+1\right)/a$, and $k$ iterations are needed to go from having $x^{k+2}$ to $x^2$ in the denominator of the integrand. Noting that $\left(k\right)\left(k-1\right)\dots \left(2\right)\left(1\right)=k!,$ we thus have

$$I=\frac{k!}{a^k}\frac{1}{a}{\int }_0^{\infty }\frac{a{e}^{-a/x}}{x^2}dx=\frac{k!}{a^k}\frac{1}{a}{\left[e^{-a/x}\right]}_0^{\infty }=\frac{k!}{a^{k+1}}.$$

Lemma 2.

If the constants $A$, $a$, and $b$ are positive and $k$ and $m$ are positive integers, then

$${\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+1}}log\frac{A{e}^{-b/x}}{x^m}dx=\frac{\left(k-1\right)!}{a^k}\left[m\psi \left(k\right)-\frac{kb}{a}+log\frac{A}{a^m}\right],$$

where $\psi \left(k\right)$ is the digamma function.

Proof.

The integrand is

$$\frac{e^{-a/x}}{x^{k+1}}log\frac{A{e}^{-b/x}}{x^m}=\frac{e^{-a/x}}{x^{k+1}}logA-\frac{b{e}^{-a/x}}{x^{k+2}}-m\frac{e^{-a/x}}{x^{k+1}}logx.$$

We integrate term by term. Applying lemma 1, the first term is

$$logA{\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+1}}dx=\left(logA\right)\frac{\left(k-1\right)!}{a^k},$$

noting that $k$ above is at least 1, and the second term is

$$-b{\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+2}}dx=-b\frac{k!}{a^{k+1}}.$$

The third term is

$$-m{\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+1}}logxdx,k\ge 1.$$

Substituting $t=a/x,$ $dt=-a/x^{2}dx$, we have

$$\begin{array}{ccc}& & -\frac{m}{a^{k-1}}{\int }_{\infty }^0{z}^{k-1}{e}^{-t}log\frac{a}{t}\left(-\frac{1}{a}\right)dt\hfill \\ & =& \frac{m}{a^k}{\int }_0^{\infty }{t}^{k-1}{e}^{-t}log\frac{t}{a}dt\hfill \\ & =& \frac{m}{a^k}\left[\left(-loga\right){\int }_0^{\infty }{t}^{k-1}{e}^{-t}dt+{\int }_0^{\infty }{t}^{k-1}{e}^{-t}logtdt\right]\hfill \\ & =& \frac{m}{a^k}\left[\left(-loga\right)\Gamma \left(k\right)+{\Gamma }^{\text{'}}\left(k\right)\right]\hfill \\ & =& \frac{m}{a^k}\Gamma \left(k\right)\left[\psi \left(k\right)-loga\right]\hfill \\ & =& \frac{m}{a^k}\left(k-1\right)!\left[\psi \left(k\right)-loga\right],\hfill \end{array}$$

where we have applied equations 5.2.1, 5.9.19, and 5.2.2 from DLMF (2021), and used the fact that $k$ is a positive integer. Putting everything together, we have

$$\begin{array}{ccc}\hfill {\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+1}}log\frac{A{e}^{-b/x}}{x^m}dx& =& \left(logA\right)\frac{\left(k-1\right)!}{a^k}-b\frac{k!}{a^{k+1}}+\frac{m}{a^k}\left(k-1\right)!\left[\psi \left(k\right)-loga\right]\hfill \\ \hfill & =& \frac{\left(k-1\right)!}{a^k}\left\{logA-\frac{kb}{a}+m\left[\psi \left(k\right)-loga\right]\right\}\hfill \\ \hfill & =& \frac{\left(k-1\right)!}{a^k}\left[m\psi \left(k\right)-\frac{kb}{a}+log\frac{A}{a^m}\right].\hfill \end{array}$$

Lemma 3.

Let event $i$ cue event $j$ with a fixed time interval $t_{ij}$, and let no event co-occur with event $i$. After training, at the time event $i$ occurs,

$$m^j\left(\delta \right)=M_i^j\left(\delta \right),$$

for all $\delta >0.$

Proof.

On the right-hand side, we have

$$M_i^j\left(\delta \right)=K\frac{t_{ij}^k}{{\delta }^{k+1}}{e}^{-k{t}_{ij}/\delta },\delta >0,$$

where $K=\frac{k^{k+1}}{k!}$.

To compute the left-hand side, we note that

$$f_{\delta }^i\left(\stackrel{\ast }{\tau }\right)=K\frac{{\delta }^k}{{\stackrel{\ast }{\tau }}^{k+1}}{e}^{-k\delta /\stackrel{\ast }{\tau }},\stackrel{\ast }{\tau },\delta >0.$$

Then

$$\int f_{\delta }^i\left(\stackrel{\ast }{\tau }\right)log{M}_i^j\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }={\int }_0^{\infty }K\frac{t_{ij}^k}{{\stackrel{\ast }{\tau }}^{k+1}}{e}^{-k{t}_{ij}/\stackrel{\ast }{\tau }}log\left(K\frac{t_{ij}^k}{{\stackrel{\ast }{\tau }}^{k+1}}{e}^{-k{t}_{ij}/\stackrel{\ast }{\tau }}\right)d\stackrel{\ast }{\tau }$$ (A.7)

$$=K{\delta }^k{\int }_0^{\infty }\frac{e^{-k\delta /\stackrel{\ast }{\tau }}}{{\stackrel{\ast }{\tau }}^{k+1}}log\left(\frac{K{t}_{ij}^k}{{\stackrel{\ast }{\tau }}^{k+1}}{e}^{-k{t}_{ij}/\stackrel{\ast }{\tau }}\right)d\stackrel{\ast }{\tau }.$$ (A.8)

Substituting $a=k\delta$, $b=k{t}_{ij}$, $A=K{t}_{ij}^k$, $m=k+1$ into lemma 2, the above evaluates to

$$\begin{array}{ccc}\hfill \int f_{\delta }^i\left(\stackrel{\ast }{\tau }\right)log{M}_i^j\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }& =& K{\delta }^k\frac{\left(k-1\right)!}{{\left(k\delta \right)}^k}\left[\left(k+1\right)\psi \left(k\right)-\frac{k^2{t}_{ij}}{k\delta }+log\frac{K{t}_{ij}^k}{{\left(k\delta \right)}^{k+1}}\right]\hfill \\ \hfill & =& \left(k+1\right)\psi \left(k\right)-\frac{k{t}_{ij}}{\delta }+log\frac{K{t}_{ij}^k}{{\delta }^{k+1}}-\left(k+1\right)logk\hfill \\ \hfill & =& log\frac{K{t}_{ij}^k}{{\delta }^{k+1}}-\frac{k{t}_{ij}}{\delta }-log{\kappa }_1.\hfill \end{array}$$

Thus, the left-hand side of the lemma is

$$\begin{array}{ccc}\hfill m^j\left(\delta \right)& =& {\kappa }_{1}exp\left(\frac{1}{\left|{\mathcal{E}}_t\right|}\sum _{\alpha ∊{\mathcal{E}}_t}\int f_{\delta }^{\alpha }(\stackrel{\ast }{\tau };t)log{M}_{\alpha }^j\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\right)\hfill \\ & =& {\kappa }_{1}exp\left(\int f_{\delta }^i(\stackrel{\ast }{\tau };t)log{M}_i^j\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\right)\hfill \\ & =& {\kappa }_{1}exp\left(log\frac{K{t}_{ij}^k}{{\delta }^{k+1}}-\frac{k{t}_{ij}}{\delta }-log{\kappa }_1\right)\hfill \\ & =& K\frac{t_{ij}^k}{{\delta }^{k+1}}{e}^{-k{t}_{ij}/\delta }\hfill \\ & =& M_i^j\left(\delta \right).\hfill \end{array}$$

Our choice of ${\kappa }_1={\left[k{e}^{-\psi \left(k\right)}\right]}^{k+1}$ allowed the equality $m^j\left(\delta \right)=M_i^j\left(\delta \right)$ to hold when event $i$ cues event $j$ with a fixed time interval and no event co-occurs with event $i$, without any additional constant of proportionality in the equation.

Remark.

The conclusion of lemma 3 does not hold in general if the delay interval is not fixed. Let $i$ and then $j$ occur at times 1 or 2 apart (with equal probability). Let $k=2$ (for ease of calculation) so $K=\frac{k^{k+1}}{k!}=4$. We have

$$\begin{array}{ccc}\hfill M_i^j\left(\stackrel{\ast }{\tau }\right)& =& \frac{\int {\tilde{f}}_{\delta =0}^i\left(\stackrel{\ast }{\tau };t\right)f_j\left(t\right)dt}{\int f_i\left(t\right)dt}\hfill \\ & =& \frac{1}{2}\left(K\frac{1}{{\stackrel{\ast }{\tau }}^{k+1}}\left(e^{-k/\stackrel{\ast }{\tau }}+2^k{e}^{-2k/\stackrel{\ast }{\tau }}\right)\right)\hfill \\ & =& \frac{2}{{\stackrel{\ast }{\tau }}^3}\left(e^{-2/\stackrel{\ast }{\tau }}+4e^{-4/\stackrel{\ast }{\tau }}\right)\hfill \\ \hfill M_i^j\left(\stackrel{\ast }{\tau }=1\right)& =& 2\left(e^{-2}+4e^{-4}\right)=0.417195,\hfill \end{array}$$

and as before,

$$f_{\delta }^i\left(\stackrel{\ast }{\tau }\right)=4\frac{{\delta }^2}{{\stackrel{\ast }{\tau }}^3}{e}^{-2\delta /\stackrel{\ast }{\tau }}.$$

But when $\delta =1$,

$$\begin{array}{ccc}\hfill \int f_{\delta }^i\left(\stackrel{\ast }{\tau }\right)log{M}_i^j\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }& =& 4{\delta }^2{\int }_0^{\infty }\frac{e^{-2\delta /\stackrel{\ast }{\tau }}}{{\stackrel{\ast }{\tau }}^3}log\left[\frac{2}{{\stackrel{\ast }{\tau }}^3}\left(e^{-2/\stackrel{\ast }{\tau }}+4e^{-4/\stackrel{\ast }{\tau }}\right)\right]d\stackrel{\ast }{\tau }\hfill \\ \hfill & =& -1.51366\hfill \\ \hfill m^j\left(\delta =1\right)& =& {\kappa }_{1}exp\left(-1.51366\right)\hfill \\ \hfill & =& 8exp\left[-3\left(1-\gamma \right)\right]exp\left(-1.51366\right)\hfill \\ \hfill & =& 0.495310.\hfill \end{array}$$

Since the two boxed values are different, the equality $m^j\left(\delta \right)=M_i^j\left(\delta \right)$ does not hold in general.

A.4. Credit Association and Prediction

The agent maintains credit associations $exp\mathbf{C}$ between each pair of event types, which estimates the multiplier for the agent's belief about the rate of each outcome whenever it sees a potential cue. The agent maintains predictions $\mathbf{p}$, a time line of future events based on the credit associations. As the agent experiences the world, $exp\mathbf{C}$ and $\mathbf{p}$ are iteratively updated.

This section motivates the form of the prediction $\mathbf{p}$ (see equations 2.2 and 2.3), by showing that the integral approximately subtracts the time elapsed since each cue, from the time delay between that cue and the outcome of interest. The integral thus produces a function of $\delta$ that peaks approximately at the time remaining to the outcome. This section consists of three subsections. First, we calculate the projected memory ${\tilde{\mathbf{f}}}_{\delta }$, which is an element in the prediction $\mathbf{p}$, in the case of perfect memory. Then we calculate the prediction $\mathbf{p}$ in the case of perfect memory. Finally, we discuss the case of fuzzy memory.

A.4.1. Perfect Memory: Projected Memory

We proposed the following form for the prediction:

$$p^{\beta }\left(\delta ;t\right)={\Lambda }^{\beta }exp\sum _{E∊\mathcal{E}}\int C_E^{\beta }\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^E(\stackrel{\ast }{\tau };t)d\stackrel{\ast }{\tau },$$

where ${\Lambda }^{\beta }$ denotes the long-term average of event type $\beta$, and $\mathcal{E}$ denotes the set of possible cue types. We motivate the above functional form by deriving the prediction in the case of perfect memory (i.e., $k\to \infty$) and discrete events. To do so, we would need to find the projected memory ${\tilde{f}}_{\delta }^E$ for event type $E$. Note (see equation 1.1) that

$$\underset{k\to \infty }{lim}{\Phi }_k\left(x\right)=\delta \left(x-1\right),$$

where $\delta \left(·\right)$ represents the Dirac delta function. Let the input function $\mathbf{f}$ be a series of discrete events of type $e_i$ occurring at times $t_i<t$,

$$f^E\left(\tau \right)=\sum _i\delta \left(t_i-\tau \right)I\left(e_i=E\right).$$

If we imagine that memory were perfect, then from equation 1.4, the projected memory would be represented as

$$\begin{array}{ccc}\hfill {\tilde{\varphi }}_{\delta }^E(\stackrel{\ast }{\tau };t)& =& \underset{k\to \infty }{lim}\frac{1}{\stackrel{\ast }{\tau }}{\int }_{-\infty }^t{f}^E\left(\tau \right){\Phi }_k\left(\frac{t+\delta -\tau }{\stackrel{\ast }{\tau }}\right)d\tau \hfill \\ & =& \frac{1}{\stackrel{\ast }{\tau }}{\int }_{-\infty }^t\sum _i\delta \left(t_i-\tau \right)I\left(e_i=E\right)\delta \left(\frac{t+\delta -\tau }{\stackrel{\ast }{\tau }}-1\right)d\tau \hfill \\ & =& \sum _{i}I\left(e_i=E\right){\int }_{-\infty }^t\delta \left(t_i-\tau \right)\frac{1}{\stackrel{\ast }{\tau }}\delta \left(\frac{t+\delta -\tau -\stackrel{\ast }{\tau }}{\stackrel{\ast }{\tau }}\right)d\tau \hfill \\ & =& \sum _{i}I\left(e_i=E\right){\int }_{-\infty }^t\delta \left(t_i-\tau \right)\delta \left(t+\delta -\tau -\stackrel{\ast }{\tau }\right)d\tau \hfill \\ & =& \sum _i\delta (t-t_i+\delta -\stackrel{\ast }{\tau })I(e_i=E),\hfill \end{array}$$

where ${\tilde{\varphi }}_{\delta }$ is the analog of ${\tilde{\mathbf{f}}}_{\delta }$ when memory is perfect, and we have used the property $\delta \left(\alpha x\right)=\delta \left(x\right)/\left|\alpha \right|$ of the Dirac delta function.

A.4.2. Perfect Memory: Prediction

We are now ready to derive the prediction:

$$\begin{array}{ccc}\hfill p^{\beta }\left(\delta ;t\right)& =& {\Lambda }^{\beta }exp\sum _{E∊\mathcal{E}}\int C_E^{\beta }\left(\stackrel{\ast }{\tau }\right){\tilde{\varphi }}_{\delta }^E(\stackrel{\ast }{\tau };t)d\stackrel{\ast }{\tau }\hfill \\ & =& {\Lambda }^{\beta }exp\sum _i\sum _{E∊\mathcal{E}}\int C_E^{\beta }\left(\stackrel{\ast }{\tau }\right)\delta (t-t_i+\delta -\stackrel{\ast }{\tau })I(e_i=E)d\stackrel{\ast }{\tau }.\hfill \end{array}$$

Consider the summand associated with $i=1$:

$$\sum _{E∊\mathcal{E}}\int C_E^{\beta }\left(\stackrel{\ast }{\tau }\right)\delta (t-t_1+\delta -\stackrel{\ast }{\tau })I(e_1=E)d\stackrel{\ast }{\tau }=\int C_{e_1}^{\beta }\left(\stackrel{\ast }{\tau }\right)\delta (t-t_1+\delta -\stackrel{\ast }{\tau })d\stackrel{\ast }{\tau }.$$

The above integral is a convolution of $C_{e_1}^{\beta }\left(\delta \right)$ with ${\delta }_{-\left(t-t_1\right)}\left(\delta \right)=\delta (t-t_1+\delta )$. The result is a translation of $C_{e_1}^{\beta }\left(\delta \right)$ by $-(t-t_1)$, the time interval since event $i=1$:

$$\int C_{e_1}^{\beta }\left(\stackrel{\ast }{\tau }\right)\delta (t-t_1+\delta -\stackrel{\ast }{\tau })d\stackrel{\ast }{\tau }=C_{e_1}^{\beta }(t-t_1+\delta ).$$

This makes sense because $C_{e_1}^{\beta }$ peaks at the time that $\beta$ is expected after having observed $e_1$ (say, $\tau$). Since $e_1$ occurred $t-t_1$ ago, the view of $C_{e_1}^{\beta }$ that matters for the prediction should be translated by $-(t-t_1)$. This view peaks at $\tau -(t-t_1)$, which is the time remaining until $\beta$ is expected on the basis of $e_1.$ (For example, if $\beta$ is expected five time units after $e_1$ ($C_{e_1}^{\beta }\left(\delta \right)$ peaks at $\delta =5$) and $e_1$ occurred three time units ago, then $\beta$ is expected in two time units ($C_{e_1}^{\beta }(t-t_1+\delta )=C_{e_1}^{\beta }(3+\delta )$ peaks at $\delta =2$).)

If there is only one event episode, indexed by $i=1$,

$$p^{\beta }\left(\delta ;t\right)={\Lambda }^{\beta }exp{C}_{e_1}^{\beta }(t-t_1+\delta ).$$

Thus, the prediction $p^{\beta }$ will also peak at $\delta =\tau -(t-t_1)$. (Continuing with the example, the prediction will also peak at $\delta =2$.) The form of the prediction makes sense. The quantity ${\Lambda }^{\beta }$ is the rate of occurrence of $\beta$ without any evidence or the agent's prior belief. The quantity $exp{C}_{e_1}^{\beta }$ is the adjustment factor for the rate of $\beta$ due to $e_1$. The product is the new, adjusted estimate for the rate of $\beta$.

If there are exactly two event types with one episode each,

$$p^{\beta }\left(\delta ;t\right)={\Lambda }^{\beta }\left[exp{C}_{e_1}^{\beta }(t-t_1+\delta )\right]\left[exp{C}_{e_2}^{\beta }(t-t_2+\delta )\right],$$

and so on. In other words, the adjustment factors for different event types multiply, as would be desired.

A.4.3. Fuzzy Memory

Previously, we have assumed that memory is perfect to illustrate the principles behind the form of the prediction $\mathbf{p}$. However, memory is fuzzy in real, resource-bounded systems, so the foregoing comments apply only approximately. The form of the integral in the prediction,

$$\int C_E^{\beta }\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^E(\stackrel{\ast }{\tau };t)d\stackrel{\ast }{\tau },$$

is not a convolution since the width of ${\tilde{f}}_{\delta }^E$ increases with $\delta$. However, the purpose of the integral is to approximate a convolution, so that the relevant view of $C_E^{\beta }$ can be used to generate a prediction. For example, if $E$ occurred three time units ago, ${\tilde{f}}_{\delta }^E\left(\stackrel{\ast }{\tau }\right)$ would peak at $\stackrel{\ast }{\tau }=\delta +3$ units (as $\delta$ increases, ${\tilde{f}}_{\delta }^E\left(\stackrel{\ast }{\tau }\right)$ becomes increasingly wider). If $E$ cues $\beta$ with a delay of five time units, $C_E^{\beta }\left(\stackrel{\ast }{\tau }\right)$ would peak at around five time units. The integral attains its largest value approximately when the peaks of ${\tilde{f}}_{\delta }^E\left(\stackrel{\ast }{\tau }\right)$ and $C_E^{\beta }\left(\stackrel{\ast }{\tau }\right)$ are aligned, which is around $\delta =2.$ This makes sense, because $\beta$ is expected in two time units from the present.

In summary, we have shown that the integral in equation 2.3 approximately subtracts the time elapsed since each cue, from the time delay between that cue and the outcome of interest. The integral thus produces a function of $\delta$ that peaks approximately at the time remaining to the outcome.

A.5. Worked Examples

In section 3.2, we noted that fuzzy memory induces fuzziness in the prediction $\mathbf{p}$ and that the latter could have equivalently been induced by intrinsic temporal uncertainty in the input in an agent with perfect memory, for every given snapshot in time. In section 3.4, we noted that other things equal, cues closer in time to outcomes tend to receive more credit for those outcomes in the case of fuzzy memory. We provided one example each for illustration. We provide the mathematical details for the example in section 3.2 and then for section 3.4.

A.5.1. Worked Example 1: Forward Conditioning

For this example, event $x$ cues event $y$ with a fixed delay of time interval $\tau$. We work out the agent's prediction for $y$ at the time $x$ occurs.

The pairwise association is given in equation 1.6,

$$M_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)={\Phi }_k(\tau /\stackrel{\ast }{\tau })/\stackrel{\ast }{\tau }.$$

By lemma 3, when $x$ occurs,

$$m^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)=M_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)={\Phi }_k(\tau /\stackrel{\ast }{\tau })/\stackrel{\ast }{\tau }.$$

When $x$ occurs, the memory is empty, so

$${\left(p^{-}\right)}^{\mathrm{Y}}={\Lambda }^{\mathrm{Y}}.$$

Thus, after training, the credit due to $x$ for $y$ is

$$exp{C}_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right)=\frac{p^{+}\left(\stackrel{\ast }{\tau }\right)}{p^{-}\left(\stackrel{\ast }{\tau }\right)}=\frac{{\Phi }_k(\tau /\stackrel{\ast }{\tau })}{{\Lambda }^{\mathrm{Y}}\stackrel{\ast }{\tau }}.$$

The projected memory for $x$ when $x$ occurs is

$${\tilde{f}}_{\delta }^{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)={\Phi }_k(\delta /\stackrel{\ast }{\tau })/\stackrel{\ast }{\tau }.$$

The prediction for $y$ at the time $x$ occurs is

$$\begin{array}{ccc}\hfill p^{\mathrm{Y}}\left(\delta \right)& =& {\Lambda }^{\mathrm{Y}}exp\sum _{E∊\mathcal{E}}\int C_E^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^E\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\hfill \\ & =& {\Lambda }^{\mathrm{Y}}exp\int C_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }.\hfill \end{array}$$ (A.9)

The integral is

$$\begin{array}{ccc}\hfill \int C_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }& =& \int \frac{{\Phi }_k(\delta /\stackrel{\ast }{\tau })}{\stackrel{\ast }{\tau }}log\frac{{\Phi }_k(\tau /\stackrel{\ast }{\tau })}{{\Lambda }^{\mathrm{Y}}\stackrel{\ast }{\tau }}d\stackrel{\ast }{\tau }\hfill \\ & =& {\kappa }_0{\delta }^k\int \frac{e^{-k\delta /\stackrel{\ast }{\tau }}}{{\stackrel{\ast }{\tau }}^{k+1}}log\frac{{\kappa }_0{\tau }^k{e}^{-k\tau /\stackrel{\ast }{\tau }}}{{\Lambda }^{\mathrm{Y}}{\stackrel{\ast }{\tau }}^{k+1}}d\stackrel{\ast }{\tau }.\hfill \end{array}$$ (A.10)

By lemma 2, we have

$${\int }_0^{\infty }\frac{e^{-a/x}}{x^{k+1}}log\frac{A{e}^{-b/x}}{x^m}dx=\frac{\left(k-1\right)!}{a^k}\left[m\psi \left(k\right)-\frac{kb}{a}+log\frac{A}{a^m}\right],$$

so we substitute $a=k\delta$, $A={\kappa }_0{\tau }^k/{\Lambda }^{\mathrm{Y}}$, $b=k\tau$, and $m=k+1$ to find

$${\int }_0^{\infty }\frac{e^{-k\delta /\stackrel{\ast }{\tau }}}{{\stackrel{\ast }{\tau }}^{k+1}}log\frac{{\kappa }_0{\tau }^k{e}^{-k\tau /\stackrel{\ast }{\tau }}}{{\Lambda }^{\mathrm{Y}}{\stackrel{\ast }{\tau }}^{k+1}}d\stackrel{\ast }{\tau }=\frac{\left(k-1\right)!}{{\left(k\delta \right)}^k}\left[\left(k+1\right)\psi \left(k\right)-\frac{k\tau }{\delta }+log\frac{{\kappa }_0{\tau }^k}{{\Lambda }^{\mathrm{Y}}{\left(k\delta \right)}^{k+1}}\right].$$

Therefore,

$$\begin{array}{ccc}\hfill \int C_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }& =& {\kappa }_0{\delta }^k\frac{\left(k-1\right)!}{{\left(k\delta \right)}^k}\left[\left(k+1\right)\psi \left(k\right)-\frac{k\tau }{\delta }+log\frac{{\tau }^k}{{\Lambda }^{\mathrm{Y}}k!{\delta }^{k+1}}\right]\hfill \\ & =& \left(k+1\right)\psi \left(k\right)-\frac{k\tau }{\delta }+log\frac{{\tau }^k}{{\Lambda }^{\mathrm{Y}}k!{\delta }^{k+1}},\hfill \end{array}$$

and the prediction for $y$ at the time $x$ occurs is

$$\begin{array}{ccc}\hfill p^{\mathrm{Y}}\left(\delta \right)& =& {\Lambda }^{\mathrm{Y}}exp\int C_{\mathrm{X}}^{\mathrm{Y}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)d\stackrel{\ast }{\tau }\hfill \\ & =& {\Lambda }^{\mathrm{Y}}exp\left[\left(k+1\right)\psi \left(k\right)-\frac{k\tau }{\delta }+log\frac{{\tau }^k}{{\Lambda }^{\mathrm{Y}}k!{\delta }^{k+1}}\right]\hfill \\ & =& {\Lambda }^{\mathrm{Y}}exp\left[\left(k+1\right)\psi \left(k\right)\right]\frac{{\tau }^k}{{\Lambda }^{\mathrm{Y}}k!{\delta }^{k+1}}{e}^{-k\tau /\delta }\hfill \\ & =& \frac{exp\left[\left(k+1\right)\psi \left(k\right)\right]}{k!}\frac{{\tau }^k}{{\delta }^{k+1}}{e}^{-k\tau /\delta }\hfill \\ & =& \frac{exp\left[\left(k+1\right)\psi \left(k\right)\right]}{k^{k+1}}\frac{1}{\delta }{\kappa }_0{\left(\frac{\tau }{\delta }\right)}^k{e}^{-k\tau /\delta }\hfill \\ & =& \frac{{\kappa }_1^{-1}}{\delta }{\Phi }_k\left(\frac{\tau }{\delta }\right).\hfill \end{array}$$

This equation is equation 3.1. section 3.2 makes use of this result.

A.5.2. Worked Example 2: Credit and Temporal Proximity

For this example, event $x$ cues event $y$ and event $z$ with a fixed delay of time interval $2-t_{\mathrm{YZ}}$ and 2, respectively (relative to the time of occurrence of $x$). We work out $exp{C}_{\mathrm{Y}}^{\mathrm{Z}}$ as a function of $t_{\mathrm{YZ}}$. The significance of this result is discussed in section 3.4.

We need ${\left(p^{+}\right)}_{\mathrm{Y}}^{\mathrm{Z}}$, which we write as shorthand for “the degree to which $z$ appears in ${\mathbf{p}}^{+}$ due to $y$” and ${\left(p^{-}\right)}^{\mathrm{Z}}$ at the time $y$ occurs,

$${\left(p^{-}\right)}^{\mathrm{Z}}={\Lambda }^{\mathrm{Z}}exp\sum _{E∊\mathcal{E}}\int C_E^{\mathrm{Z}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^E(\stackrel{\ast }{\tau };t_{\mathrm{Y}}^{-})d\stackrel{\ast }{\tau },$$

where $t_{\mathrm{Y}}^{-}$ refers to the moment just before $y$ occurs. At that time, the memory contains only $x$, which occurred $(2-t_{\mathrm{YZ}}$) ago, so

$${\left(p^{-}\right)}^{\mathrm{Z}}={\Lambda }^{\mathrm{Z}}exp\int C_{\mathrm{X}}^{\mathrm{Z}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^{\mathrm{X}}(\stackrel{\ast }{\tau };t_{\mathrm{Y}}^{-})d\stackrel{\ast }{\tau }.$$ (A.11)

The credit due to $x$ for $z$ is

$$exp{C}_{\mathrm{X}}^{\mathrm{Z}}\left(\delta \right)=\frac{{\Phi }_k(2/\stackrel{\ast }{\tau })}{{\Lambda }^{\mathrm{Z}}\stackrel{\ast }{\tau }},$$

and the projected memory for $\mathrm{x}$ when $y$ occurs is

$${\tilde{f}}_{\delta }^{\mathrm{X}}\left(\stackrel{\ast }{\tau }\right)={\Phi }_k\left((\delta +2-t_{\mathrm{YZ}})/\stackrel{\ast }{\tau }\right)/\stackrel{\ast }{\tau }.$$

Thus, the integral is

$$\int C_{\mathrm{X}}^{\mathrm{Z}}\left(\stackrel{\ast }{\tau }\right){\tilde{f}}_{\delta }^{\mathrm{X}}(\stackrel{\ast }{\tau };t_{\mathrm{Y}}^{-})d\stackrel{\ast }{\tau }=\int \frac{1}{\stackrel{\ast }{\tau }}{\Phi }_k\left(\frac{\delta +2-t_{\mathrm{YZ}}}{\stackrel{\ast }{\tau }}\right)log\frac{{\Phi }_k(2/\stackrel{\ast }{\tau })}{{\Lambda }^{\mathrm{Z}}\stackrel{\ast }{\tau }}d\stackrel{\ast }{\tau }.$$

This integral is the same as equation A.10 in the previous example with $\delta$, $\tau$ and ${\Lambda }^{\mathrm{Y}}$ replaced with $\delta +2-t_{\mathrm{YZ}}$, 2 and ${\Lambda }^{\mathrm{Z}}$, respectively. The same relationship holds between ${\left(p^{-}\right)}^{\mathrm{Z}}$ here (see equation A.11) and $p^{\mathrm{Y}}$ in the previous example (see equation A.9). Referring to the previous result, we thus have

$${\left(p^{-}\right)}^{\mathrm{Z}}\left(\delta \right)=\frac{{\kappa }_1^{-1}}{\delta +2-t_{\mathrm{YZ}}}{\Phi }_k\left(\frac{2}{\delta +2-t_{\mathrm{YZ}}}\right).$$

On the other hand,

$${\left(p^{+}\right)}_{\mathrm{Y}}^{\mathrm{Z}}\left(\delta \right)=m^{\mathrm{Z}}\left(\delta \right)=M_{\mathrm{Y}}^{\mathrm{Z}}\left(\delta \right)={\Phi }_k(t_{\mathrm{YZ}}/\delta )/\delta .$$

Thus, after training, the credit due to $y$ for $z$ is

$$\begin{array}{ccc}\hfill exp{C}_{\mathrm{Y}}^{\mathrm{Z}}\left(\delta \right)& =& \frac{{\left(p^{+}\right)}_{\mathrm{Y}}^{\mathrm{Z}}\left(\delta \right)}{{\left(p^{-}\right)}^{\mathrm{Z}}\left(\delta \right)}\hfill \\ \hfill & =& \frac{{\Phi }_k(t_{\mathrm{YZ}}/\delta )/\delta }{\frac{{\kappa }_1^{-1}}{\delta +2-t_{\mathrm{YZ}}}{\Phi }_k\left(\frac{2}{\delta +2-t_{\mathrm{YZ}}}\right)}\hfill \\ \hfill & =& \frac{{\left(t_{\mathrm{YZ}}/\delta \right)}^{k}exp\left(-k{t}_{\mathrm{YZ}}/\delta \right)/\delta }{\frac{{\kappa }_1^{-1}}{\delta +2-t_{\mathrm{YZ}}}{\left(\frac{2}{\delta +2-t_{\mathrm{YZ}}}\right)}^{k}exp\left(-\frac{2k}{\delta +2-t_{\mathrm{YZ}}}\right)}\hfill \\ \hfill & =& {\kappa }_1{\left(\frac{t_{\mathrm{YZ}}}{2}\right)}^k{\left(\frac{\delta +2-t_{\mathrm{YZ}}}{\delta }\right)}^{k+1}exp\left(-k\left[\frac{t_{\mathrm{YZ}}}{\delta }-\frac{2}{\delta +2-t_{\mathrm{YZ}}}\right]\right).\hfill \end{array}$$

Figure 8 plots this expression for various values of $t_{\mathrm{YZ}}$. See section 3.4 for a discussion.

A.6. Demonstration: Methods

Given a time-ordered set of events $[e_1,e_2,\dots ,e_n]$, where each $e_i=(x_i,t_i)$ comprises a discrete-valued type and real-valued timestamp, we are interested in predicting the type $x_{n+1}$ of the next event given its time of occurrence $t_{n+1}$. In the demonstration, we apply the prediction algorithm (“$\mathcal{C}$-based”) to a superposition of independent MRPs and compare its predictions to those of a pairwise event association model (“$\mathcal{M}$-based”). In the simulation, both the $\mathcal{C}$- and $\mathcal{M}$-based predictors have memories spanning ${10}^{-5}$ to 80 time units into the past, each covered by 200 log-spaced memory nodes.

Within each MRP, the probability of the type and time of an event depends solely on the type of the most recent past event, so for MRP $k$,

$$P\left((x_{n+1}^k,t_{n+1}^k)\right|{\left\{(x_m^k,t_m^k)\right\}}_{m\le n})=P\left((x_{n+1}^k,t_{n+1}^k)\right|x_n^k),$$

where $t_{n+1}^k>t_n^k$. The set of event types within each MRP is discrete and finite, while transition times $\Delta t_{n+1}=t_{n+1}-t_n>0$ are real and strictly positive; this allows only one event to occur at a given time. Within each MRP, the probability of the type of the next event is given by the transition matrix,

$$P_{ij}=P(x_{n+1}^k=j|x_n^k=i)=\left(\begin{array}{ccc}0.05& 0.75& 0.2\\ 0.2& 0.05& 0.75\\ 0.75& 0.2& 0.05\end{array}\right).$$

The transition times from $i$ to $j$ in MRP $k$ follow a truncated normal distribution $\mathcal{N}({\mu }_{ij}^k,{\sigma }_{ij}^{2k})$, with a lower-bound cutoff of ${10}^{-5}$ (to ensure positivity).

We use two approaches that generate superposed processes differently. We discuss the first approach, used for Figure 9c. The means ${\mu }_{ij}^k$ and variances ${\sigma }_{ij}^{2k}$ of the transition time distributions are drawn uniformly from the intervals (0, 10) and (0, 2), respectively. The same values are used across all six runs of the simulation. For each run, we generate exactly seven MRPs, labeled $k=1,\dots ,7$, each with 500 event episodes. We then construct seven superposed processes from the MRPs as follows. The first superposed process consists of one MRP, namely, the MRP $k=1$; the second superposed process consists of two MRPs, namely, the MRPs with $k=1$ and $k=2$; and so on. Each component MRP has three types of events, so the total number of event types in the superposed process is $3N$, where $N$ is the number of MRPs superposed.

We now discuss the second approach, used for Figure 9d. We draw exactly one set of transition time distribution parameters ${\mu }_{ij}$ and ${\sigma }_{ij}^2$ as before. This same set of parameters is used across all six runs of the simulation, and for all MRPs, $k=1,\dots ,7$. We generate exactly seven MRPs of 20,000 event episodes each and generate seven superposed processes therefrom by incrementally superposing the MRPs as in the first approach. Every MRP has three types of events (u, v, w). In the superposed processes, the event types are not distinguished according to the MRP of origin (e.g., a u from one MRP and a u from another MRP are both of type u in the superposed process). Thus, in contrast to the previous approach, the algorithms observe only three types of events in the superposed MRPs.

In both Figures 9c and 9d, 80% of each superposed process is used for training and the rest for testing. For the $\mathcal{C}$-based prediction, accuracy on the test set is computed by checking if, at every time $t_n$ that an event occurs, the prediction evaluated at $t_{n+1}$, the time of the next event, ${\text{argmax}}_i{p}^i(\delta =\Delta t_{n+1};t=t_n)$ matches the event that actually occurs at that time. For the $\mathcal{M}$-based prediction, the computation is analogous, except the prediction is found via ${\text{argmax}}_i{m}^i(\delta =\Delta t_{n+1};t=t_n)$, where $j=x_n$, the type of the event at $t_n$. The simulation is run six times and the average accuracy is reported.

Code Availability

The code that supports the demonstration in section 4 can be found at https://predicting.gitlab.io.