Recency, consistent learning, and Nash equilibrium

Abstract

We examine the long-term implication of two models of learning with recency bias: recursive weights and limited memory. We show that both models generate similar beliefs and that both have a weighted universal consistency property. Using the limited-memory model we produce learning procedures that both are weighted universally consistent and converge with probability one to strict Nash equilibrium.

There is substantial evidence from both the laboratory and the field that people display “recency bias,” meaning that they react more heavily to recent observations and experiences than they do to older ones and continue to do so even once they have accumulated lots of data (for example, ref. 1 for evidence of recency in the field, ref. 2 for that in the laboratory, and ref. 3 for a recent survey of recency bias in single-agent decision experiments). Recency bias has been incorporated into both belief-based and reinforcement-based models of learning in games, by adding a parameter that controls the speed of informational discounting as in, for example, refs. 4–7 or by supposing that individuals retain only a finite sample in their memory as in ref. 8.

Our first result, Theorem 1, shows that informational discounting and a particular class of finite-memory models are similar. In Theorem 2 we show that when recency is slight, if the agent chooses each period’s action as a smooth best response to his current beliefs, he will obtain at worst about as much utility as would be possible as if he knew the sequence of outcomes in advance and had to commit to a particular action. This property, called weighted universal consistency, is a variation on the notion of universal or Hannan consistency in games (9, 10).

Finally in Theorem 3 we construct weighted universally consistent finite-memory learning procedures that converge with probability one to strict Nash equilibrium. This result differs from other results on procedures that always converge to Nash equilibrium, both because it allows for persistent small recency bias and because these procedures can be “decentralized” in the sense that the details of each agent’s learning rule can be set independently of the others. Earlier studies such as refs. 11 and 12 give convergence results, but do not show that the procedures have universal consistency properties, whereas the calibration rules of ref. 13 require that all agents base their forecasts on a shared deterministic “public forecasting rule,” which unlike our method requires that players coordinate the starting point of their learning procedures.

The Model

We consider a one-person decision problem. Each period $t=1,2,\dots$ the player chooses an action a from a finite set of actions A and then observes an outcome $y\in Y$, a finite set. The utility from action a when the outcome is y is denoted by $u\left(a,y\right)$. The space of probability distributions over a (finite) set S is denoted by $\mathrm{\Delta }\left(S\right)$. Mixed actions are denoted by $\alpha \in \mathrm{\Delta }\left(A\right)$ and mixed outcomes by $\gamma \in \mathrm{\Delta }\left(Y\right)$. We write $u\left(\alpha ,\gamma \right)$ for the expected utility to mixed actions and mixed outcomes. A strategy for the player can depend only on the information available to him when he moves, namely the past values of his own play and the outcome. A history of play for the player is denoted by $h_t=\left(a_1,y_1,\dots ,a_t,y_t\right)$, with $h_0$ the null history and H the space of all histories of play. A (behavior) strategy for the player is a map $\sigma :H\to \mathrm{\Delta }\left(A\right)$, whereas an outcome function is a map $\rho :H\to \mathrm{\Delta }\left(Y\right)$. Each strategy–outcome function pair induces a stochastic process over action–outcome pairs, where given the history $h_{t-1}$ the conditional probability of $a_t,y_t$, is $\sigma \left(h_{t-1}\right)\left[a_t\right]\rho \left(h_{t-1}\right)\left[y_t\right]$. In other words, the player and nature must base their play only on the history of actions and outcomes. In some interpretations, the outcomes may be chosen by other players rather than by nature.

Notions of Recency

Belief-Based Strategies.

Because we explicitly construct Markovian belief-based learning rules that satisfy the long-term performance criterion of weighted universal consistency, we restrict from the outset to this class of rules. A Markov belief-based strategy consists of a prior belief ${\varphi }_0\in \mathrm{\Delta }\left(Y\right)$, a Markov kernel $P\left(\varphi |{\varphi }_{t-1},y_t\right)$ that specifies how beliefs are updated, and a map $\alpha \left({\varphi }_{t-1}\right)$ from beliefs at time $t-1$ to a mixed action at time t. One such map is the best-response map (although this is not single-valued, we can make an arbitrary choice in the case of indifference); we also consider various smooth approximations to the best-response map. For the moment we focus on modeling the evolution of beliefs. In doing so, it is convenient to define $f\left(y|y_t\right)$ to be equal to 1 if $y=y_t$ and 0 otherwise and $f_{\tau }\left(y|h_t\right)=f\left(y|y_{\tau }\right)$, the indicator function for whether the period-τ outcome was equal to y.

Recursive Weighting.

We are given a weight $0<\mu <1$, an initial condition ${\varphi }_0$, and the deterministic kernel $\varphi \left(y|h_t\right)=\mu f_t\left(y|h_t\right)+\left(1-\mu \right)\varphi \left(y|h_{t-1}\right)$, where $\varphi \left(y|h_0\right)\equiv {\varphi }_0\left(y\right)$. An alternative representation of this belief process is to solve the difference equation and define $\lambda =1/\left(1-\mu \right)$, so that

$$\begin{array}{c}\varphi \left(y|h_t\right)=\frac{{\displaystyle {\sum }_{\tau =1}^t{\lambda }^{\tau }{f}_{\tau }\left(y|h_t\right)}+{\displaystyle {\sum }_{\tau =-\infty }^0{\lambda }^{\tau }{\varphi }_0\left(y\right)}}{{\displaystyle {\sum }_{\tau =-\infty }^t{\lambda }^{\tau }}}\\ =\frac{f_t\left(y|h_t\right)+{\displaystyle {\sum }_{\tau =-\infty }^{t-1}{\left(1/\lambda \right)}^{t-\tau }\varphi \left(y|h_{t-1}\right)}}{{\displaystyle {\sum }_{\tau =-\infty }^t{\left(1/\lambda \right)}^{t-\tau }}}.\end{array}$$

This shows that beliefs are a weighted average of past observations and the prior. Note that $\lambda \to \infty$ corresponds to the case $\mu \to 1$, where only the most recent observation matters, whereas Bayesian updating with a Dirichlet prior corresponds to time-varying weights ${\mu }_t\to 0$. We now show how the recursive weighting process is related to several others.

Base Rate Neglect.

In the case where the agent believes that ρ is generated by independent and identically distributed (i.i.d.) sampling, let ${\pi }_t\in \mathrm{\Delta }\left(\mathrm{\Delta }\left(Y\right)\right)$ be beliefs over $\mathrm{\Delta }\left(y\right)$. Bodoh-Creed et al.* propose a model of base rate neglect with the updating rule ${\pi }_t\left(\varphi \right)=\varphi \left(y_t\right){\left[{\pi }_{t-1}\left(\varphi \right)\right]}^{\nu }/{\displaystyle \int \varphi \left(y_t\right){\left[{\pi }_{t-1}\left(\varphi \right)\right]}^{\nu }d\varphi }$. Note that if $\nu =1$, this is ordinary Bayesian updating. If the prior is Dirichlet, the posterior mean is simply the maximum of this function with respect to ϕ. For $\nu \ne 1$ the posterior mean is difficult to compute, so we continue to measure the central tendency by taking the maximum of the function, that is, the posterior mode rather than the mean. To consider the maximum of this function with respect to ϕ, we can ignore the denominator and maximize the logarithm $\log \varphi \left(y_t\right)+\nu \log \left[{\pi }_{t-1}\left(\varphi \right)\right]={\displaystyle {\sum }_{\tau =1}^t{\nu }^{t-\tau }\log \varphi \left(y_{\tau }\right)+{\nu }^t\log {\pi }_0\left(\varphi \right)}$. If we assume that the prior is such that $\log {\pi }_0\left(\varphi \right)={\displaystyle {\sum }_{\tau =-\infty }^0{\nu }^{-\tau }\log \varphi \left(y_{\tau }\right)}$ for some weighted fictitious prior sample ${y_{\tau }|}_{\tau =-\infty }^0$, then we may write the log-likelihood with prior as $\log \varphi \left(y_t\right)+\nu \log \left[{\pi }_{t-1}\left(\varphi \right)\right]={\displaystyle {\sum }_{\tau =-\infty }^t{\nu }^{t-\tau }\log \varphi \left(y_{\tau }\right)}$ and the maximum likelihood is simply given by the weighted sample averages $\widehat{\varphi }\left(y|h_t\right)={\displaystyle {\sum }_{\tau =-\infty }^t{\nu }^{-\tau }{f}_{\tau }\left(y|h_t\right)}/{\displaystyle {\sum }_{\tau =-\infty }^t{\nu }^{-\tau }}$ so that if we take $\lambda ={\nu }^{-1}$ and define ${\varphi }_0\left(y\right)=\widehat{\varphi }\left(y|h_0\right)$, this is exactly the same point belief as generated by recency generated by weighted sampling.

Limited Memory.

So far we have supposed in effect that there is unlimited memory for past observations or at least that any value of ϕ can be recorded in the memory. We now instead suppose that the memory has size M, where M refers to the number of observations that can be stored. (Note that this is different from the limited-history processes considered by ref. 8, where the memory always contains the M most recent observations.) To do this we define a k, p, M procedure where $0<p\le 1,1\le k\le M$ as follows: (i) Choose randomly a subset of M of size k, (ii) discard each observation in the subset independently and randomly with probability p, and (iii) replace all of the discarded observations with the observation from the current period.

The simplest version of this procedure has $k=1,p=1$; that is, choose one observation at random from memory and discard it. In this case when the signal y is i.i.d., the ergodic distribution is multinomial.

The k, p procedure allows us to largely separate memory size M from λ, while allowing the construction of procedures with arbitrary values of λ. Our goal is to show that when M is large, the agent receives about the same utility as he would with infinite memory.

Recursive Weighting vs. Limited Memory

The recursive memory weighting model has a deterministic transition kernel and an infinite state space. The limited-memory model has random transitions and a finite state space. The latter has some advantages in analyzing properties such as universal consistency. In the case where ρ generates i.i.d. values of y, the stationary distribution of the recursive weighting model can be extremely complicated and need not have a density even when y is binary. The stationary distribution of the limited-memory model always exists and in some cases is quite simple: In the case in which $k=1,p=1$ each observation is drawn from exactly the same distribution $\varphi \text{*}$, so the belief each period is simply a multinomial with M observations drawn from $\varphi \text{*}$. (The same is true in the limited-history model of ref. 8, where the agent always forgets the oldest observation in the memory. However, that model does not fit our framework as it requires the state to keep track of the order in which the observations were acquired.)

We now want to relate the two models. Suppose that $\mu =pk/M$, so that the expected weight that the limited-memory model gives to the most recent observation is the same as in the recursive model, and initialize the two systems so that both begin with the same prior. Fix any sequence of observations $y_t$ and consider the deterministic sequence ${\varphi }_t$ from recursive weighting and consider the random process ${\tilde{\varphi }}_t$ from limited memory. Then we have the following:

Theorem 1.

For any fixed $\mu \in \left(0,1\right)$, as $M\to \infty$ then $E\left[\left|{\tilde{\varphi }}_t\left(h_t\right)-{\varphi }_t\left(h_t\right)\right|\right]\to 0$ uniformly in t and the sequence of observations $\left(h_1,h_2,\dots \right)$.

Remark:

The idea of the proof is that when M is large and $\mu =pk/M$, both the average in the limited memory M and the weighted average are close to the full sample average in terms of mean square distance.

Proof:

Fix the sequence $f_t$, define $z_t=\left({\tilde{\varphi }}_t-\left(1-\mu \right){\tilde{\varphi }}_{t-1}\right)/\mu -f_t$, and observe that ${\tilde{\varphi }}_t-{\varphi }_t={\displaystyle {\sum }_{\tau =1}^t{\lambda }^t{z}_t/{\mathrm{\Lambda }}_t}$, where ${\mathrm{\Lambda }}_t={\displaystyle {\sum }_{\tau =1}^t{\lambda }^{\tau }}$. Hence to prove Theorem 1 it is sufficient to prove that $E\left[\left|z_t\right|\right]\to 0$.

Now let $\ell$ be the number of observations in updating to period t that are discarded and let ${\tilde{\varphi }}_{t-1}^{\ell }$ be the frequencies in the remaining sample. Note that we can think of this as drawing $M-\ell$ observations from ${\tilde{\varphi }}_{t-1}$ without replacement, and we arbitrarily define ${\tilde{\varphi }}_{t-1}^M={\tilde{\varphi }}_{t-1}$. Because $\mu <1$, it is less than some $\overline{\mu }<1$. Simple algebra detailed in SI Appendix shows that

$$\begin{array}{l}E\left[\left|z_t\right||{\tilde{\varphi }}_{t-1},\ell \right]\\ \le 2\left|\frac{\ell }{\mu M}-1\right|+\frac{1-\mu }{\mu }\left(Pr\left(\ell >\left(\frac{\mu }{\sqrt{\overline{\mu }}}\right)M\right)+\underset{\ell \le \left(\mu /\sqrt{\overline{\mu }}\right)M}{\max }E\left[\left|{\tilde{\varphi }}_{t-1}-{\tilde{\varphi }}_{t-1}^{\ell }\right||{\tilde{\varphi }}_{t-1},\ell \right]\right),\end{array}$$ [1]

where the inequality follows from the facts that each of ${\tilde{\varphi }}_{t-1}^{\ell }$, $f_t$, and the difference between ${\tilde{\varphi }}_{t-1}^{\ell }$ and ${\tilde{\varphi }}_{t-1}$ is between 0 and 1.

Next we observe that $E\left|X\right|\le 2\sqrt{E{\left|X\right|}^2}$ and hence it is enough to prove that each of the expectations on the right-hand side of [1] has square deviation that goes to zero. Examining first $E{\left|\ell /\mu M-1\right|}^2$, recall that $\mu =pk/M$ and $E\ell =pk$, so we need only compute the variance of $\ell /\mu M$. The variance of $\ell$ is $p\left(1-p\right)k=\mu M\left(1-p\right)$, so the variance of $\ell /\mu M$ is $\mu M\left(1-p\right)/{\mu }^2{M}^2\le 1/\mu M$.

Turning to the second term $E\left[{\left|{\tilde{\varphi }}_{t-1}-{\tilde{\varphi }}_{t-1}^{\ell }\right|}^2|{\tilde{\varphi }}_{t-1},\ell \right]$, observe that $E\left[{\tilde{\varphi }}_{t-1}^{\ell }|{\tilde{\varphi }}_{t-1},\ell \right]={\tilde{\varphi }}_{t-1}$ and that the variance is bounded above by sampling with replacement, which is at worst $1/\left(M-\left(\mu /\sqrt{\overline{\mu }}\right)M\right)\le 1/\left(M\left(1-\sqrt{\overline{\mu }}\right)\right)$. Hence ${\max }_{\ell \le L}E\left[\left|{\tilde{\varphi }}_{t-1}-{\tilde{\varphi }}_{t-1}^{\ell }\right||{\tilde{\varphi }}_{t-1},\ell \right]\le 2\sqrt{1/\left(M\left(1-\sqrt{\overline{\mu }}\right)\right)}\to 0$.□

Approximate Universal Consistency of Slightly Weighted Sampling

Ref. 9 showed that when $\lambda =1$ (no recency at all), any smooth fictitious play is $\epsilon -$ universally consistent, meaning that regardless of the probability law of the $y_t$, the agent does at most ε worse in terms of time-average payoff then if he knew the empirical distribution of play ahead of time. We show that this extends to recency, in the sense that recency adds an additional error term to the lower bound on payoffs that vanishes as recency goes to 0. We continue to let ${\varphi }_t$ denote the beliefs of the weighted sampling scheme and let ${\gamma }_t$ denote the weighted beliefs through and including observations at time t excluding the prior.

Fix a scale parameter $\overline{U}>0$ and a function ν on probability distributions over actions that maps the interior of the simplex to the real numbers, is bounded by $\overline{U}$, is smooth and strictly differentiably concave, and satisfies the boundary condition that as γ approaches the boundary of the simplex the norm of the derivative becomes infinite. Now for $0<\zeta \le 1$ we define $v\left(\alpha ,\gamma \right)=u\left(\alpha ,\gamma \right)+\zeta \nu \left(\alpha \right)$. This perturbation of the utility function serves to induce mixing and allows the approximation of the best response function by the smoothed best response $\widehat{\alpha }\left(\gamma \right)=\arg {\max }_{\alpha }v\left(\alpha ,\gamma \right)$. The function $v\left(\widehat{\alpha }\left(\gamma \right),\gamma \right)$ is Lipschitz—shown in ref. 14—and from the implicit function theorem the Lipschitz constant has the form $B\overline{U}/\zeta$, where B depends only on ν.

For an arbitrary decision rule σ let $u_t\left(\sigma \right)={\displaystyle {\sum }_{\tau =1}^t{\lambda }^{\tau }u\left(\sigma \left(h_{\tau }\right),f_{\tau }\right)}$ be the total weighted expected utility received through period t, where $f_t$ is the distribution that places weight one on $y_t$. The notion of universal consistency can be motivated by considering what a procedure that is based only on frequencies can hope to achieve: Obviously if there are deterministic cycles, no procedure based on frequencies can hope to do as well as a procedure that is designed to recognize cycles. To reflect the fact that these procedures are expected to do well only against frequencies, let us assume that the entire sample is known, but restrict the strategy space to playing the same action each period—that is, not reacting to cycles. In this case we could be maximizing ${\displaystyle {\sum }_{\tau =1}^t{\lambda }^{\tau }u\left(\alpha ,f_{\tau }\right)}$, and because $u\left(a,\gamma \right)$ is linear in γ, this is the same as maximizing $u\left(\alpha ,{\gamma }_t\right)$. Recall that ${\mathrm{\Lambda }}_t={\displaystyle {\sum }_{\tau =1}^t{\lambda }^{\tau }}$, let $U\left(\varphi \right)={\mathrm{\Lambda }}_t{\max }_{a}u\left(a,\varphi \right)$ be the maximized value, and set $c_t={\mathrm{\Lambda }}_{t}U\left({\varphi }_t\right)-u_t$. Then $c_t/{\mathrm{\Lambda }}_t$ measures how much worse the player did when he used $\widehat{\alpha }\left({\varphi }_t\right)$ relative to how well he might have done had he known the sample in advance (but been constrained to time invariant strategies).

Definition:

A learning rule is ε-universally consistent with respect to λ if $\lim {\sup }_{t\to \infty }{c}_t/{\mathrm{\Lambda }}_t\le \mathit{ϵ}$ regardless of the probability law generating the ys.

We provide conditions on ζ and μ so that playing a smooth best response to the weighted beliefs is ε-universally consistent with respect to λ. To see why this terminology makes sense, note that in the case $\lambda =1$ this is the approximate universal consistency condition in the usual sense, as $c_t/{\mathrm{\Lambda }}_t$ reduces to $U\left({\varphi }_t\right)-u_t/t$. The condition $\lambda >1$ is stronger than $\lambda =1$ in that it places more weight on the next (unknown) observation than on past observations. Hence, conceptually the bigger λ is, the stronger the notion of universal consistency.

Theorem 2.

For any ν there exists a constant $B>0$ such that for all utility functions $\left|u\left(a,y\right)\right|\le \overline{U}$ the recursive memory model with parameters $\mu ,\zeta$ is $\mathit{ϵ}‐universally$ consistent with respect to λ, with $\mathit{ϵ}=7\overline{U}\left|\zeta +B\mu /\zeta \right|$.

The Proof, which adapts the method used in ref. 14 for the case $\lambda =1$, can be found in SI Appendix. In light of Theorem 1 this result extends immediately to the case of the finite-memory model with an additional error term that goes to zero as $M\to \infty$.

Convergence to Strict Nash Equilibrium

We study simultaneous move games with observable actions $w\left(a_1,a_2,\dots ,a_n\right)$. Say that a pure action profile is a δ-strict Nash equilibrium if each player loses at least δ from deviating to any pure action. Then we can show the following:

Theorem 3.

For any $\mathit{ϵ},\psi$, and $\overline{U}$ there exist recursive-memory learning procedures that are ε-universally consistent for each player such that if $\left|w\right|\le \overline{U}$ and the game w has a $\psi \overline{U}$-strict Nash equilibrium, then regardless of the initial samples of the individual players with probability one the learning procedures converge to some strict Nash equilibrium.

Remark:

The coordinated learning procedures of ref. 13 require all players to have the same initial sample and so do not satisfy the conclusion of the theorem.

Proof:

The idea is to start with a finite-memory universally consistent procedure where ε is very small and modify it so that when the sample shows that each other player assigns probability close to one to a pure strategy, and the smooth best response is to play a pure strategy with very high probability, play that pure strategy with probability one instead of assigning small positive probabilities to other strategies. The modified procedure is also universally consistent, albeit for slighter larger ε, but it allows play to converge as it gets “stuck” at strict Nash equilibria.

Set ${\mathit{ϵ}}_1=\mathit{ϵ}/4$ and ${\mathit{ϵ}}_2=\mathit{ϵ}/\left(4\overline{U}\right)$ (and also smaller than 1/2). Define $y^i=a^{-i}$. For each player choose a ${\nu }^i$ such that $u\left(a^i,{\gamma }^i\right)\ge u\left({\tilde{a}}^i,{\gamma }^i\right)$ implies ${\widehat{\alpha }}^i\left({\gamma }^i\right)\left[a^i\right]\ge {\widehat{\alpha }}^i\left({\gamma }^i\right)\left[{\tilde{a}}^i\right]$ (for example, the entropy function); note that these functions need not be the same for each player. Next choose ζ sufficiently small that two properties hold. First, $7\overline{U}\zeta \le {\mathit{ϵ}}_1$. Second, note that as $\zeta \to 0$ then $\widehat{\alpha }$ approaches the best response, so in particular the probability of a strict best response goes to 1. Hence we can also choose ζ small enough that if $a^i$ is any $\psi \overline{U}$-strict best response, then ${\widehat{\alpha }}^i\left({\gamma }^i\right)\left[a^i\right]\ge 1-{\mathit{ϵ}}_2$. Then choose μ such that $7\overline{U}B\mu /\zeta \le {\mathit{ϵ}}_1$. This is $2{\mathit{ϵ}}_1$ universally consistent by Theorem 2. By Theorem 1 we can choose $M^i$ large enough that $\overline{U}E\left[\left|{\tilde{\varphi }}_t^i-{\varphi }_t^i\right|\right]\le {\mathit{ϵ}}_1$ and suppose that $k^i=M^i$; that is, we potentially discard all observations. Then the procedure replacing ${\varphi }_t^i$ with ${\tilde{\varphi }}_t^i$ is $3{\mathit{ϵ}}_1$ universally consistent, because the payoffs remain within that distance. Now define a procedure ${\overline{\alpha }}^i\left(h_t^i\right)$ such that if all of the observations in the memory are identical and ${\widehat{\alpha }}^i\left({\gamma }^i\right)\left[a^i\right]\ge 1-{\mathit{ϵ}}_2$, then ${\overline{\alpha }}^i\left({\gamma }^i\right)\left[a^i\right]=1$ (we call this the stuck state); otherwise, ${\overline{\alpha }}^i\left({\gamma }^i\right)\left[a^i\right]={\widehat{\alpha }}^i\left({\gamma }^i\right)\left[a^i\right]$. This procedure is no worse than $3{\mathit{ϵ}}_1+\overline{U}{\mathit{ϵ}}_2=\mathit{ϵ}$ universally consistent.

Suppose that ${\alpha }^i,{\alpha }^{-i}$ is $\psi \overline{U}$ strict and that ${\gamma }^i$ contains only these observations for $i=1,2,\dots ,n$. Then we see from construction that such a state is absorbing.

Next suppose that all players are in the stuck state. Observe that all must play a strict best response, because the probability of a nonstrict best response is less than 1/2 and so less than ${\mathit{ϵ}}_2$ in the original procedure. If the best responses are identical to the samples, we are absorbed in a strict equilibrium. If not, then the sample must change for all but one player, and in particular the next period at least one player is not in the stuck state.

Now suppose that at least one player is not in the stuck state. Then that player plays an action different from his last period action with a positive probability bounded below with a bound that depends only on $\zeta ,\nu$, both of which are fixed, and with positive probability remains unstuck. Hence the next period there is a positive probability that all players are unstuck. When all players are unstuck, there is a positive probability that all play the $\psi \overline{U}$ strict equilibrium action, and there is a positive probability that all observations in all their samples are replaced with this action, resulting in the absorbing state.

□

Note that we do not assert that all weighted universally consistent learning procedures converge to strict Nash equilibria. Proposition 7 of Benam et al. (7) suggests that this is unlikely to be the case in games that have both a stable Shapley cycle and a strict Nash equilibrium.

What about mixed equilibria or, because mixed equilibrium will be difficult to hit with a finite number of states, mixed approximate equilibrium? Ref. 12 gives uncoupled learning algorithms that converge with probability one to a mixed approximate equilibrium, but these procedures are not universally consistent because once in equilibrium play never changes regardless of the data. By contrast, ref. 11’s procedure continues learning and the set of approximate Nash equilibria does have probability near one in the ergodic distribution; we do not know whether their procedure is universally consistent. At this point the issue of what sorts of extensions of Theorem 3 apply to games with only mixed equilibria remains open; we hope to explore it in future work.

Conclusion

We examine two models of learning with recency, recursive weights and limited memory. These are similar in the sense that they have the same mean beliefs and with high probability beliefs that are very close. We show that recursive weights with suitably smoothed best responses are weighted universally consistent and argue that this is a sensible criterion. It follows that limited memory has the same property provided the grid is fine enough. This is useful because it can produce limited-memory algorithms that are weighted universally consistent and also converge with probability one to strict Nash equilibrium. This is the first example of which we are aware of learning processes that are robust to initial conditions, have global convergence to Nash equilibrium, and are also shown to satisfy any sort of criteria of adequate responsiveness to individual incentives.