Genuine learning biases persist after accounting for temporally decreasing learning rates: Insight from fitting six datasets

In a recent paper, Godara (1) investigated the computational cognitive mechanisms underlying reinforcement learning in two-armed Bernoulli bandit tasks (TABB). The article makes two claims. The first claim is a mathematical demonstration that the Bayes-optimal solution to the TABB involves temporally decreasing learning rates, which can be shown to be functionally similar to a delta-rule with asymmetric updates. The second, empirical, claim is that previously reported learning biases may be statistical artifacts and that human behavior is better explained by models with temporally decreasing learning rates. While we have no quarrel with the first claim, we have substantial concerns regarding the second one.

To show why, we reanalyzed six datasets originally collected to investigate learning biases in TABBs, fitting three models (2):

(i) a model with asymmetric update (Bias model):

$$\begin{array}{c}Q{\left(s,a\right)}_{t+1}=Q{\left(s,a\right)}_t + \left\{\begin{array}{ll}{\alpha }_{+}\times P{E}_t,& \text{if} P{E}_t>0\\ {\alpha }_{-}\times P{E}_t,& \text{if} P{E}_t<0\end{array}\right.\hfill \end{array}$$

(ii) a model with a temporally decreasing learning rate as derived from Bayes-optimal principles, according to Godara (Bayes model):

$$Q{\left(s,a\right)}_{t+1}=Q{\left(s,a\right)}_t+\left(\frac{1}{t+3}\right)·{\mathit{PE}}_t$$

(iii) a model with a temporally decreasing learning rate, where the decrease is captured by an additional parameter (Decay model):

$$\alpha (s{)}_{t+1}=\alpha (s{)}_t·\delta$$

Second, we fitted and analyzed a model combining both features (Bias+Decay model) to assess the impact of including temporal decreasing learning rates on the estimation of the learning biases. All models were adapted to account for counterfactual update in complete feedback experiments, as previously described (2, 3).

Model comparison (using the average BIC, as in Godara) revealed that the Bias model systematically outperformed both the Bayes and the Decay models (Fig. 1). At the individual level, the Bias model better explained the majority of the participants, except in one experiment. This was also true in the dataset reanalyzed by Godara (“Palminteri et al., 2017c” in the figure).

Fig. 1.

Model comparison across experiments. In each scatter plot, each blue dot corresponds to one participant, and each pink cross indicates the mean ± SEM. The dependent variables are the BIC differences between the Bayes model and the Bias model (x-axis) and between the Decay model and the Bias model (y-axis). Positive values therefore indicate a better fit of the Bias model relative to the Bayes model (x-axis) and relative to the Decay model (y-axis). We selected the datasets to include all the experiments used in the previous meta-analysis on learning biases, ensuring that they all featured the same number of choices per pair of options. This was necessary to allow a meaningful comparison of the temporal decay of learning-rate curves (24 trials per pair of options). The formula for the BIC is $BIC=2·NLL+df·\mathrm{l}\mathrm{o}\mathrm{g}\left(n_{trials}\right)$. NLL: negative log-likelihood; df: number of free parameters (Bayes: 1; Decay and Bias: 3); n_trials: number of trials in the experiments [Lefebvre et al. (4), 96; the others (5, 6): 192].

We then turned to the analysis of the parameters of the full model, which provides further insight because it allows us to quantify i) the impact of introducing temporally decreasing learning rates on the positivity bias, and ii) the potential asymmetry in the decay parameters themselves (Fig. 2A). Concerning the first point, contrary to what hypothesized by Godara, the positivity bias remained robustly present in all studies. Concerning the second point, we uncovered a previously unreported asymmetry in the decay rates – specifically, the negative learning rate decayed faster than the positive one – further corroborating the idea that the learning bias persists across the whole learning curve (Fig. 2 B and C).

Fig. 2.

Parameters’ analysis. (A) Learning rates of the Bias+Decay model (blue and pink dots), and meta-analytical average of the difference in learning rate (gray and black dots). (B) Decay parameters of the Bias+Decay model (blue and pink dots), and meta-analytical average of the difference in decay parameters (gray and black dots). (C) Learning rate dynamics for the Bias+Decay model (thin lines represent individual studies, thick ones average across studies). Blue line: ${{\alpha }_{+}(s)}_{t+1}={{\alpha }_{+}(s)}_t\times {\delta }_{+}$; Pink lines: ${{\alpha }_{-}(s)}_{t+1}={{\alpha }_{-}(s)}_t\times {\delta }_{-}$). In (A) and (B): La and Lb correspond to the datasets originally published in ref. 4; Pp (partial) and Pc (complete) correspond to the datasets originally published in ref. 6; Sp (partial) and Sc (complete) correspond to the datasets originally published in ref. 5.

Our analyses indicate that temporally decreasing learning rates do not explain learning biases. Beyond the present work, this conclusion is further supported by the repeated observation of positivity bias in volatile TABB paradigms, where learning rates do not (and should not) decrease over time (7, 8).

In conclusion, despite repeated and rigorous scrutiny, the positivity bias has proven to be a remarkably robust feature of human (and animal) reinforcement learning (5, 9). We emphasize that while positivity bias exists in human reinforcement learning, it should not be necessarily regarded as a cognitive flaw. On the contrary, studies have shown that such asymmetry can confer computational advantages in several environments (10).