Behavioural variability contributes to over-staying in patchy foraging

Abstract

Foragers often systematically deviate from rate-maximizing choices in two ways: accuracy and precision. That is, they use suboptimal threshold values and also show variability in their application of those thresholds. We hypothesized that these biases are related and, more specifically, that foragers' widely known accuracy bias––over-staying––could be explained, at least in part, by their imprecision. To test this hypothesis, we analysed choices made by three rhesus macaques in a computerized patch foraging task. Confirming previously observed findings, we found high levels of variability. We then showed, through simulations, that this variability changed optimal thresholds, meaning that a forager aware of its own variability should increase its leaving threshold (i.e. over-stay) to increase performance. All subjects showed thresholds that were biased in the predicted direction. These results indicate that over-staying in patches may reflect, in part, an adaptation to behavioural variability.

1. Introduction

Many foragers roughly approximate rate-maximizing behaviour [1–5]. Two major apparent deviations stand out. First, individual foragers tend to exhibit much more behavioural variability than they should [6–10]. This is true even in carefully controlled computerized tasks [11–13]. A decision-maker that shows variability will harvest less reward than one that does not because it is often suboptimal on individual trials. Second, foragers tend to have systematically suboptimal average thresholds and, particularly in patch-leaving contexts, tend to over-stay [13–20].

Variability may be explained by several factors. Potential internal factors include deliberate exploration, sensory or motor variability and cognitive noise [21–25]. External factors may include a limitation in our measurement of foragers’ behaviour or our inability to accurately quantify the statistical properties of the environment [26–29]. It can be difficult to disambiguate these possibilities in field studies because much of the information that drives foraging decisions is difficult to measure [2].

The systematic error in thresholds is less readily explained [20]. Here we propose a new hypothesis, that foragers' use of suboptimal thresholds may be a rational response to their own variability, and may be less costly than it appears. Specifically, if foragers are intrinsically variable and foraging payoff curves are asymmetric (as they typically are), the optimal strategy will be to use a threshold biased in the direction of the shallower slope of the payoff curve. If the forager is both aware of its own variability and unable to reduce it, then the forager may potentially increase its harvest rate by over-staying. Given the wide variety of endogenous and exogenous constraints that foragers face, this adjustment may have evolved as a simple, versatile adaptation. As demonstrated by Houston & McNamara [30], behavioural variability can be incorporated into any optimal foraging strategy as a constraint itself rather than assumed to be a suboptimal deviation.

Here, we examined a dataset consisting of choices made by three foraging macaques in a computerized foraging task [14]. All subjects lived in controlled laboratory environments with stable food provisioning and were over-trained on their tasks for months to minimize subjective uncertainties about task structure. All subjects nonetheless showed high behavioural variability, consistent with observations from less controlled field studies; they also showed over-staying. These results suggest that subjects have the flexibility to adjust strategies to account for their own variability.

2. Material and methods

Three macaque subjects performed a patch-leaving task [13,14,31] and were trained on the task for several months prior to the experiment. All procedures were approved by the University Committee on Animal Resources at the University of Rochester and were designed and conducted in compliance with the Public Health Service's Guide for the Care and Use of Animals. Subjects had never previously been exposed to foraging decision-making tasks. Previous training history for these subjects did include several types of gambling tasks (specifically: Azab & Hayden [32,33], Heilbronner & Hayden [34], Farashahi et al. [35] and Blanchard et al. [36]) and, in one case, a cognitive set-shifting task [37]. All data and analysis code are available in the electronic supplementary material accompanying this article.

(a). Patch-leaving task

Stimuli were coloured rectangles on a computer screen. A rectangle's colour indicated reward available from that option. Its size indicated the delay associated with choosing it. On each trial, subjects fixated a point stimulus and were then presented with two targets representing the ‘stay’ in patch and ‘leave’ patch options. After 100 ms, subjects were free to select either target by shifting their gaze to it. The selected target then shrank at a constant rate. Its height therefore provided an unambiguous cue to the delays associated with the two options on every trial. Staying in the current patch always resulted in a 0.6 s handling time delay, while the travel time associated with leaving varied randomly from patch to patch between 0.5 and 10.5 s (figure 1).

Figure 1.

Task. After fixation, the subject views two targets and chooses one by shifting gaze to it. Choice of the orange rectangle (stay in patch) yields a short delay and reward whose value diminishes by 13 µl per trial. Choice of the grey rectangle (leave patch) yields no reward and a long delay (travel time) whose duration is indicated by the height of the bar, and resets the value of the orange rectangle at 230 µl. Patch travel time varies randomly from patch to patch between 0.5 to 10.5 s. ITI, inter-trial interval.

Stay choices resulted in water reward following the handling delay. This reward declined on each subsequent ‘stay’ decision and was reset upon arrival in a new patch. The reward amount for subject C was 200 μl initially, which was reduced in decrements of 13 μl. The reward amount for subjects E and O was 230 μl, which was decreased stochastically, in decrements of between 12.7 and 15.5 μl. Further description of the task is available in Hayden et al. [13].

(b). Patch-leaving data analysis

We aggregated 14 413 trials across 33 sessions from subject C, 2010 trials across four sessions from subject E, and 1717 trials across four sessions from subject O. In order to control for slight differences in travel times and handling times between the two datasets, we analysed the subjects' foraging behaviour in terms of the number of ‘stay’ decisions per patch. Mean patch residence time was defined as the mean number of trials spent in-patch across all patches. Reward rate was defined as the mean amount of water received per trial (per decision).

(c). Patch-leaving task simulation

To calculate the impact of noise on a subject's optimal threshold value, we constructed a simulation of the patch-leaving task as described above. Each simulated subject ran 10 000 behavioural sessions of 439 trials (the average number of trials per session over all three datasets). Patch-leaving thresholds ranged from 1 to the maximum number of rewarded trials (18 for subject C, 17 for subjects E and O), depending on initial reward and decrement size. We uniformly distributed the possible thresholds across the patches in each session. The expected reward rate for a given threshold (figure 2b) was calculated as the average reward rate from all patches, in all sessions where that threshold was in effect.

Figure 2.

Patch-leaving behaviour and simulation. (a) Patch length distributions for each subject denoted by number of trials (‘stay’ decisions). (b) The reward rate available via behavioural policies of leaving after a threshold number of trials, per simulation. (c) The optimal patch-leaving threshold as a function of a set amount of Gaussian noise applied to the threshold (quantified by the standard deviation σ).

The total accumulated reward in a given patch was based on the number of ‘stay’ decisions, initial reward and reward decrement amount. We therefore normalized the reward rate to the number of trials, rather than raw time, spent in-patch. The reward rate g was defined as

$$g=\frac{\text{reward}\hspace{0.17em}\text{accumulated}}{(t\times n_{\mathrm{trials}})+n_{\mathrm{trials}}}.$$

The travel time t was converted into a fraction of the total patch residence time (defined here as n_trials) for each possible patch length based on our behavioural dataset. For example, the travel times on patches in which subjects stayed for 10 trials were observed in our behavioural data to be 14.2% ± 9.4% of the total patch time. Thus for every simulated 10-trial patch, an ‘adjusted’ travel time was drawn from a normal distribution with µ = 0.142 and σ = 0.094, and multiplied by the length of the patch (now in trials) to get the normalized value.

To simulate the effect of threshold noise on its optimal value, we interpolated the reward curve 20-fold and performed trapezoidal numerical integration for a Gaussian-distributed matrix centred on each candidate threshold value. For standard deviations ranging from 0 to one-half of the maximum number of rewarded trials, we calculated the noise-adjusted threshold value that maximized the area under the reward rate curve.

3. Results

Three macaques (subjects C, E and O) performed a patch-leaving task (see Material and methods and Hayden et al. [13]). To control for slightly differing travel and handling time regimens between our three subjects, we defined patch residence time as the number of decisions to stay in a patch. All three subjects showed residence times that were close to, but longer than, the dictates of foraging theory [38], which yields an optimal patch residence time of 4.01 (±0.74) trials for subject C's version of the task and 4.16 (±0.78) trials with subject E and O's slightly different version of the task. In practice, all three subjects' mean patch residence times were significantly longer than optimal: 5.84 (±0.08) trials for subject C, 9.26 (±0.41) trials for subject E, and 12.09 (±0.42) trials for subject O (all significant, two-sided Student's t-test, p < 0.0001).

Subjects’ trial-to-trial behaviour in this task showed significant variability, which was costly. A non-variable forager with subject C's observed threshold would have obtained a reward rate of 168.4 µl per trial, yet the subject's actual reward rate was only 128.4 µl per trial. Similar costs also affected the reward rates of subject E (174.4 µl per trial without variability, 161.0 µl per trial observed) and subject O (147.1 µl per trial without variability, 135.1 µl per trial observed).

We calculated the trial-to-trial variability in each subject's patch-leaving behaviour as the standard deviation in its threshold over a large sample of bootstrapped 500-trial bins. Based on an empirical threshold standard deviation of 4.15 trials per patch, subject C's uncertainty-adjusted optimal patch residence threshold was 7.40 trials. In other words, optimizing given his variability can account for some of his observed rate adjustment (+1.84 trials) relative to optimal, assuming no variability (+3.40 trials). A similar pattern held for subjects E and O, with respective standard deviations of 6.02 trials per patch and 5.04 trials per patch both suggesting a compensatory increase in threshold. While their behavioural adjustments are in the same direction as suggested by accounting for uncertainty, however, the observed adjustments are actually larger than optimal (+4.26 trials per patch observed versus +3.05 predicted for subject E, and +7.09 trials observed versus +2.95 trials predicted for subject O). The difference between subjects' observed adjustments and the non-variable optimum adjustments is likely related to factors beyond behavioural variability.

4. Discussion

We examined previously collected datasets of rhesus macaques performing a computerized patch foraging task. [14] Macaques showed two behavioural patterns that are characteristic of many foragers. First, they showed a systematic deviation from an optimal foraging threshold by over-staying. Second, they showed a strong and costly variability in behaviour. We propose that these two phenomena are at least partly related. First, we conjecture that behavioural variability is for some reason unavoidable (and our results do not offer any explanation for it). Given this unavoidability, we showed that the normative reward-maximizing threshold increases. We then showed that all three subjects showed changes in the direction that improved harvest rates.

Previous studies of foraging behaviour have suffered two limitations that ours avoids. First, many studies are limited by data quantity. Using well-trained macaques performing a computerized task, we were able to analyse several thousand trials. Second, many studies have sources of unmeasurable noise. These include difficulty quantifying the details of the environment and difficulty knowing that the forager has had sufficient experience with the environment to make the same calculations we as observers would. The carefully controlled nature of our computerized task enables us to fix all relevant task variables and ensure over-training on those specific tasks. We were further able to reduce unmeasured variability by controlling the animals’ learning environment in their juvenile and adult lives and by ensuring a relatively stable food supply over that entire period of time.

It is perhaps surprising that subjects' behavioural variability on this task was so high despite the stable environment and months of preparatory task training. Our own study does not shed any light on the source of behavioural variability. A long tradition in foraging theory emphasizes the importance of ‘informational constraint’ on suboptimal behaviours [39–42]. Other factors may also be relevant, including information-seeking [43,44], temporal discounting [19], curiosity [45], computational noise, or exploratory behaviour [46–48]. One recent study showed that noise associated with evidence accumulation through a drift-diffusion process, a common theoretical underpinning of models of decision-making, could generate systematic suboptimal deviations in patch foraging strategies [49]. This suggests that the foragers may be unable to directly change their own variability, but flexible enough to adjust surrounding aspects of their behaviour.

Foraging is a major driver of the animal brain [50–54]. It also likely influences the structure of human choice systems [54–56]. As such, a science of the neural basis of choice ought to go hand in hand with an understanding of foraging psychology. Systematic deviations from optimal foraging provide an important measure of the forces of evolution; because they are costly, they likely reflect the existence of tradeoffs that are not obvious to evolutionary biologists. Many of these likely come from constraints on the costs of computing foraging strategies in full.

Ethics

All procedures were approved by the Animal Care and Use Committee (IACUC) at the University of Rochester (protocol no. 1706-34897A) and were designed and conducted in compliance with the Public Health Service's Guide for the Care and Use of Animals.

Data accessibility

Data and code (with readme file for replication) are available at https://figshare.com/collections/Overstaying_in_patchy_foraging_can_be_explained_by_behavioral_variability_Cash-Padgett_Hayden_2020_/4852557 as well as at https://www.haydenlab.com/data.

Authors' contributions

Both authors made substantial contributions to the conception and design of the study and analysis and interpretation of data. Both authors also participated in drafting and revising the article, gave final approval for it to be published and agree to be held accountable for the content contained therein.

Competing interests

The authors have no competing interests to declare.

Funding

This research was supported by a National Institute on Drug Abuse grant no. R01 DA038106 (to B.H.).