Honey bees selectively avoid difficult choices

Significance

Here we show that honey bees (Apis mellifera) can adaptively alter their behavior in a choice test in response to trial difficulty. Bees preferentially opt out of difficult trials and by doing so, improve their success rate. We discuss whether this choice involves assessing degree of uncertainty (considered a definition of basic metacognition) or whether this task might be solved by associative mechanisms. We propose a hypothesis for how uncertainty might be processed within the known circuitry of the insect brain to frame the concept of uncertainty as a topic for neurobiological analysis.

Abstract

Human decision-making strategies are strongly influenced by an awareness of certainty or uncertainty (a form of metacognition) to increase the chances of making a right choice. Humans seek more information and defer choosing when they realize they have insufficient information to make an accurate decision, but whether animals are aware of uncertainty is currently highly contentious. To explore this issue, we examined how honey bees (Apis mellifera) responded to a visual discrimination task that varied in difficulty between trials. Free-flying bees were rewarded for a correct choice, punished for an incorrect choice, or could avoid choosing by exiting the trial (opting out). Bees opted out more often on difficult trials, and opting out improved their proportion of successful trials. Bees could also transfer the concept of opting out to a novel task. Our data show that bees selectively avoid difficult tasks they lack the information to solve. This finding has been considered as evidence that nonhuman animals can assess the certainty of a predicted outcome, and bees’ performance was comparable to that of primates in a similar paradigm. We discuss whether these behavioral results prove bees react to uncertainty or whether associative mechanisms can explain such findings. To better frame metacognition as an issue for neurobiological investigation, we propose a neurobiological hypothesis of uncertainty monitoring based on the known circuitry of the honey bee brain.

Often a correct choice can only be estimated rather than absolutely known. To aid in this estimation, humans are able to monitor their degree of uncertainty and use that knowledge to improve their decisions (1, 2). The ability to monitor one’s own cognitive processes is considered a form of metacognition (1, 2). When uncertain, humans will often defer choosing and seek more information rather than risk the consequences of a wrong choice. Whether the ability to monitor uncertainty exists in nonhuman animals is currently highly contentious. Smith et al. (3, 4) developed the opt-out paradigm to test uncertainty monitoring in animals in which an animal must solve a discrimination task that varies in difficulty. The animal is rewarded when correct and punished for an incorrect choice. A third option is then introduced where the animal can “opt out” by responding in some different way to avoid the discrimination task, thereby usually beginning a new trial. If animals opt out more on difficult than easy tasks, if opting out improves performance on difficult tasks, and if they can apply the opt-out strategy to a novel task, then this has been taken as evidence that animals can modify their decision-making strategy based on their degree of uncertainty. This result has been reported for nonhuman primates, dolphins, dogs, and rats (3–12). However, some strongly argue that all comparative studies using opt-out paradigms can be explained through associative mechanisms that do not require judgments of uncertainty (3, 13–15).

Because two alternative mechanisms have been proposed to explain the same behavioral data, Morgan’s Canon (16) cautions that when presented with two alternative explanations, we are obliged to choose the simpler of the two. However, directly comparing these hypotheses is difficult because there is no agreed neural model for uncertainty processing. Therefore, it is challenging to judge which explanation is truly the simpler. Here we contribute to the debate by examining how an invertebrate, the honey bee, performed in the opt-out paradigm. We found that bees met all of the necessary criteria considered indicative of uncertainty monitoring. Bees have a far smaller brain than any mammal, and we hypothesize how uncertainty monitoring might be achieved by this animal.

Results

To assess how honey bees responded to choices based on limited information, we examined their performance in a visual discrimination task in which task difficulty was varied. We combined a configural-learning task, where bees had to learn stimuli placed above or below a referent (17), with the option to opt out of trials (5) to determine if bees adaptively changed behavior in response to task difficulty.

We trained free-flying bees to enter a two-chamber test apparatus (Fig. 1). During training, stimuli were placed either in the first decision chamber or the second. During testing, both chambers contained stimuli. The stimuli were two variable targets that were positioned either above or below a reference bar (cf. ref. 17) (Fig. 1B). One group of bees was trained that the above target contained 2 M sucrose (reward) and the below target contained 50 mM quinine (punishment) (18). For the second group, the contingency was reversed. During this stage, bees were trained with five different targets that differed in shape, size, and color. We changed targets and vertical position of targets and reference bars pseudorandomly to eliminate the possible use of associative cues. In other words, the shape, color, and size of targets changed from trial to trial and the distance of the targets from the bottom of the chamber and with respect to the reference bars varied from trial to trial. This format ensured that low-level cues, such as cumulated area (the area covered by both reference bar and target) or configural identity (the shape and orientation of both target and reference bar together), were not informative for solving the task (Fig. 1B) (17). Bees’ performance increased over six blocks of five trials (Fig. 2A) (ANOVA for repeated measures; n = 33 bees; block effect: F_5, 155 = 18.127, P < 0.001). There was no difference in performance between below- and above-trained groups (F_1, 31 = 0.422, P = 0.521). Data were therefore pooled and presented together from Fig. 2A onwards. All bees that performed at ≥ 80% correct on the final 10 trials of stage 1 (23 bees) were subsequently tested using a novel target for five additional trials to determine if they could generalize the task to a novel target (stage 2) (Fig. 1B). In this stage bees chose the correct target 80 ± 1.9% (mean ± SEM: n = 23 bees) of the time, indicating they had successfully learned the task (Fig. 2A). This result was significantly different from chance (assumed distribution of 50%) considering the first choice (binomial test: P < 0.001) or cumulative choices (one sample t test: t₂₁ = 20.120, P < 0.001), corroborating previous findings (17).

Fig. 1.

Experimental set-up and design. (A) Design of decision chambers. (B) Experimental design. Stage 1: The reference was the same on both left and right sides: a horizontal black bar 5-cm long and 2-cm high. Targets were the same on both sides but variable between trials. Five of the possible six targets were used during this stage. Reward (2 M sucrose) or punishment (quinine) was placed in a translucent microcentrifuge tube in the center of the targets (white trapezoid). Within a trial, stimulus pairs were identical except for vertical position relative to the reference bar and relative to the bottom of the chamber. Between trials, targets and positions of targets and reference bars were varied so that bees could only learn the above/below relation of targets to the references as predictors of reward. Approximately half of the bees were rewarded on the above relation and half on the below relation. Stage 2: The transfer tests were unrewarded and used a novel target not used in stage 1. Stage 3: Difficulty was varied by changing the distance between the target and the horizontal of the reference bar. For easy trials, the target was clearly above or below the reference bar and did not overlap with the reference bar. For hard trials, targets overlapped, with the reference bar. For impossible trials the center of both targets were in line with that of the reference bar. Stage 4: test of generalization of the opt-out rule. Bees were trained to learn new targets associated with reward and punishment. Posttraining bees were tested with confusing and unrewarded targets merging features of both learned targets.

Fig. 2.

Performance of bees. (A) Percentage of correct choices as a function of blocks of five trials (stage 1) and performance in nonrewarded transfer test (stage 2). (B) Performance on tests varying in difficulty (stage 3). Bees performed better on easy (83.8 ± 2.5%) than on hard (52.4 ± 2.5%) or impossible (44.5 ± 7%) trials. (C) Proportion opting out relative to easy (horizontal bars) and SE (vertical bars). Bees tended to opt out more on hard and impossible trials than on easy trials (stage 3). Bees also opted out significantly more on difficult trials (average of hard and impossible trials) than on easy trials. (D) Performance on unforced and forced trials (stage 3). Bees performed better on unforced hard trials than forced hard trials. Bees performed better overall on easy trials (ANOVA for repeated measures; n = 10; F_1, 9 = 6.028, P < 0.036), and performed similarly on forced and unforced easy trials (ANOVA for repeated measures; n = 10; F_1, 9 = 0.459, P = 0.515). This result is consistent with most bees being in a low-uncertainty state for the majority of easy trials. All bars in all panels represent mean values and all error bars represent SEM.

In stage 3 we tested bees for a further 50 trials that varied in difficulty (Fig. 1B). For easy trials, the target was clearly above or below the reference bar, and did not overlap with the reference bar (stage 3) (Fig. 1B). For hard trials, targets were offset, but overlapping, with the reference bar (stage 3) (Fig. 1B). For impossible trials (so-called because objectively they had no correct answer and therefore were rewarded pseudorandomly), the centers of both targets were in line with that of the reference bar (stage 3) (Fig. 1B). Bees performed better on easy trials than on hard or impossible trials (Fig. 2B) (ANOVA for repeated measures; n = 16; F_2, 26 = 19.94, P < 0.001).

Training and testing through stages A and B often took more than 3 h, because the duration of training was determined by the rate at which bees returned to the apparatus from their colony. Of the 23 bees that completed stages A and B, only 16 returned for stage 3 and only 10 completed all 50 trials of this stage, which usually took another 4 h or more to complete. During stage 3 targets were presented alternately in the first and second decision chamber (Fig. 1A). When presented in the first decision chamber bees had the option to opt out and fly through to the second chamber, where a different set of targets were present. When presented in the second chamber bees had no option to opt out and their choice was forced. We used χ² analyses to compare the rates of opt out on easy, hard, and impossible trials. As individuals, 3 of 10 bees opted out more in hard than easy trials (Fig. 3, row 6). Independent χ² values can be summed, and summing values for individual bees (19) (Fig. 3, row 6) showed that as a group these 10 bees opted out more often on difficult trials (hard and impossible) than on easy trials (additive χ²; χ² = 25.349, df = 10, P = 0.005). In addition, bees opted out more often on hard or impossible trials than on easy trials (Fig. 2C) (additive χ²; χ² = 31.268, df = 20, P = 0.052). Bees could potentially have solved this task either by learning to attend to the position of the target relative to the references (above or below), or by attending to the geometry of the references and targets as compound stimuli, but regardless of the specific strategy bees used to solve the task, as a group bees opted out more on difficult than easy trials.

Fig. 3.

Comparison of performances across tests for individual bees. Top row: bee identification number. Second row: Difference between probability of opting out on difficult trials (average of hard and impossible) minus probability of opting out on easy trials. Bees are ordered in increasing ranking of opt-out differential. Shading represents ranking for each category (row). lighter/whiter represents better performance. Third row: Performance difference between unforced trials and forced trials. Fourth row: Performance on transfer of opt-out concept first trial (+, opted out; −, made choice and did not opt out). Fifth row: cumulative performance on transfer of opt-out concept (five unrewarded trials). Sixth row: χ² values for probability of opting out in hard vs. easy trials. Seventh row: P values for each χ² test. Dashed line box: indicates best performances/measurements in each behavioral test. Dotted line box: indicates 7 of 10 bees that opted out more on difficult trials than easy trials.

Bees performed poorly on difficult tasks and opted out more (Fig. 2 B and C), but did opting out improve performance? To answer this question, we compared how often bees correctly solved the task in hard trials where bees could opt out (unforced trials) with hard trials where bees could not opt out (forced trials). Unforced trials were those in the first decision chamber because bees had access to the exit port and could therefore opt out. Forced trials were those in the second decision chamber because there was not an exit port to use (Fig. 1A). If bees were using the opt-out option adaptively and selectively, then they should perform better on unforced hard trials than forced. If, however, bees opted out more on difficult trials because they had simply learned to generally avoid this configuration irrespective of trial-specific certainty, then bees would be predicted to show the same level of performance in forced and unforced hard trials. Pooling data from all 10 bees, bees performed better on unforced hard trials than forced hard trials (ANOVA for repeated measures; n = 10; F_1, 9 = 6.028, P = 0.036) (Fig. 2D), indicating that bees opted out selectively when their likelihood of successfully solving the task was lowest.

To assess whether bees could transfer the concept of opting out to a novel difficult task, the 10 bees that completed stage 3 received training (similar to stage 1) for an additional 20 trials on a novel task (stage 4) in which they had to discriminate two targets (Fig. 1B). After 20 training trials, we tested these 10 bees with unrewarded “confusing” targets, which were a combination of both trained targets (Fig. 1B) (stage 4). We hypothesized that if bees were uncertain about the outcome of choosing these new targets, they would opt out of decisions that contained them. Of the 10 bees that completed stages C and D, no bees opted out during training, suggesting they did not select to opt out in response to a novel stimulus. However, 4 of 10 bees opted out when presented with the “confusing” targets on their first trial of test stage 4, and at least three of the five test trials with the confusing targets (Fig. 3).

Discussion

Although individual bees differed in performance, we have shown that some bees individually, and as a group, significantly altered their choice behavior in response to variation in task difficulty. Bees showed: (i) greater likelihood of opting out in hard compared with easy trials; (ii) better performance in unforced than forced hard trials; and (iii) transferred opting out to a new situation.

Performance differed greatly among bees tested (Fig. 3), but the individuals that opted out significantly more on difficult trials than easy trials also ranked highly on performance difference between unforced and forced trials and the generalization test (Fig. 3). This finding suggests the bees opting out most in the hard trials were those most capable of solving the hard task, rather than bees that could not discriminate the hard trials and avoided them.

Variation in performance between bees has been documented in every prior study of bee cognition, including rule learning (17, 20), contextual learning (21), categorization (22), and counting (23, 24). Indeed, animal cognition studies in general report great heterogeneity in performances between subjects (4–9, 11, 25). Therefore, it is little surprise that only some of the bees exposed to our arduous testing regime individually showed adaptive use of the opt-out rule in hard trials. These interindividual differences could involve variation in motivation, cognitive ability, choice strategy, or any combination of these. The proportion of bees able to generalize the opt-out rule to a novel task in our study was similar to that reported for primates (5).

Our data suggest some bees appeared capable of adapting their decision strategy in response to variation in task difficulty by adaptively using the option to opt out. Previously it has been shown that bees responded to variation in task difficulty or task risk by slowing down to improve accuracy (18, 26), and in an ethological setting bees responded to a cryptic predation risk by shifting their foraging strategy, perhaps akin to opting out of difficult discriminations in our assay (27). These findings raise the question of how bees are doing this. Adaptive use of an opt-out option in hard trials has been interpreted as evidence that animals are able to react to their degree of uncertainty (4). However, it has been argued that behavioral results like this can be explained by simple associative mechanisms that do not require uncertainty monitoring. Le Pelley (14) proposes that by an associative learning mechanism, conditioned response strengths (both positive and negative) will be associated with different learned stimuli in the discrimination task. The perceptual system is assumed to be noisy so that no stimulus is expected to be recognized perfectly, and learned responses can be generalized to similar stimuli (28). Through training, difficult stimuli will become associated with a high likelihood of punishment because of the high error rate in these trials. Although not itself rewarding, the opt-out response still has a conditioned response strength because it provides a way to avoid punishment. These assumptions alone are sufficient to predict that animals will opt out more for hard trials than easy in certain paradigms, and that performance should be better in unforced than forced hard trials (14, 15).

Two different mechanisms have therefore been proposed to explain the same behavioral phenomenon, and we are obliged to reject the more complex. However, objectively assessing which mechanism is most complex is presently difficult because, although the neurobiology of associative learning is well understood in bees, we have no neurobiological model of how uncertainty monitoring (if it occurs) might operate. Kepecs et al. (29) argue that uncertainty estimation is no more complex than simply recognizing and responding to a stimulus. If neural computations of the match between a new and a learned stimulus operate according to Bayesian theory, then a certainty estimate of the match would be intrinsic to that calculation. In rats, single neurons in the orbitofrontal cortex show firing patterns that correlate with degree of certainty modeled in this way (29).

The circuitry of learning in bees is much simpler than that of any mammalian system. Because of this finding, honey bees could prove to be a valuable system for determining conclusively whether simple animal brains can assess degree of certainty. In honey bees, associative learning occurs in a circuit involving the 170,000 Kenyon cells of the mushroom bodies and the extrinsic neurons, which output from the mushroom bodies to premotor regions (30–32). Stimuli are coded as a sparse cross-fiber pattern within the Kenyon cell population. Kenyon cells and extrinsic neurons are linked by a convergent connection matrix that is plastic and modifiable by experience, according to Hebbian processes (33, 34). Learning involves changing synaptic weights between the Kenyon cells activated by the learned stimulus and extrinsic neurons triggering the learned response (34, 35). In a differential conditioning paradigm in which a bee is trained that one stimulus is rewarded and one not, during learning the sensory representations of the two stimuli diverge (36, 37). Each stimulus activates different motor responses, and divergent sets of extrinsic neurons (34).

An ambiguous stimulus between the two learned stimuli would potentially coactivate both possible motor responses. It has been proposed that resolution of this conflict is likely achieved by mutual inhibition between neurons in the extrinsic neuron population, so that only a single dominant motor pattern is activated (33–35). If this assumed neurocomputational framework is correct, then the degree of match between a stimulus and a learned stimulus-response contingency is an intrinsic part of the associative mechanism. Certainty would correlate with the extent of coactivation of incompatible motor responses by the activated extrinsic neurons, the degree to which those competing possible outputs could be resolved by mutual inhibition, and the strength of the consensus motor output. In theory this correlation could operate as a neural signal of degree of uncertainty that requires minimal circuit elements over and above those required for associative learning. We do not know what elements in the bee brain might then allow an uncertainty estimate to influence likelihood of choice or opting out. In vertebrates, the neuromodulators acetylcholine and noradrenaline are involved in modulating choice by uncertainty (38, 39). Therefore, we propose that experimental testing of whether or not uncertainty estimation guides bee behavior should focus on computation within the association matrix between the Kenyon cell and extrinsic neuron populations of the bee brain, and the roles of neuromodulators in this system.

In summary, we have shown that honey bees are able to selectively avoid making choices when information is limited, and by doing so they improve their success in a choice test. This finding shows that even simple invertebrates are capable of making complex and adaptive decisions, and are sensitive to variation in task difficulty. Such behavior could be explained by direct assessment of uncertainty, or by associative mechanisms that weight the outcomes of hard and easy trials differently. However, the neural computation of uncertainty may be no more complex than that needed for classification and association. Determining whether simple invertebrates monitor uncertainty in decision making—or simply appear to do so—can only be addressed by electrophysiological testing of a neurobiological model for uncertainty monitoring. However, whichever computational strategy is used, our data suggest that a capacity to respond adaptively to difficult choices may be more general than has been previously thought.

Materials and Methods

Experiments were performed with bees maintained at Macquarie University (Sydney) between December 2012 and April 2013 (the Southern Hemisphere summer). Bees were of the standard commercial stock available to Australia (mostly a Ligustica background) containing a naturally mated queen. Our experiments used forager bees from colonies within an open-space bee yard, each containing ∼30,000 bees. Forager bees were attracted to the testing apparatus with a sucrose solution. Individual bees were marked on their thorax with paint and trained from their hive to fly to the decision chambers located several meters away. Cylindrical open-topped decision chambers were illuminated by sunlight and covered with insect screen through which an observer recorded decisions. The entrance was a 4-cm hole and the inner dimensions were 20 cm × 25 cm. The inside of each cylinder provided a white, UV-reflecting background for targets.

The training targets in stages A, B, and C (Fig. 1B) were a 4 × 4-cm vertical grating and a 4 × 4-cm radial three-sectored pattern that were achromatic “black,” a 4 × 4-cm cyan-magenta-yellow-black (CMYK) yellow cross, 4-cm-diameter CMYK orange circle, a 5 × 2.5-cm CMYK violet diamond, and a 3 × 5-cm CMYK magenta rectangle. All targets were printed with a high-resolution laser printer. Targets, feeding tubes, and quinine and sugar solutions were all replaced between each learning trial.

Training Protocol.

A differential conditioning protocol was used in training. Paired targets were placed relative to a 2 × 4-cm black rectangular reference bar. The target placed above the reference bar offered 50 µL 2 M sucrose solution. The target below the reference bar offered 50 µL 50 mM quinine hydrochloride dihydrate solution (which is aversive to bees). A separate group of bees were trained with the reverse contingencies. Reward and punishment were offered in a cut transparent microcentrifuge tube attached to the targets. In each trial the first choice of a bee was recorded.

Bees were first trained to the first and second decision chambers using one of the targets above for two trials in each where they received two correct configurations with reward. This process was to ensure that the bees knew that the targets could be located within the second chamber. Bees were prevented from re-entering the first chamber after they entered the second. In stage 1 of the experiment (Fig. 1B), if a bee chose the correct target (e.g., above the reference bar), they were allowed to drink ad libitum 2 M sucrose. If incorrect (e.g., below the reference bar), as soon as they were observed tasting the quinine solution with their proboscis an air puff was applied until the bee released from the target. Trials alternated between the first and second chamber of the apparatus in order for bees to learn that flying through the first chamber led into a second decision chamber (Fig. 1A).

Generalization Test.

During generalization tests (stage 4) (Fig. 1B), bees were trained with two targets one rewarded (2 M Sucrose) and one punished (50 mM quinine + air puff). Targets for the generalization test were each three tear-drop diamonds arranged radially and were colored HSB (hue, saturation, brightness: 242, 38, 67) and HSB (54, 56, 89). Trials alternated between the first and second decision chamber to ensure the bees were aware that these targets could be located within either chamber. A choice was recorded when a bee either landed on a target or flew through the exit port (opting out). At either point, the bee was released to begin a new trial. This process was repeated five times per bee.