RISKY CHOICE IN PIGEONS: PREFERENCE FOR AMOUNT VARIABILITY USING A TOKEN-REINFORCEMENT SYSTEM

Abstract

Pigeons were given repeated choices between variable and fixed numbers of token reinforcers (stimulus lamps arrayed above the response keys), with each earned token exchangeable for food. The number of tokens provided by the fixed-amount option remained constant within blocks of sessions, but varied parametrically across phases, assuming values of 2, 4, 6, or 8 tokens per choice. The number of tokens provided by the variable-amount option varied between 0 and 12 tokens per choice, arranged according to an exponential or rectangular distribution. In general, the pigeons strongly preferred the variable option when the fixed option provided equal or greater numbers of tokens than the variable amount. Preference for the variable amount decreased only when the alternatives provided widely disparate amounts favoring the fixed amount. When tokens were removed from the experimental context, preference for the variable option was reduced or eliminated, suggesting that the token presentation played a key role in maintaining risk-prone choice patterns. Choice latencies varied inversely with preferences, suggesting that local analyses may provide useful ancillary measures of reinforcer value. Overall, the results indicate that systematic risk sensitivity can be attained with respect to reinforcer amount, and that tokens may be critical in the development of such preferences.

Risky choice is an area of research and interpretation concerned with how choices are affected by the variance associated with their outcomes. In a typical study, subjects choose between a certain (fixed) versus an uncertain (risky) outcome (e.g., $5 with certainty versus $0 or $10 with equal 50% probability, or 2 units of food with certainty versus 1 or 3 units with equal 50% probability). Preference for the certain alternative is termed risk aversion and preference for the uncertain alternative is termed risk prone behavior. The study of risky choice has developed into a multidisciplinary research domain, studied by psychologists, biologists, and economists, in both laboratory and field environments (Bateson & Kacelnik, 1998; Fantino & Romanowich, 2006; Kahneman & Tversky, 2000). In addition to its clear conceptual significance, risky choice has emerged in recent years as a promising laboratory model of gambling (Fantino, Navarro, & O'Daly, 2005; Potenza, 2009; Zentall, 2011). A better understanding of the basic mechanisms of risky choice is thus of theoretical as well as practical importance.

The extensive body of research on risky choice with nonhuman animal subjects and food reinforcers is usefully divided along the dimension in which risk is assessed: reinforcer delay or reinforcer amount (see Kacelnik & Bateson, 1996, for a review). When choosing between fixed and variable delays to reinforcer delivery (with equivalent response requirements), strong preference for the variable option is seen (e.g., Cicerone, 1976; Davison, 1969, 1972; Fantino, Arbaraca, & Ito, 1987; Herrnstein, 1964; Hursh & Fantino, 1973; Lagorio & Hackenberg, 2010; Mazur, 1984, 2004). Similar risk-prone behavior is demonstrated in choices between fixed- and variable-ratio schedules (e.g., Madden & Hartman, 2006; Mazur, 1986), an effect usually attributed to preference for variable food delays.

When choosing between fixed and variable amounts of reinforcement, however, results are more varied. Unlike the strong risk-prone behavior seen with reinforcer delays, the most common finding is risk aversion— preference for fixed over variable amounts of reinforcement (Bateson & Kacelnik, 1995; Behar, 1961; Clements, 1990; Real, 1981; Reboreda & Kacelnik, 1991). Some studies have also reported indifference between the choice alternatives (Behar, 1961; Mazur, 1989; Waddington, 1995; Wunderle & O'Brien, 1985), while others have reported risk-prone behavior (Barnard, Brown, Houston, & McNamara, 1985; Kendall, 1989; Zentall & Stagner, 2011). Some of the differences across studies are undoubtedly due to differences in the methods used in such a large and diverse literature that includes many different species and settings. However, even when confined to studies using pigeons as subjects under laboratory conditions, with food as reinforcers, and similar deprivation or energy budget conditions (i.e., subjects maintained at 80–85% of free-feeding body weights) the results are highly variable: Findings range from risk aversion (e.g., Hamm & Shettleworth, 1987; Menlove, Inden, & Madden, 1979) to indifference (e.g., Staddon & Innis, 1966) to strong risk-proneness (e.g., Essock & Reese, 1974; Young, 1981; Zentall & Stagner, 2011).

These mixed results have important conceptual implications. Showing that risky choice varies across different risk dimensions suggests that a single comprehensive model treating amount and delay similarly may be inadequate to account for the full range of effects. The mixed results are also problematic as far as laboratory models of gambling are concerned, as reinforcer amount is usually seen as the dimension most analogous to monetary gains and losses in human gambling. What is needed is a set of methods capable of producing consistent effects across the full range of risk-sensitive preferences—from risk-averse to risk-prone choices. Exploring risk sensitivity across a wide range of conditions would be useful not only in the development and testing of quantitative and theoretical models, but would also advance laboratory models of gambling. Rarely have laboratory studies focused on parts of the risk-sensitivity function generating negative utility. Human gambling is notable not only in showing risk sensitivity, but in risk sensitivity with suboptimal long-term consequences. Laboratory models of gambling must at minimum capture this important feature of human risk taking.

The present study was designed to more fully characterize risk-sensitive choices between fixed and variable amounts of food with pigeons. Risk sensitivity was assessed across a wider parametric range of conditions than employed in prior research. Relatively few studies have arranged contingences such that the fixed and variable amounts produce different average payoffs (Dunn & Spetch, 1990; Madden, Ewan, & Lagorio, 2007; Stagner & Zentall, 2010). Instead, most studies examine risky choice only when the fixed amount is equal on average to the variable amount. Although studies arranged in this manner provide important binary information as to which choice outcome is preferred (and, hence, whether subjects are risk prone or averse at that particular value), they do not allow for the assessment of risk sensitivity across a range of conditions, which permits the quantification of how much one alternative is preferred to another.

The present study also differed from prior studies in its use of a token reinforcement system. Pigeons chose between fixed and variable numbers of tokens (lights on the experimental panel) exchangeable for food at the end of each choice trial. Each token could be exchanged for 2.5-s access to the food hopper during regularly scheduled exchange periods. Food deliveries were arranged successively, with the number of deliveries correlated with the number of tokens on the panel. Such token-based procedures maintain good correspondence between programmed and obtained food deliveries (e.g., Foster & Hackenberg, 2004), where consumption varies as a linear function of reinforcer amount (i.e., hopper access time).

This correspondence between consumption and programmed reinforcement amount is important because the theoretical cost-benefit ratios of risky choice are usually computed assuming a linear relation between consumption and programmed reinforcer amount: raising the hopper for 8 s is assumed to provide twice the consumption of raising it for 4 s. Epstein (1981) found that the value function with some commonly used food hoppers may be nonlinear, however, with longer periods of access providing less consumption per unit time than shorter periods of access. If food consumption is a nonlinear function of hopper access time (particularly for programmed durations exceeding 6 s), then risk aversion may make good adaptive sense—choosing the smaller but certain reinforcer produces a higher unit value than probabilistic access to a larger reinforcer. At least some of the mixed results using pigeons and timed access to food reinforcers may be due to this mechanical feature of the procedures rather than to genuine risk aversion (e.g., Menlove et al., 1979; Reboreda & Kacelnik, 1991). With token-based procedures, however, one can assume good proportionality between programmed and obtained food across a wide range of values. In addition, sensitivity to reinforcer amount is enhanced by the discriminative functions of the tokens (Hackenberg, 2009; Jackson & Hackenberg, 1996; Yankelevitz, Bullock, & Hackenberg, 2008). Thus, token-based procedures may prove methodologically beneficial in assessments of risky choice.

In the present study, the fixed-amount option was held constant across blocks of conditions within a phase, and the average amount produced by the variable option was set above and below the fixed amount. Functions relating preference to relative reinforcer amounts were obtained at different fixed-amount payoff values, generating a detailed contingency space, and enabling an assessment of risk sensitivity across a range of outcomes, from risk averse to risk prone. To assess the role of the tokens in the choice patterns, some conditions were run without tokens. If tokens act to make the options more discriminable, removing them may be expected to turn preferences toward indifference.

METHOD

Subjects

Seven male White Carneau pigeons (Columba livia), designated 75, 995, 967, 710, 1447, 732, and 820 served as subjects. Pigeon 732 died during the experiment, and was replaced by Pigeon 820. All pigeons except 820 had previous experience with token-reinforcement schedules (see Lagorio & Hackenberg, 2010). Pigeons were individually housed in a temperature- and humidity-controlled colony room (lights on from 7:30 a.m. to 11:00 p.m.) with continuous free access to water and grit outside of the experimental sessions. They were maintained at approximately 85% of their free-feeding weights by means of supplemental postsession feeding.

Apparatus

Experimental sessions were conducted in a Lehigh Valley Electronics operant conditioning chamber, measuring 31 cm long, 35 cm wide, and 37 cm high. The modified control panel was equipped with three translucent Plexiglas response keys, requiring a force of approximately 0.26 N to operate, each 2.5 cm in diameter and horizontally spaced 5.5 cm apart. The two side response keys could be illuminated yellow, red, or green; the center response key could be illuminated white or red. A 7-W houselight was located 5 cm above the center response key. Twelve red stimulus lamps served as tokens and were positioned 2.5 cm above the response keys and arranged horizontally 0.8 cm apart. A 0.1-s tone accompanied each token illumination and darkening. A grain hopper, accessible through a square opening 5 cm in diameter, 9 cm below the center response key, provided access to mixed grain. The hopper contained a photocell beam that allowed for precise timing and control of food access. The hopper area contained a light that was illuminated during food deliveries. The chamber was contained within a sound-attenuating box, with a ventilation fan and white noise generators continuously running to mask extraneous noises. Experimental contingencies were arranged and data recorded via a computer programmed in MedState Notation Language with Med-PC® software for Windows.

Procedure

Preliminary training

Prior to the experiment proper, each pigeon was trained in token production and exchange responding for several weeks of daily sessions. Using a backwards-chaining procedure, pigeons were initially presented with the center key lit red in the presence of one illuminated token. One response on this red “exchange” key turned off the token light and presented food via the hopper for 2.5 s. Once this exchange response was established, pigeons were required to produce (illuminate) a token by pecking one or the other side key (illuminated either green or yellow with equal probability per trial). This token production was accompanied by a 0.1-s tone and led immediately to the exchange period, during which the center key was illuminated red and the side keys darkened. As before, one response on the center exchange key turned off the token light, produced a 0.1-s tone and provided 2.5-s access to mixed grain. This training continued until pigeons were reliably producing and exchanging tokens.

Experimental procedure

Sessions were conducted 7 days per week at approximately the same time each day. The first four trials in each session were always forced choice, in which only one choice alternative was available at a time (two of each trial type were presented per session). These forced-choice trials were designed to ensure exposure to both choice contingencies. Remaining trials were free choice, in which both options were available. The number of free-choice trials was typically 12 per session but was reduced to 10 in conditions in which reinforcer payoffs averaged over 80 per session.

A procedural schematic describing experimental contingencies during a free-choice trial is displayed in Figure 1. A trial began with a trial-initiation response—a single peck on the illuminated center white key; this produced the two side choice keys, one yellow and the other green. Pecks on the yellow key produced a variable amount (VA) of tokens whereas pecks on the green key produced a fixed amount (FA) of tokens. The location of the VA and FA options was determined randomly each trial. A single response (fixed ratio 1, FR 1) on either choice alternative extinguished the side key lights, and produced tokens. Token delivery occurred sequentially with one token illuminated every 0.3 s, accompanied by the 0.1-s tone. Immediately upon making a choice and concurrent with token delivery, a fixed interval (FI) 10-s delay was programmed on the center white key. This schedule requirement held constant the time to token exchange regardless of the number of tokens earned. A single peck to the white key at the end of the FI 10-s interval changed the center key color to red. This constituted the exchange period, during which each response on the center extinguished a single token light, initiated a 0.1-s tone, and provided 2.5-s access to mixed grain via the food hopper. All illuminated tokens were then exchanged for food in succession. Once all tokens had been exchanged, a 10-s intertrial interval (ITI) occurred before a new choice trial began. On trials when zero tokens were earned, the FI 10-s requirement remained in place and a single response on the exchange key was required prior to the ITI.

Fig. 1.

Procedural schematic of a free-choice trial. Earned (illuminated) tokens in the figure are denoted by white circles above the response keys (though in the chamber these were illuminated red). W = white, Y = yellow, G = green, R = red, VA = variable amount, FA = fixed amount.

Experimental conditions were defined as blocks of sessions at a given VA and FA value; phases were defined as blocks of conditions at a given FA value. Across conditions within a phase (i.e., at a constant FA payoff) the average token amount provided by the VA was manipulated. The VA distributions were either exponential or rectangular, and were chosen in order to hold constant the minimum and maximum number of possible token payoffs per trial (0 and 12, respectively). Including zeros in the VA distribution allowed the exploration of a wider range of payoffs (including VA 0.5 and VA 1) while still providing occasional large token payoffs. As illustrated in Figure 2 (which shows obtained rather than programmed distributions), when the average VA was between 0.5 and 5 tokens per trial, the exponential distribution was positively skewed and provided between 0 and 12 tokens. This distribution type provides a higher probability of small token payoffs but also allowed for the possibility of a larger token payoff. When the average VA was 6 tokens per trial, token amount probabilities were arranged according to a rectangular distribution that produced an approximately equal probability of receiving all token amounts between 0 and 12. When the average VA was 8 tokens per trial, amount probabilities were arranged according to a negatively skewed exponential distribution where there was a higher probability of large token payoffs.

Fig. 2.

Frequency distributions of obtained token amounts at each VA value, summed across pigeons and conditions. Panels display the different variable amount conditions. The number in each panel shows the average obtained amount. Panel along the bottom displays range of obtained variable amounts in each VA condition (0.5 - 8) for all pigeons.

Table 1 lists the sequence of phases and the number of sessions per condition for each pigeon. Conditions remained in effect for a minimum of 14 sessions and until proportion VA choice was stable. Choice was deemed stable when the following criteria had been met: (1) none of the three-session means over the last nine sessions were the highest or lowest three-session mean of the condition, (2) none of these three-session means deviated from the nine-session mean by more than 12% or one choice (whichever was higher), and (3) no monotonic trend in the three-session means was observed. In the first phase for all subjects, both FA and average VA payoffs were four tokens. Upon stability, the VA value in the next condition was dictated by each pigeon's preference in the preceding condition. If, at stability, the FA alternative was chosen on more (less) than 50% of the free-choice trials, the VA value was increased (decreased). Each pigeon was exposed to between two and eight different VA values (ranging from 0 to 8 tokens per trial) with interspersed replication conditions at four different FA values (2, 4, 6, and 8 tokens per trial).

Table 1.

Condition sequence and number of sessions per condition (in parentheses) for each pigeon.

Subject	Phase	Condition
		VA 0	VA0.5	VA 1	VA 2	VA 3	VA 4	VA 5	VA 6	VA 8
P75	FA 4			4(93), 7(33)	2(19)	6(15), 8(21)	1(23), 3(43)		5(18), 25 (24), 27(14)
	FA 6	18(27)	16(14)	10(30), 14(17)	12(28)	9(61), 19(21)	17(28)	15(24)	11(36), 13(33)
	FA 8	23(22)		22(14)		20(29)	24(28)		21(32)
P995	FA 4			4(30), 7(45)	2(31)	6(24), 8(48)	1(34), 3(25)		5(20), 22(32), 24(14)
	FA 6		14(36)	10(29)	12(62)	9(74)		13(71)	11(36)
	FA 8		15(50)	20(26)	18(24)	21(16)	16(50)		19(48)	17(33)
P967	FA 4			4(27), 7(40)	2(33)	6(26), 8(53)	1(28), 3(21)		5(43)	22(29), 24(14)
	FA 6	17(21)	16(26)	10(27), 14(20)	12(53)	9(34)		13(20)	11(48), 15(37)	18(23)
	FA 8	21(31)				20(31)				19(20)
P710	FA 4			4(55), 7(22)	2(45)	6(15), 8(28)	1(28), 3(24)		5(14), 21(23), 23(22)
	FA 6		17(32)	10(44), 15(28)	12(18)	9(41), 14(35)	16(36)	13(22)	11(22)
	FA 8		18(14)	20(24)			19(19)
P1447	FA2		13(33)	8(45), 11(32)	6(35), 12(16)	10(20)	14(14)	9(14)	7(18)
	FA 4			3(54), 16(24)	5(22)		1(64), 15(19)	4(27)	2(71), 19(21)	24(21), 26(15)
	FA 8	22(14)			21(21)		23(27)		20(17)
P732	FA4			4(28)	6(27)	8(63)	1(90), 3(23)	5(46)	2(38), 7(23)
	FA 6				11(28)	9(30)			10(32)
P820	FA2						5(24)		4(40)
	FA 4			7(32)			1(45), 6(14)	8(53)	2(63)	3(47)

No-token control conditions

For 5 of the pigeons, three final conditions were implemented to assess the role of the token stimulus complex in risk preferences. Conditions were conducted in an A–B–A sequence, where A was a replication of a previous FA 4 condition (with tokens) in which the VA alternative was preferred: FA 4 versus VA 6 (Pigeons 75, 995, and 710), FA 4 versus VA 8 (Pigeons 967 and 1447); B was an identical condition but without token illumination or the tone; A was a return to standard conditions with tokens. All other contingencies in the B condition remained in effect, including the FI 10 s requirement that produces the red (exchange) key and the exchange responses to gain access to food. Thus, time spent in each trial remained approximately the same in both A and B conditions; the only difference was the presence or absence of tokens. Stability criteria remained the same during this phase.

RESULTS

Variable Distributions

The obtained VA distributions well approximated the programmed distributions. Figure 2 displays the obtained token distributions for each VA value, summed across pigeons and conditions (each session is included). The value inset into each panel of Figure 2 is the across-subjects mean obtained VA value. The bottom-most plot in the figure displays the range of mean obtained VA values, with each line representing one VA condition for one pigeon. Given the close correspondence between the programmed and obtained amounts, the programmed distributions are used in the data analyses.

Choice Patterns

Figure 3 shows the proportion of choices for the VA alternative as a function of the mean number of tokens delivered by this alternative (x-axis). The four panels correspond to the different FA payoff phases. The figure legend identifies individual-subject data. The horizontal dashed lines reference indifference and the solid vertical lines indicate the FA in each phase. Connected points indicate data from initial conditions, with unconnected data points from replication conditions. Each data point is the mean of the last nine stable sessions in a condition.

Fig. 3.

Proportion of choices for the VA (y-axis) as a function of the average VA payoff (x-axis). Panels display the different FA payoffs (two, four, six, or eight tokens). Connected data points indicate initial exposure to a condition; unconnected data are from replication conditions. Horizontal dashed lines at 0.5 are reference lines indicating indifference between the two alternatives; vertical solid lines indicate the current FA payoff.

A majority of subjects were strongly risk prone—preferring the VA when it provided an average payoff equal to, and in some cases much less than, the FA; the latter data appear in the upper-left quadrant of each panel. Only when the VA and FA values were widely divergent did these pigeons demonstrate preference for the FA alternative. For each phase, in those conditions that arranged equal VA and FA values, the across-subjects mean preference exceeded 0.5, indicating risk proneness.

Two pigeons (1447 and 820) did not initially prefer the VA when the FA and VA values were set to four tokens. In these cases the FA was subsequently lowered to two tokens while a range of VA payoffs was implemented across conditions. During these conditions the pigeons showed almost exclusive preference for the VA (see the top left panel in Figure 3). This preference occurred when the VA provided an equal number of tokens on average, and also when it provided fewer tokens on average (as seen in the VA 2 and VA 1 conditions). Upon subsequent return to the FA 4 phase, these pigeons strongly preferred the VA 4 (0.98 and 0.93 for 1447 and 820, respectively), more in line with the preferences seen for the other pigeons.

Upon completion of the FA 4 conditions, the remaining 5 pigeons were moved to the FA 6 phase. These pigeons strongly preferred the VA 6 over the FA 6, and 3 of 4 pigeons tested at VA 3, 4 and 5 continued to prefer the VA alternative. At lower VA values preference shifted in favor of the FA alternative. Similar results were obtained when the FA provided eight tokens, with pigeons strongly preferring the VA even in cases in which it provided only half as many tokens as the FA 8 (see VA 4, VA 6, and VA 8 conditions).

Preferences for the VA were frequently a graded function of the VA payoff, as is evident from the individual-subject functions across conditions within a phase in Figure 3. In fact, it was rare for pigeons to exclusively prefer one alternative to the other within a daily session. Most pigeons would sample from both alternatives each session, even during conditions in which the VA alternative provided no food (in the FA 6 and FA 8 phases).

Side biases were common in this procedure, typified by strong preference for one key. Because the FA and VA choice alternatives alternated sides randomly, a strong side bias was indistinguishable from indifference between the alternatives (choice proportions around 0.5). Indeed, side biases occurred in conditions in which indifference might be expected based on choice proportions from prior conditions. Additionally, it was found that these side biases were condition-specific and replicable. When biases were evident, altering the conditions such that the FA and VA contingencies were more disparate would often eliminate the bias. Returning to the condition in which the bias occurred would bring the biased performance back. Biased performance was defined as (and conditions were terminated based on) fewer than two choices of a side key per session across the previous nine sessions. Because the biases were so easily identifiable, and were separable from sensitivity to the contingencies, performances during those conditions were not included in the summary analyses presented in Figure 3. Given the frequency of side biases in this experiment, there are fewer data points representing choice preferences in the 0.4 - 0.6 range, giving the function more of a sigmoidal, or step-like, than a graded appearance.

The graded sensitivity of preferences to payoff ratios can be seen more clearly in Figure 4 which shows preference for the VA alternative plotted as a function of the proportion of tokens obtained from that alternative (VA / VA+FA). Data points correspond to individual subjects' stable choices in all the FA phases. The solid line provides the best-fit linear regression whereas the dashed line is a reference indicating exact correspondence between relative choices and relative amount payoff, irrespective of the influence of risk from the VA alternative. With side biases excluded (top panel), choice proportions are more extreme than would be predicted from the simple expected-value prediction presented by the dashed line (best-fit slope = 1.513). The lower panel of Figure 4 presents data from all conditions, including those in which side biases occurred.

Fig. 4.

Proportion of choices for the VA (y-axis) as a function of the relative VA payoff (VA/VA+FA) across phases. Each data point is the stable proportion of choices for the VA for each subject and condition. Dashed lines show where responding would be if the proportion of choices perfectly matched the proportional payoff. The solid lines display best-fitting linear regression. The top panel shows conditions in which no side biases were present; bottom panel includes data from conditions with side biases.

We also examined choice as a function of the amount of tokens obtained on preceding free-choice trial. Of particular interest was the effect of the preceding VA of tokens obtained, ranging from 0 to 12 tokens, on the current choice. Figure 5 displays the across-subject mean proportion of VA choices on the current trial as a function of the VA tokens received on the previous trial. Error bars show standard deviations. Collapsed across the x-axis values, pigeons tended to choose the VA alternative after choosing that alternative on the preceding trial. Only when the preceding VA choice resulted in zero tokens did preference show any evidence of shifting toward the FA option. In general, there was a graded positive relation between number of tokens earned on the previous trial and probability of choosing the VA on the upcoming trial. This finding was apparent not only in the aggregate data, as seen in Figure 5, but also was demonstrated at the individual-subject level (data not shown).

Fig. 5.

Probability of choosing the VA on the upcoming trial (y-axis) as a function of the VA payoff received on the previous trial (between 0–12 tokens, x-axis). Bars represent across-subject means, with error indicating standard deviation. Note the truncated y-axis.

Figure 6 shows the effect of token illumination on VA choice in the 5 pigeons tested. These tests were conducted in a condition in which the VA alternative provided more tokens, on average, than the FA-4 alternative (see inset VA values in each panel). For 4 of 5 pigeons, removal of the tokens substantially decreased preference for the VA alternative, resulting in indifference or clear preference for the FA. The exception was Pigeon 75; this subject's preference initially approximated indifference but with continued exposure to no-token sessions, preference for the VA alternative emerged. The other pigeons' choices in the No-Token condition remained well below Token levels when tested for up to 7 weeks. Once tokens were reinstated, preference for the VA alternative was quickly replicated for all pigeons.

Fig. 6.

Proportion of choices for the VA alternative with and without tokens present for each pigeon. The conditions are FA 4 versus VA 6 (pigeons 75, 995, and 710) or FA 4 versus VA 8 (pigeons 967 and 1447). The dashed lines at 0.5 are reference lines indicating indifference between the two alternatives. The data are based on the means and standard errors over the last nine sessions per condition.

Response Latencies and Rates

Figure 7 shows between-subject mean latencies to choose the VA or FA alternatives as a function of preference for the VA alternative in that session. As preference for the VA increased, latencies to choose that alternative decreased and latencies to choose the FA alternative increased. Averaged across subjects and conditions, latency to choose the VA (1.18 s) was shorter than the FA (1.46 s; p = 0.016, two-tailed, Wilcoxon signed rank test).

Fig. 7.

Latency to choose the VA (closed circles) and FA (open triangles) as a function of that session's proportion of choices for the VA alternative. Data points represent across-subject averages (with standard error); reference lines show mean overall latency to respond for VA and FA across subjects and all phases of the experiment.

After tokens were illuminated, pigeons were required to complete an FI 10-s schedule on the center key before the tokens could be exchanged for food. Figure 8 shows the average rate of responding during the FI as a function of the number of tokens illuminated. Response rates were calculated by dividing the number of responses made after the final token was illuminated by the time remaining in the FI after that final token was lit. For this analysis, only responses made within the 10-s interval were included (i.e., the response that terminated the FI, and the time it took to make it, were not included). In Figure 8, data are shown for individual pigeons, and also averaged across pigeons (bottom right panel). In general, rates were substantially lower on trials with no tokens present than with tokens present. Rates then tended to increase as a function of the number of tokens accompanying the 10-s FI period. This overall trend was generally mirrored by the individual-subject data.

Fig. 8.

Mean responses per second during the 10-s FI in the presence of different numbers of tokens for each pigeon. Lower-right panel shows across-subject means.

DISCUSSION

The present study explored risky choice between fixed- (FA) and variable-amount (VA) token reinforcers. The initial phases of the experiment examined choices when the FA alternative provided two, four, six, or eight tokens (across phases) and the VA alternative provided between zero and eight tokens on average (across conditions). Strong and reliable preferences for the VA alternative were found, even when the VA alternative produced lower mean payoff rates than the FA option (Figure 3).

The preferences for VA outcomes seen in the current experiment are in line with studies examining risky choice with delays to reinforcement, in which strong preference for the variable delay is the typical finding (e.g., Bateson & Kacelnik, 1995; Herrnstein, 1964; Hursh & Fantino, 1973; Case, Nichols, & Fantino, 1995; Killeen, 1968; Lagorio & Hackenberg, 2010; Navarick & Fantino, 1975), but are inconsistent with much of the prior literature examining risky choice between reinforcer amounts (e.g., Essock & Reese, 1974; Logan, 1965; Mazur, 1989; Staddon & Innis, 1966). We will consider below some of the possible reasons for the discrepant results.

The most conspicuous difference between the present and past experiments on risky choice with reinforcer amounts is the present use of a token-reinforcement system, in which the different reinforcer magnitudes were correlated with the number of illuminated tokens. One feature of the token system that may have contributed to the present results is the close coupling of programmed and obtained reinforcer amounts. Each token was exchangeable for a discrete period of food access (i.e., 2.5 s per token). This differs from other studies in which reinforcer amounts are presented in a more continuous fashion, as duration of hopper access (e.g., Menlove et al., 1979; Reboreda & Kacelnik, 1991). The token-reinforcement arrangement not only allowed the programmed amounts to closely approximate the obtained amounts, but may also have provided additional discriminative stimuli (tokens) correlated with reinforcer amount. Absent the tokens, the fixed 2.5-s reinforcer cycles alone were clearly insufficient to produce the systematic preferences for the VA option, as evidenced by performances in the final phase of this experiment when tokens were removed from the experimental context (Figure 6). Under these conditions, preferences were more in line with previous research—indifference or risk aversion. Given that all of the other experimental features remained the same, it is clear that the tokens played a key role in sustaining preference for the VA over the FA alternative.

The tokens likely served multiple functions, reinforcing as well as discriminative. The specific procedural arrangements were optimal for establishing the tokens as conditioned reinforcers (see Hackenberg, 2009). The tokens (a) provided an immediate stimulus change following the choice, (b) bridged the 10-s delay between a choice and the exchange period, and (c) were paired with food during the exchange period. Once presented, the tokens may also serve important discriminative functions, as they signaled both (a) temporal proximity to the exchange period, and (b) reinforcer amount available during such exchange periods. Evidence for discriminative functions comes from the temporal distribution of responding across the 10-s FI period separating choices from the exchange period (see Figure 8). Response rates were appreciably lower in the absence than in the presence of tokens—consistent with a discriminative role—and increased slightly as a function of the number of tokens present during the delay. Because the number of tokens was correlated both with temporal proximity to the exchange period and amount of food in the exchange period, the relative contribution of each variable is difficult to evaluate in any precise fashion. What is clear is that the tokens were essential in generating and maintaining the observed preferences for the VA option. It would be wise to explore this more closely in future research.

It is of interest that preferences were graded rather than the all-or-none preferences more typical of discrete-trial choice procedures (e.g., Mazur, 1987; 2001; Snyderman, 1983). Pigeons sampled both options consistently, even when the VA option provided less food on average than the FA option (see Figure 3). Such choice patterns bear some resemblance to that seen in the literature on preferences for “partial” or probabilistic reinforcement (e.g., Dunn & Spetch, 1990; Gipson, Alessandri, Miller, & Zentall, 2009; Mazur, 1995; 1996; Spetch, Belke, Barnet, Dunn, & Pierce, 1990; Zentall & Stagner, 2011). These studies have found consistent preferences for a leaner probability of reinforcement (e.g., preferring an option providing 50% reinforcement over one providing 75% reinforcement), but only when different stimuli are presented during reinforced versus during nonreinforced trials (e.g., the stimuli that typically precede reinforcement are omitted from nonreinforced trials; Gipson et al. 2009; Mazur, 1989). In a similar manner, pigeons in the present study tended to prefer the VA option even when it produced an overall lower payoff than the FA option, and this preference occurred despite the substantial number of trials with zero token payoffs (Figure 2 displays the number of empty trials for each VA distribution). During trials in which pigeons earned food, the presence of the tokens and tone provided a distinct discriminative signal that food was forthcoming. Perhaps the evident absence of tokens on empty trials is in accord with these previous demonstrations of preference for leaner reinforcement schedules, and can help to explain the strong preference for the VA alternative even when it produced lesser amounts of food than the FA.

The VA distributions used in the current experiment were positive and negative exponential and rectangular, providing a minimum of zero and a maximum of 12 tokens. It has been noted that exponential distribution types (negative binomial in particular) represent food sources that many species experience in their natural foraging environments (Ito, Takatsuru, & Saeki, 2000; Iwasa, Higashi, & Yamamura, 1981). Because a range of distribution types have been employed in risky choice studies, such as bimodal (two-valued; e.g., Caraco, Martindale, & Whittam, 1980; Hamm & Shettleworth, 1987; Leventhal, Morrell, Morgan, & Perkins, 1959), it is not entirely clear how differently programmed variability may affect risk-sensitive preferences. A meta-analysis by Shafir (2000) suggested that the coefficient of variation (CV, [standard deviation / mean] x 100) of the variable option affects risk sensitivity, with smaller CVs (below approximately 100) promoting indifference, and larger CVs correlating both with increased risk-averse and risk-prone behavior. While the VA distributions in the current study did not have widely divergent CVs (ranging from 43 in the VA 8 condition to 132 in the VA 2 condition), future research might profitably explore the possibility that CV correlates with risk sensitivity, along with continuing to examine the use of bimodal, exponential, normal, and rectangular distributions to examine their possible influence on choice for variable over fixed amounts (Caraco & Chasin, 1984; Hamm & Shettleworth, 1987; Locey, Pietras, & Hackenberg, 2009; Paulsen, Platt, Huettel, & Brannon, 2011).

In addition to global measures of preference, some local measures of performance were found to vary systematically with the contingencies. For example, the probability of a VA choice was higher following trials in which the VA payoff was large, and lower following trials in which the VA option produced zero tokens (see Figure 5). This finding suggests that choices are sensitive not only to the overall payoffs but also to more local payoff contingencies. Even in conditions with a higher overall preference for the VA (e.g., VA 6 conditions), pigeons tended to switch to the FA option on the trial immediately following a VA choice that provided zero tokens.

Choice patterns were also found to covary with choice latencies. As shown in Figure 7, higher session-wide preferences for a given alternative were correlated with shorter latencies for that alternative. Collapsed across subjects and conditions, latencies were significantly shorter for the VA than the FA alternative, a difference that echoes the general preference seen for the VA option across conditions. This inverse relationship between preference and choice latency has been shown in other discrete-trial choice studies (Bateson & Kacelnik, 1995; Lagorio & Hackenberg, 2010; Reboreda & Kacelnik, 1991), and demonstrates the utility of examining choice latencies as an ancillary measure of reinforcer value.

While the current experiment was not conducted to compare quantitative models of reinforcer value, its parametric nature may make the data useful for testing predictions made by several recent models, such as one proposed by Locey and Dallery (2009), modified here to account for probabilistically delivered reinforcer amounts:

where the value (V) of a reinforcer is determined by the amount provided (A) after a delay (D), and k and z are free parameters associated with rate of delay discounting and amount sensitivity, respectively. To determine the value of a variable-amount reinforcer, the value of each of n possible amounts is calculated separately and multiplied by its probability of occurrence (Pi); with overall value equal to the summed total of these individual reinforcers. When reinforcer delays are constant (as they were in the present experiment), this equation reduces to state that reinforcer value is determined by reinforcer amount, with considerations given to the probabilities of different amounts in a distribution and to amount sensitivity (z). If z is equal to one, then the predicted value of a reinforcer is the same as its amount. When z is greater than one, the value of a reinforcer increases exponentially as a function of amount size, and a greater relative weighted value is assigned to larger amounts.

Equation 1 has had some success in describing behavior from several studies examining impulsive and risky choice (Dallery & Locey, 2005; Locey & Dallery, 2009), yet the utility of this equation for describing a broader range of conditions—particularly those focusing solely on variable amount discriminations—is unclear. To test its generality, we fitted Equation 1 to our data from conditions in which no side biases were observed. Because our obtained amounts were similar to programmed probabilities (see Figure 2), the value of a variable amount was calculated by summing the values of each amount in the distribution multiplied by its programmed probability of occurrence. Allowing z to vary by individual produced values ranging from 1.5 to 11.2 (mean = 4.3), with 65 % of variance in the data accounted for (VAF, accomplished by reducing the sum of the squared residuals using the Solver function in Microsoft Excel ®). Fits were improved to 77% (ranging from 66% to 92% across subjects) by allowing amount sensitivity parameter z to vary separately when calculating the value of the VA and FA options. While Equation 1 accounted for a large amount of variance in the data, the free parameter varied substantially across subjects, making its theoretical interpretation less clear. Future research parametrically manipulating both amount and delay would improve our understanding of how Equation 1 might best be modified to describe how reinforcer value is jointly affected by amount and delay sensitivity.

In conclusion, the present study expanded the analysis of risk-sensitive decision making in several important ways. First, methodologically, the study explored risky choice over a wider parametric range—31 unique choice situations on a within-subject basis—than has been attempted in prior research, including 27 conditions in which the net programmed gain for fixed and variable options differed. Most prior research has been limited to choices with equal net gain. Second, empirically, the present findings are the first to show clear, robust and replicable preferences for variable over fixed reinforcer amounts with nonhuman subjects. Insofar as such preference for variable outcomes often came at the expense of overall reinforcer amount, such patterns had negative net utility, and in this sense may provide a good laboratory model of gambling. To this point, laboratory models have been limited in the ability to produce genuinely suboptimal choice patterns—a prerequisite for a model of gambling. Third, conceptually, the token system utilized in the present research connects to an expanding body of research using token economic systems to explore a range of topics across species and settings, including conditioned reinforcement (Bullock & Hackenberg, 2006; Foster, Hackenberg, & Vaidya, 2001; Tarbox, Ghezzi, & Wilson, 2006), punishment (Pietras & Hackenberg, 2005; Raiff, Bullock, & Hackenberg, 2008), and reinforcer accumulation (Yankelevitz et al., 2008), in addition to impulsive and risky choice (Hackenberg & Vaidya, 2003; Jackson & Hackenberg, 1996; Lagorio & Hackenberg, 2010; Locey et al., 2009; Paulsen et al, 2011; Reed & Martens, 2011). Token systems have a number of distinctive features that make them especially useful in a translational analysis of decision making, including currencies more analogous to monetary-based economies in human affairs, bringing a wide range of economic variables—income, savings, expenditures, wages, and prices—within reach of an experimental analysis.