Empirical Matching, Matching Theory, and an Evolutionary Theory of Behavior Dynamics in Clinical Application

Abstract

This article provides an overview of highlights from 60 years of basic research on choice that are relevant to the assessment and treatment of clinical problems. The quantitative relations developed in this research provide useful information about a variety of clinical problems including aggressive, antisocial, and delinquent behavior, attention-deficit/hyperactivity disorder (ADHD), bipolar disorder, chronic pain syndrome, intellectual disabilities, pedophilia, and self-injurious behavior. A recent development in this field is an evolutionary theory of behavior dynamics that is used to animate artificial organisms (AOs). The behavior of AOs animated by the theory has been shown to conform to the quantitative relations that have been developed in the choice literature over the years, which means that the theory generates these relations as emergent outcomes, and therefore provides a theoretical basis for them. The theory has also been used to create AOs that exhibit specific psychopathological behavior, the assessment and treatment of which has been studied virtually. This modeling of psychopathological behavior has contributed to our understanding of the nature and treatment of the problems in humans.

The purpose of this article is to provide an overview of highlights from 60 years of basic research on choice that are relevant to clinical applications. In the first two sections, empirical matching and early matching theory are reviewed, followed by a discussion of their clinical applications. In the third section, later developments in matching theory and their applied implications are discussed. And in the fourth section, an evolutionary theory of behavior dynamics is presented, its relation to empirical matching and matching theory is discussed, and its applied implications are examined.

Empirical Matching and Early Matching Theory

Empirical matching was first reported in 1961 by R. J. Herrnstein, who studied the key pecking of three pigeons on concurrent variable interval variable interval (VI) schedules of grain reinforcement (Herrnstein, 1961). Herrnstein found that the pigeons’ relative rate of responding on a concurrent schedule component equaled, or “matched,” the relative rate of reinforcement obtained for that responding. This is expressed mathematically as

$$\frac{B_1}{B_1+B_2}=\frac{R_1}{R_1+R_2},$$ 1

where B and R are rates of responding and obtained reinforcement, respectively, and the numerical subscripts refer to the two components of the concurrent schedule. Equation 1 is a straight line (y = x) with unit slope and 0 intercept. Expressed in words, the equation says that if a pigeon obtains a specific proportion of reinforcers from an alternative, then it will put that proportion of pecks on the alternative. It is important to recognize that this phenomenon is not trivial inasmuch as the number of pecks allocated to an alternative may exceed the number of reinforcers obtained from it by a factor of 100 or more (Herrnstein, 1961). Equation 1 can also be written as a ratio by taking the reciprocal of both sides, separating terms, subtracting 1 from both sides, and taking the reciprocal of the result:

$$\frac{B_1}{B_2}=\frac{R_1}{R_2}.$$

A large body of evidence collected after 1961 showed that Equation 1 applies to only a small subset of data. Consequently, Staddon (1968) suggested that a power-function version of the equation might be more widely applicable. The power-function version is constructed from the ratio form of Equation 1 and is written,

$$\frac{B_1}{B_2}=b{\left(\frac{R_1}{R_2}\right)}^a,$$ 2

where b and a are parameters that are estimated from data. Note that, like Equation 1, Equation 2 is an empirical statement, that is, it is not derived from any theory. Note also that Equation 2 reduces to Equation 1 in its ratio form when a = b = 1. Equation 2 therefore subsumes Equation 1 and goes beyond it.

The b parameter in Equation 2, which is usually referred to as the bias parameter, reflects asymmetries between the two alternatives of a series of concurrent schedules other than the asymmetry caused by reinforcement rate differences. If there are no additional asymmetries across a series of concurrent schedules, then b = 1. But if, for example, the magnitude of the reinforcer is greater on alternative 1 than on alternative 2, then b will be greater than 1; if the asymmetry favors alternative 2, then b will be less than 1. Many asymmetries can be arranged experimentally, such as differences in the force required to operate the manipulanda, differences in the delay of reinforcement, qualitative differences between the reinforcers (e.g., buckwheat on one alternative versus mixed grain on the other), and so on. All asymmetries affect the b parameter. These asymmetries can also be accommodated by a more complicated form of Equation 2 known as the concatenated matching law (Baum & Rachlin, 1969; Davison & McCarthy, 1988; Killeen, 1972; Rachlin, 1971). This more complicated form will not be considered in this article.

The a parameter in Equation 2 accounts for deviations from perfect matching (which is Equation 1 in its ratio form) that may occur in the direction of indifference (a < 1) or in the direction of exclusive preference (a > 1). These deviations are usually referred to as undermatching and overmatching, respectively. In the case of undermatching, behavior on a series of concurrent schedules is less sensitive to reinforcement than required by perfect matching. This means that preference is not as extreme as the ratio form of Equation 1 requires. Put another way, undermatching indicates that preference is closer to indifference than is required by perfect matching. In the case of overmatching, behavior on a series of concurrent schedules is more sensitive to reinforcement than required by perfect matching. In other words, preference is more extreme, closer to exclusive preference (where all behavior is allocated to one alternative), than the ratio form of Equation 1 requires.

Staddon’s (1968) proposal of Equation 2 was echoed by other researchers, and additional studies and extensive reanalysis of data in the ensuing 15 years confirmed that Equation 2 was indeed widely applicable (Baum, 1974, 1979; Baum & Rachlin, 1969; de Villiers, 1977; Myers & Myers, 1977; Staddon, 1972; Wearden & Burgess, 1982). Equation 2 is often referred to as the generalized matching law (GML; e.g., Baum, 1974).

While research on empirical matching was underway, Herrnstein (1970) suggested that Equation 1 could be extended to responding on single-alternative schedules by considering the single schedule to be one component of a concurrent schedule. The other component could be considered an aggregate of all other behaviors that might be emitted in the experimental environment. According to this conceptualization, even behavior on single schedules entails choice, for example, between pecking the key and doing anything else (such as preening). If behavior on single schedules entails choice, then Equation 1 must apply as follows,

$$\frac{B}{B+B_e}=\frac{R}{R+R_e},$$ 3

where B and R refer to the response rate and obtained reinforcement rate on the single schedule, and B_e and R_e refer to the response rate and obtained reinforcement rate for all other behaviors considered as an aggregate. The e subscript stands for extraneous behavior, that is, behavior other than responding on the single schedule. Herrnstein next made use of the well-known finding in concurrent schedules that when reinforcement rate is increased in one component, response rate in that component increases, while at the same time response rate in the other component decreases (Catania, 1966). This phenomenon is known as contrast; it is as though responses that are added to a component because of an increase in reinforcement rate have been taken away from the other component. Perhaps it is the case, then, that changes in reinforcement simply cause a reallocation of behavior, such that the total amount of behavior remains constant. This hypothesis can be written

$$k=B+B_e,$$

where k, the total amount of behavior, is asserted to remain constant across all changes in reinforcement. The principal benefit of this hypothesis is that it allows Equation 3 to be solved for the absolute response rate, B, on the single schedule. Setting the denominator on the left side of Equation 3 to k, and solving for B gives

$$B=\frac{\mathit{kR}}{R+R_e},$$ 4

which is a rectangular hyperbola that passes through the origin (R = 0), increases monotonically, and has a y-asymptote at B = k (as R becomes very large). Equation 4 will be referred to here as early matching theory. It is a theory because specific interpretations of the parameters, k and R_e, are required to obtain and apply the equation. It is clear that the equation cannot be obtained unless k is constant across all changes in reinforcement. Also, the equation cannot be applied to response and reinforcement rates in a series of single schedules unless R_e also remains constant across all changes in reinforcement in the series of schedules, including changes in R. If it could be shown that k varies with, for example, reinforcer magnitude, and/or that R and R_e are negatively correlated (i.e., they show contrast), then the theory that gave rise to Equation 4 would be falsified. Herrnstein fitted Equation 4 to data from a study of six pigeons pecking keys for food on single VI schedules (Catania & Reynolds, 1968). He found that the equation provided an excellent description of the pigeons’ response rates.

Not long after Herrnstein proposed Equation 4, he and Peter de Villiers reported fits of the equation to single-schedule data from many experiments using rats, monkeys, pigeons, and humans as subjects (de Villiers, 1977; de Villiers & Herrnstein, 1976). They found that the equation provided excellent descriptions of the data in most cases, and furthermore, that it provided a better description of the data than two other function forms with similar differential properties.

Work on Equations 2 and 4 continued well beyond the late 1970s. Approximately 40 years of research from 1961 to 2000 showed that the equations provided excellent descriptions of the behavior of many vertebrate animal species (McDowell, 2013a). An extensive body of research dating through at least 2010 has shown that human behavior is also well-described by these equations (Beardsley & McDowell, 1992; Bradshaw et al., 1976, 1977, 1978; Bradshaw et al., 1979; Buskist & Miller, 1981; Conger & Killeen, 1974; Dallery et al., 2005; Kollins, Newland, & Critchfield, 1997a, 1997b; McDowell & Wood, 1984, 1985; Moffat & Koch, 1973; Pierce & Epling, 1983; Ruddle et al., 1981; Ruddle et al., 1982; Takahashi & Iwamoto, 1986), including naturally occurring human behavior (McDowell, 1981, 1988; McDowell & Caron, 2010a; Pierce et al., 1981; Plaud, 1992; Reed et al., 2006; Stilling & Critchfield, 2010; Symons et al., 2003; Vollmer & Bourret, 2000).

Clinical Applications of Empirical Matching and Early Matching Theory

The applied relevance of empirical matching and early matching theory began to be recognized in the late 1970s, soon after Herrnstein (1970) proposed Equation 4 (Bradshaw & Szabadi, 1978; Cliffe & Parry, 1980; Epling & Pierce, 1983; McDowell, 1981, 1982; Myerson & Hale, 1984). Applications to specific clinical problems followed rapidly. These included applications to aggressive, antisocial, and delinquent behavior (Dishion et al., 1996; Dishion, Andrews, & Crosby, 1995; Dishion, French, & Patterson, 1995; McDowell & Caron, 2010a, 2010b; Snyder et al., 1996; Snyder, Horsch, & Childs, 1997; Snyder & Patterson, 1995; Snyder, Schrepferman, & St. Peter, 1997), ADHD (Kollins, Lane, & Shapiro, 1997; Murray & Kollins, 2000; Taylor et al., 2010), bipolar disorder (Bradshaw & Szabadi, 1978; Szabadi et al., 1981), chronic pain syndrome (Fernandez & McDowell, 1995), intellectual disabilities (Oliver et al., 1999), pedophilia (Cliffe & Parry, 1980), and self-injurious behavior (McDowell, 1981, 1982; Symons et al., 2003). In this literature, Equations 2 and 4 were often found to provide accurate quantitative descriptions of problem behaviors in both laboratory and natural environments. But the most useful clinical information is provided by the values of the parameters, a and b, estimated from fits of Equation 2, and by the novel contextual understanding of reinforcement entailed by Equation 4. These properties of the equations have contributed in important ways to the assessment and treatment of problem behaviors, as will be explained in the following sections.

Empirical Matching

The b parameter

One example of the clinical application of empirical matching (Equation 2) is provided by Cliffe and Parry (1980), who studied the behavior of an incarcerated individual with pedophilia on concurrent schedules where the reinforcer in each component was the opportunity to view slides of adult men, adult women, or female children. The adult slides were obtained by the experimenters from commercial magazines and were sexually provocative; the slides of female children were obtained from the offender, who had collected them from print sources. The authors stated that the slides of female children would not be considered sexually interesting by most people. All possible pairs of slide categories (women–men, women–children, and men–children) were arranged as reinforcers in three sets of concurrent schedules. This matching-based assessment of sexual preference is an alternative to penile plethysmography (Barker & Howell, 1992). It is more accessible, easier to administer, probably more acceptable to the recipient, and can be used even if the individual has erectile problems.

Equation 2 provided an excellent description of the participant’s behavior in all series of schedules, accounting for 87% to 92% of the variance in the logs of the response rate ratios. It is important to understand that Equation 2 must be fitted in logarithmic transformation (Baum, 1974) so that ratios less than 1 and ratios greater than 1 influence the fit equally. The bias parameters estimated from the fits showed a preference for women over men (b = 1.9), men over children (b = 2.7) and women over children (b = 5.4). In no case were pictures of female children preferred over pictures of adult men or women. Note that the individual’s preference for adult women over female children was exceptionally large. This finding was consistent with clinical information that the individual was attracted to adult women but had a history of problems establishing relationships with them. It also strongly supports the decision to work therapeutically with the offender on improving his social skills with adult women, which could then replace his sexual interactions with female children. The finding of a strong preference for adult women also suggested that the use of orgasmic reconditioning as a possible treatment (VanDeventer & Laws, 1978) was not indicated for this individual.

A second example of the clinical utility of the bias parameter in Equation 2 is provided by McDowell and Caron’s (2010a, 2010b) analysis of the verbal behavior of pairs of 13- and 14-year old boys with anti-social behavior from a study by Dishion et al. (1996). The boys were assigned topics to discuss, and their videotaped verbal behavior was coded into two categories, namely, rule-break talk and normative talk. Each boy’s responses to the other boy’s verbal behavior were also coded into two categories, namely, positive social (reinforcement) and other. Note that this was naturally occurring behavior (apart from the assigned topics), and that it was coded in such a way that it could be analyzed as a concurrent schedule. Note also that the behaviors on the two alternatives were different in this example, whereas in the Cliffe and Parry (1980) example, the reinforcers on the two alternatives were different.

Eighty-one of the boys in Dishion et al.’s (1996) study were also assigned a child deviance score, which quantified the amount of anti-social behavior a boy exhibited in his home, school, and neighborhood environments. McDowell and Caron (2010b) separated the sample of 81 boys into quartiles based on their child-deviance scores, and fitted Equation 2 to the data from each quartile separately. The fits provided excellent descriptions of the boys’ verbal behavior, accounting for 85% to 97% of the variance in the dependent variable in each quartile. The values of the b parameters estimated from the fits indicated that normative talk was preferred to rule-break talk by the boys in all quartiles, but that this preference was strongest for the boys in the least deviant quartile, and decreased systematically as the child-deviance score increased. Specifically, the bias parameters favoring normative talk for the boys in the least deviant to those in the most deviant quartile were 3.7, 3.1, 2.8, and 1.9, respectively. Dishion et al. noted that a boy’s interactions with peers (and others) who exhibit anti-social behavior can increase the extent of his own aggressive, anti-social, and delinquent behavior, and they consequently referred to these interactions as deviancy training. McDowell and Caron’s finding of decreased preference for normative talk as a function of increased child-deviance scores provides a striking confirmation of the effect of deviancy training, and shows that it extends to verbal behavior.

The a Parameter

One example of the clinical relevance of the a parameter in Equation 2 is provided by Kollins, Lane, and Shapiro (1997) and Taylor et al. (2010), who compared the behavior of children with and without an ADHD diagnosis on concurrent schedules arranged in the laboratory. Their fits of Equation 2 often yielded lower estimates of a for the ADHD-diagnosed children than for children without the diagnosis. This indicates that the behavior of the ADHD-diagnosed children was less sensitive to reinforcement, and provides quantitative confirmation of early theoretical speculation that reinforcement sensitivity is reduced in these children (e.g., Douglas, 1983; Haenlein & Caul, 1987). The lower a parameter for the ADHD-diagnosed children can also be understood as reflecting greater impulsivity in these children, that is, a tendency to switch away too readily from a rewarding behavior. This may appear to an observer as either hyperactivity or insufficient attention to reward.

A second example of the clinical utility of the a parameter can be found in McDowell and Caron’s (2010b) reanalysis of Dishion et al.’s (1996) data (described in the previous section). Recall that McDowell and Caron fitted Equation 2 to the verbal behavior of 81 boys with anti-social behavior who were separated into quartiles based on their child-deviance scores. Estimates of both the a and the b parameters were obtained from fits of Equation 2 to the data in each quartile. McDowell and Caron found that the a parameter decreased with increasing child-deviance scores, which indicates that behavior became less sensitive to reinforcement as the child-deviance score increased. This finding is especially noteworthy because the reinforcer was social approval. The greater the child-deviance score, the less sensitive the boy’s behavior was to social approval.

As these four examples show, an important applied contribution of Equation 2 is to clinical assessment. The person with pedophilia strongly preferred provocatively posed adult women to female children, the boys with anti-social behavior preferred normative talk less the greater their child-deviance score, ADHD-diagnosed children were less sensitive to reinforcement than nondiagnosed children, and boys with anti-social behavior were less sensitive to social reinforcement the greater their child-deviance score. Assessment information like this can also provide clues about possible intervention strategies. For example, orgasmic reconditioning was not indicated as a possible treatment for the person with pedophilia, whereas therapeutic work on his social relationships with adult women was indicated. For the boys with anti-social behavior, reinforcers other than social approval (e.g., vouchers or privileges) may be more effective in interventions.

Early Matching Theory

As noted earlier, the principal applied contributions of Equation 4 stem from its novel contextual understanding of reinforcement. According to the equation, the effect of contingent reinforcement depends on the context of reinforcement in which it occurs. As is clear from the equation, if the reinforcement context is rich, that is, if R_e is large, then a given rate of contingent reinforcement, R, will support a lower response rate, B, than if the reinforcement context is lean, that is, if R_e is small (McDowell, 1982, 1988). To illustrate the applied relevance of this property of the equation, consider an intervention that arranges contingent reinforcement for a desired behavior. According to Equation 4, such an intervention will have a stronger effect in a lean environment than in a rich one. An example of a lean environment (for, say, a child) might be a traditional classroom. In this environment, a relatively small amount of contingent reinforcement is likely to support a substantial amount of behavior, according to Equation 4. An example of a rich environment might be the child’s home, where there may be many opportunities to play video games, engage with social media and so on. In this environment, substantially more contingent reinforcement is likely to be required to produce a desired increase in behavior, according to Equation 4. The equation also suggests an alternative treatment for increasing a target behavior in a rich environment, namely, reducing the background reinforcement. This could be accomplished by, for example, restricting cell phone or video game use. The result, according to Equation 4, would be an increase in the target behavior, even if the reinforcement supporting that behavior does not change. Of course, the two interventions may be used together; reinforcement for the desired behavior can be arranged while reducing the background reinforcement at the same time. This combined intervention will increase the effectiveness of contingent reinforcement for a target behavior, according to Equation 4.

McDowell (1981, 1982, 1988) reported an interesting intervention with a 22-year-old male client that was based on Equation 4. The client, who had mild intellectual disabilities, showed oppositional and aggressive behavior at home and in a sheltered workshop. The aggressive behavior often increased in intensity during an episode, and sometimes culminated in assault. The parents sought treatment when their son was dismissed from the sheltered workshop after threatening a staff member with scissors. The treatment strategy was to first address the aggressive behavior at home, and then bring a successful treatment into the sheltered workshop. On interview it appeared that the client’s aggressive and assaultive behavior was supported by social consequences, such as attention, and also by often getting his way. One possible treatment was extinction, but the potentially explosive behavior that might occur during an extinction burst made this option unacceptable. Another treatment possibility was punishment, but the implementation of, say, a time-out contingency, for an assaultive 22-year-old man also seemed dangerous. Instead, the treatment strategy was to increase background reinforcement in the home. A token system that delivered points for self-help behaviors (e.g., showering, brushing teeth), work around the house, and academic behavior (e.g., reading) was arranged. The points could be redeemed for money in a total amount equal to what the young man was earning in the sheltered workshop. According to Equation 4, this intervention should reduce the amount of oppositional and aggressive behavior, even if the consequences for that behavior do not change. At baseline, the client showed almost daily episodes of oppositional and aggressive behavior. After 8 weeks of treatment, this behavior was reduced by about 80% to a frequency of about once per week. The mother also reported that the episodes were less intense and of shorter duration than before treatment. At this point, the young man returned to the sheltered workshop and a token reinforcement system for appropriate behavior was implemented for all workshop participants.

Later Developments in Matching Theory and Their Applied Implications

Not long after Herrnstein (1970) proposed Equation 4, the validity of the constant-k assumption necessary to derive the equation came under scrutiny. Early data bearing on the constancy of k were equivocal (de Villiers, 1977; Herrnstein, 1981; McDowell, 1980), and later experiments showed conclusively that k varied with changes in reinforcement and response properties, such as reinforcer magnitude and response effort (McDowell, 2013a; McDowell & Wood, 1984, 1985). In the course of addressing this violation of matching theory, it was noted that Equation 4 was derived from Equation 1, which, as discussed earlier, is known to apply to only a limited subset of concurrent schedule data (McDowell, 1986). Perhaps a single-alternative equation derived from the more widely applicable Equation 2 would resist disconfirmation. Starting with Equation 2 and following the same algebraic logic as Herrnstein’s (1970) derivation of Equation 4, including the assumption of a constant k, yields

$$B=\frac{k{R}^a}{R^a+\frac{{R_e}^a}{b}},$$ 5

where the a and b parameters are from Equation 2 (McDowell, 1986). Note that the second term in the denominator on the right is required to remain constant across a series of single-alternative schedules inasmuch as the three quantities that constitute the term are required to remain constant across those schedules. Hence, like Equation 4, Equation 5 is hyperbolic, passes through the origin, increases monotonically, and has a y-asymptote at B = k. It is evident that Equation 5 reduces to Equation 4 when a = b = 1, which means that it could be considered a generalized form of matching theory, just as Equation 2 is considered a generalized form of empirical matching.

Unfortunately, Equation 5 did not save matching theory from disconfirmation, that is, from the finding that the constant-k assumption entailed by the equation does not hold. Efforts to rescue the theory (e.g., McDowell, 2013a; McDowell & Popa, 2010) ultimately failed (McDowell et al., 2017; McDowell & Calvin, 2015), leading to the conclusion that matching theory must be discarded. In other words, Equation 5 cannot be obtained legitimately from Equation 2.

The Empirical Remains of Matching Theory

Although Equation 5 was generated by a theory that was later shown to be false, it nevertheless provides an accurate description of a large body of data (McDowell, 2013a). This may seem odd, but it is not unheard of in the history of science. For example, Maxwell’s theory of electromagnetism, and the descriptive equations it generated, suffered the same fate. The theory was ultimately shown to be false, even though the equations it generated described empirical phenomena accurately (McDowell, 2013b). Maxwell’s equations remain an important part of mathematical science (Harman, 2001).

Equation 5 may be rewritten without parametric interpretations as

$$B=\frac{k{R}^a}{R^a+c},$$ 6

where k, a, and c are simply constants that must be estimated from data. Hence, like Equation 2, Equation 6 is a purely empirical statement that accurately describes the behavior of many species, including humans. This means that we are left with two empirical equations from the matching literature, one for concurrent schedule responding (Equation 2), and one for single schedule responding (Equation 6).

Applied Implications of the Empirical Remains of Matching Theory

The disconfirmation of matching theory of course has no implications for the applied relevance of empirical matching, Equation 2. Details of the applied relevance of this equation discussed earlier remain valid. Regarding the applied relevance of matching theory, recall that this was due principally to its contextual understanding of reinforcement. But empirical research has shown that the effect of reinforcement is indeed contextual, that is, it depends on the level of background reinforcement present in the environment (Belke & Heyman, 1994; Bradshaw et al., 1976; Bradshaw, 1977; McDowell, 1981, 1982; Soto et al.’s, 2005, data reanalyzed by McDowell & Klapes, 2020). This literature includes studies of animal and human behavior, as well as studies of clinically interesting human behavior. Hence, the rate of background reinforcement in an environment has been shown to affect the parameter, c, in Equation 6. In particular, larger rates of background reinforcement are associated with larger values of c. This means that the applied implications of matching theory discussed earlier are also valid for the empirical remains of the theory, namely, Equation 6.

Equation 6 has at least one additional applied implication. Because k in the equation is known to vary with reinforcement and response properties such as reinforcer magnitude and response effort (McDowell et al., 2017; McDowell & Calvin, 2015) it can serve a scaling function in single schedules just as the parameter, b, in Equation 2 serves a scaling function in concurrent schedules. An example is provided by Bradshaw and Szabadi (1978) and Szabadi et al. (1981), who studied the behavior of two individuals with bipolar disorder who pressed buttons on VI schedules of monetary reinforcement in a laboratory setting. The participants worked on the schedules during manic, euthymic, and depressed phases of their illness. Bradshaw and colleagues fitted Equation 6 with a = 1 to 8 sets of data from the participants and found that it provided an excellent description of their behavior, accounting for between 85% and 99% of the variance in response rates (mean = 92%). Estimates of k and c obtained from the fits depended on the participant’s mood state. The ks were larger and the cs were smaller during manic phases of the illness than during euthymic phases. This suggests that the benefit/cost of responding was more favorable (larger k), and that responding was less affected by background reinforcement (smaller c), during manic phases of the illness. During depressed phases, the ks were smaller and the cs were larger than during euthymic phases, suggesting that the benefit/cost of responding was less favorable and that responding was more affected by background reinforcement during the depressed phases. These findings make sense from a clinical perspective. In addition, they suggest that it may be possible to use Equation 6 to obtain a quantitative assessment of the overall severity of bipolar disorder, the extent to which the manic or the depressed phase of the disorder is more pronounced, and the responsiveness of the disorder to treatment. In general, the more the parameter estimates in the manic and depressed phases differ from those in the euthymic state, the more severe the illness. If these differences are greater in one of the disordered phases than in the other, then that phase is more severe. And finally, treatment effectiveness would be reflected in reduced differences overall. It is also worth noting that this scaling of mood states could not be accomplished using concurrent schedules and the b parameter in Equation 2 because the two alternatives cannot be associated with different mood states.

An Evolutionary Theory of Behavior Dynamics and its Applied Implications

A recent development in this line of research is an evolutionary theory of behavior dynamics (ETBD; McDowell, 2004). This theory provides a theoretical foundation for the empirical Equations 2 and 6, and then goes beyond them. It is a complexity theory (McDowell, 2013b, 2013c; McDowell & Popa, 2009) stated in the form of simple low-level rules, the joint operation of which produces high-level emergent outcomes. The ETBD does not generate equations that describe behavior. Instead, it generates behavior in artificial organisms (AOs) that are animated by the theory. The behavior of the AOs can then be studied just as the behavior of live organisms is studied. Finding that their behavior is indistinguishable from live-organism behavior constitutes support for the theory.

The low-level rules of the ETBD are Darwinian rules of selection, reproduction, and mutation. They operate on a population of potential behaviors that animates the AOs. Each behavior in the population is represented by a decimal (base 10) integer, which is referred to as the behavior’s phenotype, and by the binary (base 2) expression of that integer, which is referred to as the behavior’s genotype.

One behavior is emitted at random from the population of potential behaviors at each tick of algorithmic time. Following this emission, pairs of what may be referred to as parent behaviors are chosen from the population. If the emitted behavior resulted in a benefit to the AO, such as a reinforcer, then the theory’s selection rule operates. According to this rule, parents are chosen such that behaviors closer in phenotype to the just-emitted behavior are more likely to become parents than behaviors farther from that phenotype. If the emitted behavior did not result in a benefit, then the selection rule does not operate, and parents are chosen at random from the population.

Once pairs of parents are chosen, the theory’s reproduction rule operates. According to this rule, each pair of parents contributes random bits (binary digits, i.e., zeroes and ones) from their genotypes to construct a child genotype. The result is a new population of child behaviors that then replaces the current population. If parent behaviors were chosen via the selection rule, then reproduction tends to concentrate the child behaviors near the phenotype of the behavior that was just emitted, making that phenotype and phenotypes close to it more likely to be emitted in the next tick of algorithmic time. If parent behaviors were chosen at random, then reproduction tends to spread the child behaviors out over the entire range of phenotypes in the population.

After a new population of behaviors is built, the mutation rule operates. According to this rule, a percentage of behaviors is chosen at random from the population and a random bit in each of these behaviors is flipped (i.e., changed from 0 to 1, or from 1 to 0). The percentage of behaviors that undergoes mutation is referred to as the mutation rate.

The operation of these rules is described in detail for general readers by McDowell (2013d), and for behavior-analytic readers by McDowell (2019). It may be worthwhile to consider how the Darwinian rules of the theory make contact with the material world. Many authors have suggested that the brain operates as a selectionist system (e.g., Edelman, 1978, 1987; Hayek, 1952a, 1952b; McDowell, 2010; Pringle, 1951). Most recently, McDowell and Riley (2020) have identified neural structures and events that may implement the processes of selection, recombination, and mutation specifically. An alternative to direct neural implementation is the assertion of computational equivalence between the algorithmic operation of the theory and the material operation of the brain (McDowell, 2013b). Computational equivalence means that the theory’s operations and the brain’s operations give the same answer, even though the operations themselves are different. In the case of computational equivalence, the algorithmic operation of the theory can be said to supervene on the material operation of the brain (McDowell, 2017, pp. 136–137).

The simple rules of the ETBD tell an appealing story about behavior, namely, that it evolves in ontogenetic time under the selection pressure of consequences from the environment. As noted earlier, the behavior of AOs animated by the theory and working on concurrent and single schedules is accurately described by the empirical Equations 2 and 6 (McDowell, 2004, 2019; McDowell et al., 2008; McDowell & Caron, 2007; McDowell & Popa, 2010). This is a remarkable finding given that the theory operates algorithmically by executing rules unrelated to the equations. However, because Equations 2 and 6 are emergent properties of the ETBD, the theory provides a theoretical foundation for them; it asserts that they are consequences of evolutionary dynamics. It follows that the ETBD effectively addresses the principal shortcoming of matching theory (Equations 4 and 5), namely, that single schedules must be conceptualized as concurrent schedules with a constant total amount of behavior allocated between the target and background components. The ETBD does not require this conceptualization, which, as noted earlier, has been shown to be false. The theory generates an accurate quantitative description of behavior on single schedules (Equation 6) on its own, by means of evolutionary dynamics.

The ETBD also goes beyond Equations 2 and 6 by describing a host of additional phenomena known to characterize the behavior of live organisms (McDowell, 2019). These include the relation between changeover rates and preference (McDowell, 2013d), the tracking of changes in reinforcement-rate ratios by response-rate ratios (Chi, 2019; Kulubekova & McDowell, 2013), response allocation when both the rate and magnitude of reinforcement are varied (McDowell et al., 2012), quantitative properties of responding on each alternative of a concurrent schedule (McDowell, 2013d; McDowell & Calvin, 2015; McDowell & Popa, 2010), and response allocation on concurrent ratio schedules (McDowell & Klapes, 2018). Most recently, the theory has been extended to behavior on concurrent schedules where punishment is superimposed on one or both alternatives (McDowell & Klapes, 2019), and to behavior on single schedules where background reinforcement is varied (McDowell & Klapes, 2020). In these extensions of the theory, the behavior of AOs animated by the theory was found to be indistinguishable from the behavior of live organisms.

Because the ETBD generates behavior in AOs, it is not limited to providing static descriptions of steady-state behavior, like those provided by Equations 2 and 6. The theory permits the study of behavior dynamics as well, that is, how behavior changes with time. One example of this is Kulubekova and McDowell’s (2013) use of the theory to replicate Davison and Baum’s (2000) study of dynamic changes in pigeons’ key pecking. Davison and Baum reported detailed, systematic, small timescale, changes in the pigeons’ key pecking during unsignaled presentations of different concurrent schedules in single sessions. These findings were replicated by Kulubekova and McDowell’s AOs with remarkable fidelity. Another example is Chi’s (2019) application of the theory to replicate Corrado et al.’s (2005) study of dynamic changes in the macrosaccadic eye movements of monkeys. Corrado et al. arranged concurrent schedules of fixation on visual targets where the scheduled reinforcement rate ratios changed abruptly and unpredictably during sessions. They found that the monkeys’ response rate ratios tracked the changes in reinforcement rate ratios with a short lag. Chi reported the same result for AOs animated by the evolutionary theory.

Applied Implications of the Evolutionary Theory

It is possible to use the evolutionary theory to create an AO that exhibits at least some forms of psychopathology. If the theory is correct, and if the method of establishing the psychopathology is reasonable, then it should be possible to learn about the psychopathology by studying the AO’s behavior. It should also be possible to test therapeutic interventions by applying them to the AO’s behavior.

Most of this work is just getting started, but a fairly advanced example is provided by research on ADHD. Recall, as discussed earlier, that the a parameter estimated from fits of Equation 2 to concurrent schedule data is often lower for ADHD-diagnosed children than for children without the diagnosis, and that this indicates a lower sensitivity to reinforcement in the ADHD-diagnosed children. Increasing the mutation rate for AOs animated by the theory has the same effect, that is, it reduces the estimates of the a parameter (McDowell et al., 2008; McDowell & Popa, 2010). Hence, AOs with elevated mutation rates are possible models of ADHD.

To test this possibility, Popa (2013) conducted an extensive study of the effects of high mutation rates on the behavior of AOs animated by the evolutionary theory. He found that they had higher changeover frequencies on concurrent schedules, lower rates of target responding, and obtained fewer reinforcers than AOs with more moderate mutation rates. Popa also reported that at higher mutation rates, AOs

. . . emitted fewer bouts [of target behavior], of shorter length, and took much longer to re-engage in sustained behavior once a bout was terminated. They exhibited much smaller proportions of sustained behavior, their target behavior being sporadic and disorganized. The temporal disorganization was accompanied by abrupt topographical changes [i.e., in phenotype]. (p. 42)

Popa then provided evidence that these detailed dynamic features of the behavior of AOs with elevated mutation rates also characterize the behavior of children with ADHD. He concluded that

[t]he resemblance between the behavioral constellations of children that received an ADHD diagnostic (sic) and those of virtual organisms [AOs] characterized by high mutation rates is striking. The fact that this entire constellation emerged freely, unguided, from the reiterations of Darwinian processes was even more remarkable. (pp. 42–43)

Popa (2013) also investigated treatments for the dysfunctional behavior of AOs with high mutation rates. He found that increasing the rate or magnitude of reinforcement resulted in improvements in almost all the dysfunctional features of their behavior, including an increase in sensitivity to reinforcement as reflected in a larger a parameter. As Popa noted, children with ADHD are known to respond to these types of interventions in the same way. Although many behavioral details of ADHD and of how the disorder responds to treatment were known before Popa’s application of the evolutionary theory, his work constitutes a proof of concept, namely, that it is possible to use AOs animated by the ETBD to model psychopathological behavior and evaluate treatment interventions.

A second example of the applied relevance of the ETBD is provided by McDowell and Caron’s (2010b) discussion of their finding that the a parameter in Equation 2 for boys with anti-social behavior decreased with increasing child deviance. It was suggested earlier that this finding may indicate that the behavior of boys with higher deviance scores is less sensitive to social reinforcement than the behavior of boys with lower deviance scores, and that this may be the result of deviancy training. Put another way, social reinforcement may be less valuable (i.e., of lower magnitude) the more deviant the boy’s behavior. But McDowell and Caron noted that for AOs animated by the ETBD, the a parameter varies as a function of both reinforcer magnitude and degree of impulsiveness (generated by a higher mutation rate). As a result, the theory suggests that McDowell and Caron’s finding regarding the a parameter might be due to a more general impairment in impulsivity in the boys with anti-social behavior, rather than to a specific reduction in the value of social reinforcement. McDowell and Caron noted that this suggestion could be tested in boys with anti-social behavior by arranging concurrent schedules that deliver points or money, rather than social approval, as reinforcers. Finding that the a parameter for these schedules decreased with child-deviance scores would be consistent with the explanation of a more general impairment in impulsivity. On the other hand, finding that the a parameter did not decrease with child-deviance scores in these schedules would support the conclusion that social reinforcers are less valued than points or money the greater the boy’s deviance score. The treatment implications of the findings of such an experiment are obvious; if a decreased with child-deviance score, then nonsocial reinforcers could not be expected to control the deviant behavior any better than social approval.

A final example of the clinical applicability of the ETBD is provided by Morris and McDowell (this issue, pp. xx–xx), who modeled different subtypes of automatically reinforced self-injurious behavior (ASIB) using AOs animated by the theory. ASIB is behavior that does not appear to be supported by socially mediated reinforcers. In Subtype 1 of ASIB, self-injurious behavior occurs at a high rate when the individual is alone, but at a lower rate when background reinforcement is present (e.g., in a play condition). In Subtype 2 of ASIB the self-injurious behavior occurs at a high rate regardless of background reinforcement. Proposed causes for the different subtypes include a larger magnitude of automatic reinforcement or a lower sensitivity to environmental changes for Subtype 2 than for Subtype 1. Treatment for Subtype 1 typically entails providing background reinforcement. Treatment for Subtype 2 often requires implementing a method of response suppression, such as restraint or punishment.

Morris and McDowell (this issue, pp. xx–xx) modeled the ASIB subtypes and their possible causes in AOs animated by the evolutionary theory. They arranged different reinforcer magnitudes, different sensitivities to environmental consequences (i.e., different mutation rates), and different reinforcement contexts (i.e., rates of background reinforcement) for the AOs. They found that AOs that had relatively low magnitudes of automatic reinforcement and relatively high sensitivity to environmental consequences accurately reproduced Subtype 1 of ASIB. They also found that AOs with relatively high magnitudes of automatic reinforcement and relatively low sensitivities to environmental consequences accurately reproduced Subtype 2. Morris and McDowell then tested these models by applying treatments that are known to affect them in humans. For Subtype 1 they found that increasing the background reinforcement was sufficient to decrease the rate of ASIB. For Subtype 2, they found that increasing the background reinforcement decreased the ASIB by a small amount, and that adding a punishment contingency decreased it substantially. These results are the same as those obtained with humans, which suggests that Morris and McDowell’s models of ASIB Subtypes are reasonable. Like Popa’s (2013) work, this research is a proof of concept, namely, that the ETBD can be used to model psychopathological behavior and to study treatment interventions for it.

Conclusion

The ETBD is a fairly recent development in research on choice. It generates empirical matching (Equation 2) and the empirical remains of matching theory (Equation 6) as emergent outcomes, which means that it provides a theoretical foundation for these equations, namely, they are the result of evolutionary dynamics. The ETBD then goes beyond these steady state descriptions of behavior to consider behavior dynamics in artificial organisms animated by the theory. As illustrated in this article, the steady state equations and the behavior of AOs animated by the theory can contribute to the understanding and treatment of psychopathological behavior in humans in important ways.

Future research in this area could go in many directions. One obvious direction is to study the clinical relevance of the ETBD’s recently developed account of punishment. For example, how does punishment affect the behavior of AOs that exhibit characteristics of ADHD or anti-social behavior? The results of this research would generate a priori predictions of the ETBD, which could then be tested in live experiments with human participants. Some early research on the effect of punishment on the behavior of children with ADHD has been reported by Furukawa et al. (2019).

Additional basic research with the ETBD is also important. For example, it is possible to obtain new equations that describe steady-state behavior by studying the behavior of AOs animated by the theory. This is accomplished by obtaining reliable steady-state data from the AOs, and then finding an equation that describes it accurately and without residual error. This type of work is currently under way for AOs working on single schedules where both the rate and magnitude of reinforcement are varied. Farther afield, one may ask how an AO forms a discrimination, how stimulus control might be implemented in the ETBD, how the theory handles conditioned reinforcement in a chained schedule, and so on. Possibilities for the last two phenomena have been considered by McDowell et al. (2006). These phenomena, and of course many more, are important to the basic science, and understanding them in the context of the ETBD seems likely to have implications for clinical practice.

It would also be a welcome development to see more of this applied and basic work on the ETBD undertaken by researchers in other laboratories (e.g., Berardi et al., 2018; Li et al., 2018). This could include tests of the theory’s many qualitative and quantitative predictions (McDowell, 2019; McDowell & Klapes, 2019), as well as further development of the theory itself. A particularly interesting example of the latter is the study of AOs navigating two-dimensional grid worlds (McDowell, 2019; McDowell et al., 2006), which are often used in artificial life and artificial intelligence research to study how agents learn about the spatial location of resources in two dimensions. Successful navigation of grid worlds by AOs would be a first step toward animating mechanical agents with the ETBD. The basic operation of such agents could be simple; to successfully navigate grid worlds, for example, they would only have to be capable of moving forward, turning, and sensing their location (e.g., https://www.youtube.com/watch?v=qtT0HfgJ28s).

Declarations

This research complies with ethical standards