Competing stimulus assessments: A systematic review

Abstract

The current review summarizes the literature on competing stimulus assessments (CSAs). CSAs are pretreatment assessments designed to systematically identify stimuli that reduce problem behavior (PB), ostensibly through reinforcer competition or substitution. We report on the participant characteristics, outcomes, and predictive validity of published CSAs that included (a) no-stimulus control trial(s), (b) test trials during which each stimulus was available singly and noncontingently, and (c) measurement of PB and stimulus engagement or contact. Results showed that CSAs have broad utility across a variety of topographies and functions of PB. In the majority of CSA applications for which extended analyses, or validations, were performed, stimuli shown to reduce PB during the CSA produced similar reductions during extended analysis. This was the case regardless of topography or function of PB, or whether the stimuli were assumed to be “matched” to the stimulation thought to be produced by PB. Implications for future research are discussed.

Competing stimulus assessments (CSAs) are pretreatment assessments designed to identify stimuli that, when made freely available, are associated with reductions in problem behavior (PB), which presumably occur as a function of reinforcer competition or substitution (Ahearn et al., 2005; Fisher et al., 2000; Hagopian et al., 2005; Shore et al., 1997).¹ Stimuli associated with reductions in PB during the CSA are then provided noncontingently, or freely, during treatment (e.g., Phillips et al., 2017). CSA methodologies evolved from preference assessments, and, although their application is not limited to automatically maintained PB, they owe their development to conceptual and methodological advances in its treatment.

Perhaps the most crucial of these conceptual advancements involved application of the principles of reinforcer competition (e.g., Carr & Kologinsky, 1983; Favell et al., 1982; Horner, 1980; Rincover et al., 1979). Favell et al. (1982) exemplified one of the earliest and clearest examples of this approach. The researchers provided noncontingent access to stimuli (e.g., food, mirrors) that presumably would produce the “oral stimulation” assumed to be maintaining pica and the “visual stimulation” assumed to be maintaining eye poking. Reductions in PB were observed, ostensibly as a function of reinforcer competition or substitution. Although the concept of selecting stimuli based on the principles of reinforcer competition and substitution (i.e., based on the hypothetical stimulation produced by PB and alternative responses) demonstrated a conceptual leap in the treatment of automatically maintained PB, the process of identifying stimuli to be used during treatment remained informal.

An important methodological advancement occurred with the use of stimulus preference assessments (e.g., Fisher et al., 1992; Pace et al., 1985) to systematically identify stimuli to be delivered noncontingently during treatment (e.g., Vollmer et al., 1994). For instance, Piazza et al. (1996) modified a single-stimulus preference assessment (Pace et al., 1985) by recording the frequency of PB while each test stimulus was freely available; stimuli associated with low levels of PB were made freely available during subsequent treatment. Shore et al. (1997) used a similar procedure to identify stimuli to be made noncontingently available during extended analysis. The logic of these and similar preference assessment applications was rooted in the principle of reinforcer competition, but the method did not identify reductive effects of stimuli relative to their absence. Reductions in PB were compared across stimuli (test vs. test conditions) rather than across stimulus and no-stimulus conditions (test vs. control). Thus, a methodological leap was made with the addition of a no-stimulus control trial, which allowed for quantification of relative changes in PB across conditions in which the stimuli were present and absent (i.e., the extent of reinforcer competition).

Ringdahl et al. (1997) identified stimuli to be evaluated in the treatment of automatically maintained PB using a free-operant preference assessment that included a no-stimulus control condition as well as measures of PB and engagement. The inclusion of the control condition allowed the authors to determine the extent to which PB was reduced as a function of access to the stimuli. However, multiple stimuli were concurrently available during the test conditions, which made it impossible to determine the reductive effects of each stimulus. The authors selected the stimuli to be evaluated in subsequent treatment based on relative preference (i.e., duration of engagement relative to other stimuli) rather than reduction in PB relative to the control. Interestingly, at least one stimulus was preferred (i.e., associated with high engagement) in all three assessments; yet, two of four PB topographies occurred at similar levels in the presence and absence of those stimuli. That is, despite high engagement with at least one stimulus in the test condition, half of the PB topographies were not reduced relative to the control. Perhaps unsurprisingly, noncontingent access to stimuli that did not reduce PB during the assessment did not reduce those topographies during treatment. These results demonstrate that measures of preference (e.g., engagement) do not always reflect the extent of PB reduction. They also suggest that it might be necessary to conduct isolated test trials in order to identify which stimuli from a larger set are differentially associated with PB reductions.

In an extension of Piazza et al. (1996), Piazza et al. (1998), included a no-stimulus control trial during modified single-stimulus preference assessments that included measures of both PB and engagement. In hindsight, this method can be credited as the first CSA because it identified specific stimuli that were associated with reductions in PB in their presence relative to their absence. Stimuli associated with reductions in PB relative to the control during the CSA were subsequently effective in reducing PB during extended analyses, yet stimuli associated with high levels of engagement (i.e., preferred stimuli) during the CSA were not always associated with a concomitant reduction in PB. Together, the findings of Ringdahl et al. (1997) and Piazza et al. (1998) underscore the fact that stimulus preference and reinforcer competition are not equivalent. Thus, assessments aimed at identifying either must be procedurally and terminologically distinct.

Procedurally, preference assessments and CSAs share some similarities. Both are pretreatment stimulus assessments employed in preparation for intervention. Unlike pretreatment behavioral assessments designed to identify controlling variables of behavior (e.g., functional analyses of PB), preference assessments and CSAs examine the effects of stimuli on behavior during relatively brief periods of time. However, preference assessments and CSAs differ in ways that reflect their distinct purposes. As their name implies, preference assessments reveal preference for stimuli, broadly defined as the extent to which individuals will select or engage with available stimuli. Although reinforcement contingencies are not evaluated during preference assessments, preference has been shown to be predictive of reinforcer efficacy in subsequent reinforcer assessments (e.g., Fisher et al., 1992; Hagopian et al., 2001). Thus, preference assessments empirically identify stimuli that are likely to increase a target response when delivered contingently (i.e., reinforcers). In contrast, CSAs assess the extent to which the noncontingent availability of a stimulus reduces or competes with PB, relative to when no stimuli are available. Because the reinforcers for PB are concurrently available during both the test and control trials of the CSA, they identify stimuli that are likely to compete with PB when delivered non-contingently during treatment, despite the concurrent availability of reinforcement for PB.

Over the last two decades, CSAs have become a frequently used pretreatment assessment for a variety of topographies of automatically maintained PB (Gover et al., 2019; Rooker et al., 2018), and their use has been extended to the treatment of socially maintained PB. These CSAs are similar to those of automatically maintained PB, except social reinforcers are provided contingent on PB during all test and control trials. During treatment, competing stimuli are delivered non-contingently when the social reinforcer for PB is unavailable (Fisher et al., 2004; Hagopian et al., 2005; Rooker et al., 2013).

Additionally, recent reviews indicate that treatments involving noncontingent reinforcement for automatically maintained PB are more effective when stimuli are identified by a CSA than when identified by a preference assessment (e.g., Gover et al., 2019; Rooker et al., 2018). Yet, this finding also indicates that preference assessments continue to be used to identify stimuli to be delivered noncontingently during PB treatment. The reliance on preference assessment methodologies to identify competing stimuli may be partially due to the fact that preference assessment methodologies have been the focus of several empirical studies (e.g., DeLeon & Iwata, 1996; Fisher et al., 1992) and reviews (e.g., Cannella et al., 2005; Tullis et al., 2011), whereas seemingly few studies have been devoted exclusively to CSA methodology. Indeed, it appears that most published CSAs to date have been in studies that reported data obtained during clinical service delivery, as opposed to studies that answered experimental questions related to the methodology of the CSA itself. Thus, differences in the conceptualization, methodology, and terminology used to describe published CSAs may have muddled the relative contributions of the aforementioned studies.

The purpose of the current review was to provide a quantitative summary of published CSAs in order to clarify the literature and provide focused directions for future research. As an attempt at the former, we reviewed only published CSAs for which the extent of PB reduction relative to a no-stimulus control trial (s) could be quantified for each stimulus assessed (i.e., the most methodologically stringent applications). We identified the populations with whom CSAs have been implemented. In doing so, we examined relations between PB reductions and individual characteristics (e.g., topography or function of PB), stimulus characteristics (e.g., type, number assessed), CSA parameters (e.g., trial duration), and measures of engagement or contact with stimuli. Finally, we examined the predictive validity of CSA results, or correspondence between PB reduction observed in the CSA and that observed under similar conditions during extended analysis.

Method

Article Search and Filter

In the current study, the level of analysis was individual datasets. Thus, the Preferred Reporting Items for Systematic Reviews and Meta-analyses Individual Participant Data (PRISMA-IPD; Stewart et al., 2015) checklist was used to document the search process, though not all categories were applicable to the current review.² The electronic databases, hosts, and search terms used are detailed in Table S1 of the supporting information (available online). To briefly summarize, seven electronic databases were searched for the following terms, as well as grammatical variations and synonyms thereof: (a) competing stimulus/items assessment AND behavior or (b) preference assessment AND PB. When possible, database searches were filtered to return only human-subject experiments. No restrictions were placed on year of publication, language, or peer-review status. Subsequently, the reference lists of the studies identified for inclusion were added to the search results, and Google Scholar³ was searched for the following phrases, with quotations around each phrase: competing stimulus assessment, competing items assessment, and competing item assessment.

EndNote™ X8.2 was used to create a database of search results. Covidence systematic review software (Veritas Health Innovation, Australia, available at www.covidence.org) was used to identify duplicates and filter the search results from the database.

Filtering of Search Results

The search results were filtered in two steps: screening and full-text review. During screening, search results were deemed eligible for full-text review if the title and/or abstract indicated it was a peer-reviewed, human-subjects experiment in which PB was a dependent variable. Search results were excluded during screening if the title and/or abstract indicated (a) it was not an experiment (e.g., review, book chapter), (b) the research subjects were nonhuman, and/or (c) PB was not a dependent variable. If it was unclear whether the in/exclusion criteria were met, the study was identified for full-text review.

During full-text review, the figures, tables, method, and results sections of each study were reviewed for the CSA individual data inclusion criteria (defined below). Studies were included if at least one participant’s assessment data met the CSA inclusion criteria (described below). Studies were excluded if (a) a pretreatment stimulus assessment that met our definition of a CSA was not conducted with at least one participant or (b) at least one CSA that met our criteria was conducted but there was no graph or data table of the results. Studies were also excluded if the full text was unavailable through institutional access.

Participant Data Inclusion Criteria

CSA

Individual participant datasets were included if their pretreatment stimulus assessment included: (a) no-stimulus control trial(s), (b) a series of isolated test trials during which a single stimulus was available (i.e., one stimulus at a time), (c) a measure of PB during all stimulus test and control trials, and (d) a measure of item engagement or contact during all test trials. Criteria (a), (b), and (c) were necessary for quantifying the reductive effects of each stimulus assessed on PB or the extent of reinforcer competition. Criterion (d) was necessary for examining the relation between percentage of PB reduction and percentage engagement or contact for each stimulus assessed. No datasets met criteria (a), (b), and (c) without meeting criterion (d). Only four participant datasets (from Ringdahl et al., 1997) met all criteria except criterion (b) (isolated test trials).

When it was determined that a participant’s CSA included all of the aforementioned CSA components, it was further required that measures of PB and engagement or contact were (a) separately reported and (b) reliable (i.e., inter-observer agreement was obtained on these measures for at least 30% of sessions and coefficients were above 80%). The former was required for data analysis and the latter was a quality standard that would be required of any study published in the Journal of Applied Behavior Analysis. Participant datasets were excluded if the measure did not meet these criteria.⁴ In several included studies, the CSA method and results summaries were not uniform across participants; thus, individual participant data were excluded if their CSA and/or its dependent variables did not meet all of the aforementioned criteria.

CSA Validation

One purpose of this review was to examine whether the reductions in PB observed during the CSA would be replicated in an extended analysis (i.e., for longer durations, across multiple observations). In many cases, the results of the CSA were evaluated during a subsequent extended treatment analysis. Although not all of the extended analyses were treatment analyses per se, for ease of description, we will use the term “treatment” to refer to the test condition of the extended analysis. Extended analyses were deemed CSA validations if: (a) the contingencies during the baseline and at least one treatment condition were identical to those in the CSA control and test conditions, respectively, and (b) a time-series graph of the extended analysis was available. Extended analyses were excluded if the baseline/treatment(s) included contingencies or stimuli that were not present or tested in the CSA control or test conditions. When more than one stimulus set was tested under conditions meeting these criteria, it was considered to be a separate CSA validation.

Coding Individual Participant Data

Participant Characteristics

The total number of participants in each included study was recorded, as was their demographic information (name, sex, age). The topographies and function of each participant’s PB were documented as described by the authors of the included studies (hereafter “the authors”).⁵

Number of CSA Applications and CSA Validations

In several cases, multiple CSAs and/or CSA validations were conducted with the same participant. CSA applications and validations were distinguished by the use of different independent and/or dependent variables (e.g., different stimulus sets were assessed or multiple PB measures were reported).

CSA Method

The following information was recorded for each included CSA application: the process by which stimuli were selected to be assessed, or basis for stimulus inclusion (interview, direct observation, or preference assessment); number and types of stimuli assessed (food, leisure, or attention); whether engagement or contact with the stimulus was measured; trial duration; number of series conducted (i.e., number of times the test and control trials were repeated); and the basis for selecting stimuli to be evaluated in extended analysis (by percentage engagement, percentage reduction in PB, and/or feasibility and safety in the treatment setting).

Stimulus Characteristics

Each stimulus assessed in the CSA applications was recorded and categorized as food, leisure stimuli, or attention (social interaction). Some authors hypothesized that the stimuli tested were “matched” (or “unmatched”) to the function of automatically maintained PB, meaning that engagement with the stimuli was thought to produce the same (or different) form of sensory stimulation produced by the PB. When the authors designated all or a subset of stimuli assessed as “matched/unmatched” to the hypothetical sensory properties produced by PB, those stimuli were included in the “matched/unmatched” data analysis. When the authors did not mention sensory properties or “matched” stimulation, those stimuli were excluded from the “matched/unmatched” data analysis.

Data Analysis

Data Extraction

GetData® Graph Digitizer (Version 2.26) (Fedorov, 2013) was used to extract data values from the included CSA and CSA validation graphs. Procedures were identical to those described by Hagopian et al. (2017, pp. 50-51), with the following exception: For CSA graphs, the values generated by GetData (GetData values, GDVs) were not rounded to the nearest possible value given the session duration because the data points represented the average of several trials. When CSA results were presented in a table, these values were used in lieu of GDVs. When textual data were available (e.g., average rates were summarized in the results), the GDVs were compared to the text, which validated the GDVs. In one case, the GDV was not validated by the text; however, it was clear that the text was incorrect given the values on the graph, thus the GDV was used.

For the CSA validations, the values of the last five data points of the first baseline condition and the last five data points of the relevant treatment condition (i.e., that in which the contingencies were identical to those of the CSA test condition) were extracted. This method has been used in other studies to summarize treatment analysis outcomes across multiple cases (see Greer et al., 2016; Hagopian et al., 1998; Rooker et al., 2013). If either condition had fewer than five data points, all and only those data points were extracted and averaged.

PB Reduction

Percentage reduction in PB was calculated for the CSAs (from the control to each test condition) and CSA validations (from the baseline to each included treatment condition) in the same manner: Mean responding during the test/treatment condition was subtracted from mean responding during the control/baseline condition, divided by the mean of the control/baseline condition, and multiplied by 100%.

For the CSAs, percentage reduction in PB was compared to the percentage engagement or contact during each stimulus test trial. It was also compared across stimuli the authors characterized as “matched/unmatched,” stimulus type (food vs. nonfood), and functional classes of PB. For each CSA application, we also identified the percentage of stimuli assessed that produced a greater than 80% reduction in PB, which is a commonly used criterion of treatment efficacy in reviews of behavioral research (e.g., Gover et al., 2019; Kahng et al., 2002; Kurtz et al., 2003; Rooker et al., 2013). Both percentages were compared to the variables reported in Tables 1 and 2 (i.e., participant characteristics and CSA method, respectively).

Table 1.

Participant Characteristics

First Author, Year	Participant	Age	Problem Behavior (PB)	Function of PB
Clay, 2018	Molly	12	Skin picking	Auto
DeLeon, 2005a	Ted D.	10	Agg, Pseudoseizures	Attn
			Dis, SIB	Auto
DeLeon, 2005b	James D.	10	Saliva play, Spitting	Auto
DeLeon, 2005b	Read	6	Pica	Auto
Fisher, 2004	Carl	7	Agg, SIB	Attn
Fisher, 2004	Sally	33	Agg, Dis, SIB	Attn
Hagopian, 2005	James H.	12	Agg, Dis, SIB	Tang
Hagopian, 2005	Matt	7	Agg	Attn, Tang
Hagopian, 2005	Stephen	13	Agg, Dis, SIB	Attn
Hagopian, 2009	Sissy	10	Body tensing	Auto
Hagopian, 2011	Lenney	19	Pica	Auto
Hagopian, 2011	Stephanie	13	Pica	Auto
Higgins, 2012	Malik	5	Head weaving	Auto
Jennett, 2011	Abigail	3	SIB	Auto
Long, 2005	Janelle	5	SIB, Tongue biting	Esc
Long, 2005	Marsha	19	SIB	Auto
Piazza, 1998	Brandy	17	Pica	Auto
Piazza, 1998	Tad	5	Pica	Attn, Auto
Piazza, 2000	Betsy	6	Dangerous behavior	Auto
Piazza, 2000	Tyrone	17	Hand mouthing	Auto
Newcomb, 2018	Ted N.	13	Agg	Attn
Roane, 2002c	Jonah	4	Agg, Dis, Dangerous acts	Attn
Roane, 2002d	Chad	22	Vocal tics	Auto

Note. Participants with the same pseudonym are distinguished by the first initial of the first author of the study in which their data were published. Auto = automatic reinforcement; Agg = aggression; Dis = disruption; SIB = self-injurious behavior; Attn = attention; Tang = tangible; Esc = escape.

^aDeLeon, Toole, et al. (2005)

^bDeleon, Uy, & Gutshall (2005)

^cRoane, Fisher et al. (2002)

^dRoane, Piazza et al. (2002)

Table 2.

Number of CSA Applications and CSA Validations that Met Inclusion Criteria

Name	CSA Applications (Distinction)	CSA Validations (Distinction)
Molly	1
Ted D.	2 (PB)
James D.	2 (SS)	2 (SS)
Read	1*	2 (SS)
Carl	1
Sally	1
James H.	1
Matt	1
Stephen	1
Sissy	1
Lenney	1	1
Stephanie	1	1
Malik	1	1
Abigail	1*	1*
Janelle	1
Marsha	1	1
Brandy	2 (SS)	4 (SS)
Tad	1
Betsy	1	2 (SS)
Tyrone	1	2 (SS)
Ted N.	1
Jonah	1
Chad	1

Note. When more than one CSA application/validation was conducted, the distinction of the CSA applications or validations is noted in parentheses. Blank cells indicate a validation was not conducted. PB = problem behavior; SS = stimulus sets.

^*One or more datasets were excluded due to unavailable data or the inclusion of contingencies that rendered the data incomparable across participants.

Predictive Validity

The predictive validity of the CSA results was evaluated by comparing correspondence between the percentage reduction in PB produced by a given stimulus set during the CSA and CSA validation (extended analysis). To evaluate the predictive validity of the CSA results (i.e., correspondence between CSA results and those of extended analysis), the percentage reduction in PB obtained during the CSA validation was subtracted from that obtained during the CSA. When more than one stimulus was evaluated in treatment, the highest percentage reduction in the CSA was used. Correspondence was defined as a difference of 10 or fewer percentage points.

Interrater Agreement

A second, independent rater filtered 32% (n = 150) of the search results, which were selected using a random number generator. An agreement was scored if the first author and the second rater included or excluded the reference. Interrater agreement (IRA) was calculated by dividing total agreements by the total agreements plus disagreements. The IRA on search filtering was 100%.

A second rater independently verified that each included CSA application and validation dataset met the participant data inclusion criteria (above). The second rater was provided with a spreadsheet containing the information described under Coding Individual Participant Data (above) in order to verify the accuracy of the information by comparing it to the published record. Because the information reported herein is publicly available, independent ratings were not obtained. Information the second rater deemed inaccurate was flagged and reviewed by the first author and, if there was still a discrepancy, a third rater.

Results and Discussion

Article Search and Filter

The results of the search are displayed in Figure 1. In short, 734 references were identified, 270 of which were duplicates; thus, 472 references were screened. During title and abstract screening, 277 references were excluded; thus, 195 full texts were reviewed. A total of 179 were excluded from the full-text review; 38 of these studies met all criteria except they did not contain a measure of PB, a no-stimulus control condition, and/or isolated test trials. Thus, 15 studies, published in seven different journals, were identified for inclusion. There was a total of 28 participants in these studies, but five of the participants were excluded due to the absence of a control condition in their individual pretreatment stimulus assessment. Thus, data from 23 participants were included.

Figure 1.

Modified PRISMA-Individual Participant Data (IPD) Flow Diagram

A variety of terms have been used to describe assessments intended to identify stimuli that compete with PB, thus we used a variety of terms in our search. However, this produced a large proportion of irrelevant results. Most terminological variations occurred in the excluded studies. With only one exception (Roane, Fisher et al., 2002), all of the included studies that were published after Fisher et al. (2004) used the term “Competing Stimulus Assessment.” Although very few of the excluded studies used the term CSA, many used terms that were similar (e.g., “competing items assessment,” “competing sensory analysis,” “free-operant competing stimulus assessment”). Behavior analysts have long emphasized the use of technical scientific terminology, as precision is necessary for effective scientific communication (Carr & Briggs, 2011; Schlinger et al., 1991; Skinner, 1957). Inconsistencies in terminology limit the application and refinement of the technology in question, breed confusion in practice, and preclude or impede synthesis of published data. In the interest of maintaining technicality in procedural naming (e.g., Jarmolowicz, & Lattal, 2010; LaFrance & Tarbox, 2020), it would be advantageous for researchers and clinicians to use the term “CSA” only when a no-stimulus control condition is used to evaluate the extent of PB reduction. In the absence of a no-stimulus control condition or trial, the extent of reinforcer competition under the stimulus conditions of the assessment will not be objectively definable (i.e., reductions in PB as a function of access to the test stimulus cannot be quantified).

Participant Characteristics

Participant characteristics are shown in Table 1. The 23 included participants ranged in age from 3 to 33 years (M = 11.6), 41% (n = 9) were female, and 83% (n = 19) were under the age of 18. All participants were reportedly diagnosed with one or more neurodevelopmental disorders (e.g., autism spectrum disorder, attention-deficit hyperactivity disorder, intellectual disability) or genetic disorders (e.g., Duane syndrome, Cornelia de Lange syndrome), and all but two participants were reported to have several diagnoses, both psychiatric and medical.

Participants engaged in multiple topographies and functions of PB. There were 15 CSA applications in which automatically maintained PB was measured, nine in which socially maintained PB was measured, and two in which “mixed” PB was measured. We categorized PB as a “mixed” function when multiple PB topographies reinforced by separate variables were aggregated into a single measure (Ted, from DeLeon, Uy, & Gutshall, 2005) or two separate functions were indicated over the course of the study (Tad, from Piazza et al., 1998). Hereafter, participants in these groups will be referred to as the automatic, social, and mixed groups.

Despite the variation in individual characteristics, the participants comprise a relatively homogenous sample of individuals with neurodevelopmental disabilities who engage in PB. This is a common limitation of the research on behavioral technologies at present. Future researchers should seek to refine CSA methodologies and demonstrate their utility with more diverse populations, for treatment of behavior disorders or forms of PB that are not typically characterized as such (e.g., feeding disorders, obsessive behavior, smoking, anxiety, depression).

Number of CSA Applications and CSA Validations

A total of 26 CSA applications and 17 CSA validations met our criteria. Table 2 shows the number of CSA applications and validations—and the distinctions thereof—that were included for each participant. Six participants had more than one CSA application: For Ted, one CSA was conducted, but two dependent variables (PB) were measured; for James D., Brandy, Tad, and Tyrone, two CSAs were conducted in which the stimuli differed. Abigail had three CSA applications that met inclusion criteria, but only the first was included because the contingencies in the subsequent applications rendered the PB and engagement data incomparable to the other included CSA applications. Read had two CSA applications, but data from the first were not published and thus excluded.

Seventeen CSA validations (conducted with 10 participants, described in four studies) met our inclusion criteria. In four CSA validations, the stimulus set tested during treatment did not produce reductions in PB during the CSA but was tested during treatment as a form of control (e.g., competing vs. noncompeting stimuli) or to confirm a hypothesis (e.g., “matched” vs. “unmatched” stimuli). In many of the excluded extended analyses, competing stimuli were used to supplement other treatment components (e.g., differential reinforcement of alternative behavior). However, in the absence of a treatment condition that was identical to the test condition of the CSA (i.e., one in which competing stimuli were noncontingently available), it was impossible to compare the reductions observed in the CSA to those observed in the extended analysis (i.e., impossible to isolate the reductive effects of competing stimuli alone).

CSA Method

Table 3 describes the method of each included CSA application. The process by which stimuli were identified for inclusion in the CSA varied across studies. These included interviews, direct observation preference assessments, and/or feasibility of use/safety in the treatment setting. In three studies, the authors did not specify the basis for stimulus inclusion.

Table 3.

Parameters of Each CSA Application

Participant (Application)	Selection Basis	# of Stimuli	Stimulus Type	E/C	Trial Duration	# of Series	Validation Criteria
Molly	I	8	L	E	5	3	E, PB
Ted D. (1, 2)	NS	11	L	E**	5	3	E, PB*
James D. (1)	PA	3	L	E	15	4**	E, PB
James D. (2)	PA	14	L	C	7	4	PB
Read	PA	6	F, L	E	8	4	PB
Carl	I, PA	11	A, L	E*	3	3	E, PB
Sally	I, PA	11	A, L	E*	4	3	E, PB
James	I	16	A, L	E	2	2	PB
Matt	I	13	L	E	5	3	PB
Stephen	I	9	L	E	5	4	PB
Sissy	NS	15	L	E	5	2	E, PB
Lenney	I, PA	12	F, L	C	NS	4	C, PB
Stephanie	I, PA	12	L	C	NS	4	C, PB
Malik	I	5	L	E	3	1	PB
Abigail	NS	15	L	C	2	1	PB
Janelle	I	12	L	C	10	1	PB
Marsha	I	15	L	C	6	4	PB
Brandy (1)	I, D, H	20	F, L	E	5	1	E, PB, S
Brandy (2)	I, D, H	7	L	E	5	1	E, PB, S
Tad	I, D, H	17	A, L	E	5	1	E, PB, S
Betsy	I, D, H	12	L	E	5	1	E, PB, S
Tyrone	I, D, H	25	F, L	E	2	3	E, PB, S
Ted N.	I, F	4	A, L	E*	3	3	S*
Jonah	I	2	L	E	3	3	E, PB
Chad	I, H	29	F, L	E	2	3	E, PB*

Note. I = indirect assessment, NS = not specified, PA = preference assessment, D = direct observation, H = hypothesized sensory properties, L = leisure, F = food, A = attention, E = engagement, C = contact, PB = problem behavior reduction, S = safety/feasibility.

^*Indicates a missing or ambiguous operational definition.

^**Indicates data were available from only the first series of the CSA.

The number of stimuli assessed per CSA ranged from four to 29 and a variety of stimuli were evaluated, including leisure items, attention, and food. In six CSA applications, contact was measured and defined as touching the (leisure) stimulus. In 16 CSA applications, engagement was measured and typically defined as engaging with the stimulus in a manner that was intended. However, in five applications (conducted with four participants), the operational definition of engagement was not provided. None of the applications that evaluated noncontingent attention defined “engagement” separately from engagement with leisure stimuli. In the applications that evaluated food, “engagement” was defined as consumption or ingestion (e.g., food passing the plane of the lips). In all but one application, food consumption was measured as percentage duration or percentage of stimuli consumed; thus, in all but one application, data on food consumption were comparable to percentage engagement or contact. Although operational definitions are often abbreviated in publications, the definitions of engagement and attention were so loosely described that replication would be impossible. Future researchers should consider providing more detailed operational definitions when reporting their CSA method.

Trial duration ranged from 2 to 15 min (median = 5), but in one study the trial duration was not stated. The number of series (i.e., number of times the test and control trials were repeated) ranged from one to four (median = 3).

In the majority of CSA applications, the authors defined the stimuli that would be evaluated in subsequent analysis as those that produced both “low/lower” levels of PB and “high/higher” engagement or contact. Some identified them based only on reductions in PB and others also required the stimuli to be safe and feasible to use in the treatment setting (though if the latter was not stated, it does not mean that those variables were not a factor). However, only two studies stated that the basis for stimulus evaluation was “low/lower” rates of PB relative to the no-stimulus control condition. Overall, the stimulus selection criteria varied and were not consistent with the purpose of evaluating PB relative to a control condition, which is to quantify the extent of PB reduction. In the future, it would be helpful for authors to specify the point of comparison when comparative adjectives (e.g., “lower”) are used and when reporting CSA results to quantify reductions in PB relative to the no-stimulus control condition.

Stimulus Characteristics

Across all applications, a total of 317 stimuli were tested. These stimuli included 261 leisure stimuli (e.g., toys), 51 food items, and five noncontingent attention (vocal or physical interaction). In one case (Stephanie) a food item (coffee) was categorized as a leisure activity because the participant smelled rather than ingested it. In all cases the stimuli were able to be categorized.

In eight CSA applications, the authors characterized all (Piazza et al., 1998; 2000) or a subset (Clay et al., 2018) of the stimuli assessed as “matched/unmatched” to the hypothetical sensory stimulation produced by automatically maintained PB. “Matched” stimuli were evaluated for pica and vocal stereotypy, and “unmatched” stimuli were evaluated for pica, vocal stereotypy, and skin picking. Never did the authors describe the process by which the stimuli were determined to be “matched/unmatched,” whether informal or systematic. In total, 73 “matched” and 37 “unmatched” stimuli were assessed. Notably, 49% (n = 36) of the “matched” stimuli and 0% of the “unmatched” stimuli were food.

Data Analysis

Figure 2 shows the relation between percentage reduction in PB and contact/engagement with each stimulus⁶ during the CSA, compared across groups. Negative values indicate that PB during the stimulus test conditions was higher than during the control. We did not identify a relation between the percentage of stimuli assessed that produced a greater than 80% reduction in PB, percentage engagement or contact, or percentage PB reduction and any of the variables reported in Tables 1 or 2. Those variables included age, topography of PB, trial duration, the basis for stimulus selection, types of stimuli assessed, whether engagement or contact was measured, and the number of series conducted.

Figure 2. Percentage Engagement or Contact and Percentage Reduction in Problem Behavior.

Note. Data are grouped by function of problem behavior. The dotted horizontal line indicates an 80% reduction in problem behavior, and the dotted vertical line indicates 80% engagement or contact.

The absence of an inverse relationship between PB and engagement or contact in the reviewed CSAs is counter to findings of some individual studies (e.g., Lindberg et al., 1999; Shore et al., 1997). We also evaluated the relation between PB reduction and engagement and contact separately, but no differential relations were identified, thus we reported engagement and contact as one category. The reason for this lack of relation is unclear, but it is reasonable to suspect it was a function of (a) operational definitions of engagement (e.g., engaging with an item in a way other than its intended usage), (b) whether PB and engagement could simultaneously occur (due to incompatibility or measurement as such), and/ or (c) nonmeasurement of other behavior (e.g., requests, unreported topographies of PB). It also remains unclear whether PB reduction and engagement or contact, in combination, are differentially predictive of treatment outcomes. We did not evaluate this possibility as engagement and contact data were not reported for every validation. As mentioned, it would be beneficial for future researchers to report more detailed operational definitions of engagement and it might be of value to report measures of non-PB (e.g., engagement, contact, mands, and nontargeted PB) or their aggregated values. Such information could be used to inform the selection of stimuli for use in treatment. Regardless, if the primary purpose of conducting a CSA is to identify stimuli that compete with PB, then selecting stimuli to use in treatment based on their reductive effects would be appropriate at this time.

Figure 3 shows the percentage of stimuli assessed that produced an 80% or greater reduction in PB for each application in the automatic, social, and mixed groups. For the automatic group, 40% (n = 191) of stimuli assessed produced reductions in PB greater than 80% and for the social group 80% (n = 91) of stimuli assessed produced an 80% or higher reduction in PB. The percentage of stimuli for which engagement or contact was 80% or greater was 43% (n = 191) for the automatic group and 48% for the social group (n = 91). The mixed group was composed of only two CSA applications, so it is of little value to report those outcomes as a group. In short, although the proportion of stimuli that produced high engagement was similar across groups, a higher proportion of stimuli reduced socially maintained PB greater than 80%. These results suggest that it might be easier to identify competing stimuli for socially maintained PB than for automatically maintained PB.

Figure 3. Percentage of Effective Competing Stimuli Grouped by Function of Problem Behavior.

Note. Effective competing stimuli are those that reduced PB by 80% or greater in the CSA.

Figure 4 shows the percentage reduction in PB during the CSA test trials that included stimuli the authors characterized as “matched” or “unmatched” to the hypothesized reinforcement of PB. As previously mentioned, there were differences in the types of “matched” or “unmatched” stimuli evaluated. Thus, in Figure 4, the stimuli are subcategorized as food or nonfood items (i.e., leisure or attention). Nearly 64% of the 36 “matched” food stimuli produced an 80% or greater reduction in PB (no “unmatched” food stimuli were evaluated). For nonfood stimuli, 38% of the 13 “matched” and 43% of the 37 “unmatched” nonfood stimuli produced an 80% or greater reduction in PB. In other words, there was no apparent difference between nonfood stimuli characterized as “matched” or “unmatched” by study authors. Although we did not review all published studies that evaluated “matched” and “unmatched” stimuli, these results raise questions about the process of selecting stimuli based on largely untestable hypotheses.

Figure 4. Percentage Reduction in Problem Behavior Produced by “Matched” and “Unmatched” Food and Nonfood Stimuli.

Note. These data are limited to stimuli that the authors of the included studies characterized as “matched/unmatched” to the function of automatically maintained problem behavior.

The designation of a stimulus as “matched” is a subjective judgment based on assumptions about the stimulation produced by PB and engagement. These findings indicate that researchers and clinicians should not assume that “matched” stimuli (i.e., those that would seemingly substitute for the reinforcement produced by PB) are inherently more effective at reducing PB than “unmatched” stimuli. However, it may be of practical value for future research to examine the conditions under which “matched” stimuli are more or less effective than “unmatched” stimuli. For instance, under some conditions, “matched” stimuli might better maintain PB reductions over extended durations or repeated exposures (cf. Ahearn et al., 2005). It may also be of value to determine whether “unmatched” food produces similar reductions in PB to “matched” food (e. g., Higbee et al., 2005).

Figure 5 depicts the percentage reduction in PB during the CSAs and corresponding CSA validations. In nine of the 17 comparisons, the percentage reductions in PB during the CSA were within 10 percentage points of those observed during the CSA validation. These included comparisons 1-4, 8-9, and 12-14 in Figure 5 from Brandy (1, 13), Marsha (2), Betsy (3), James (4), Malik (8), Read (9), and Tyrone (12, 14). In eight comparisons, the percentage reduction during the CSA validation was greater than 10 percentage points (i.e., the stimulus set performed worse during treatment than was expected based on the CSA results). These included comparisons 5-7, 10-11, and 15-17 in Figure 5 from Lenney (5), Brandy (6, 17), Betsy (7), Stephanie (10), James (11), Read (15), and Abigail (16). However, in three of those comparisons (15-17), the stimuli did not produce 80% reductions during the CSA; thus, their poor performance during treatment might have been predicted. Thus, in 12 of 19 applications, the percentage reduction in PB obtained during the CSA remained consistent following extended exposure. In no comparisons did the percentage reduction in the CSA validation exceed those obtained in the CSA by more than 10 percentage points.

Figure 5. Percentage Reduction in Problem Behavior During CSA and Corresponding CSA Validation.

Note. Asterisks (*) indicate that the authors did not explicitly hypothesize that the stimuli would reduce problem behavior during the extended analysis. The dotted vertical line indicates an 80% reduction in problem behavior.

We did not identify a relation between the CSA’s predictive validity and participant characteristics or CSA method (e.g., trial duration, number of series conducted). Results of one study (DeLeon, Toole, et al., 2005) tentatively suggested that competing stimuli identified during CSAs in which the trial durations were determined by the base rate of PB maintained lower rates of PB during extended analyses better than competing stimuli identified during shorter trials. Though we did not identify this effect, the reviewed datasets were limited to relatively high-rate PB. Trial duration might be an especially important consideration for low rate PB as one needs to obtain a valid sample using relatively brief trials. It is important for future research to identify the conditions under which methodological variations such as trial duration affect CSA outcomes and/or their predictive validity.

Overall, the predictive validity comparisons suggested that findings of the CSA corresponded to results obtained when stimuli were presented over longer durations, across a series of sessions. That is, the reviewed CSAs had high predictive validity when compared to extended analyses. Additionally, most of the “failed” comparisons (i.e., when there was more than 10 percentage points difference between the PB reduction observed in the CSA and the extended analysis) were evaluations of stimuli that did not reduce PB during the CSA. In other words, when “noncompeting” stimuli identified by a CSA were evaluated in extended analysis they were ineffective, as expected. Participant characteristics (e.g., function, topography of PB) and procedural variations (e.g., trial duration) were not associated with the predictive validity of the CSA validation comparisons. However, important caveats to this finding are that many participants had multiple CSA applications and validations (i.e., multiple comparisons) and many of the studies were published by authors from the same institution. It is possible that there was not a sufficiently large or diverse enough sample to identify relations between the aforementioned variables. Additionally, potential publication bias cannot be eliminated. It would be beneficial for future researchers to evaluate CSA results in extended analyses using conditions that resemble those used in the CSA prior to combining competing stimuli with other procedures. It would also be beneficial for researchers to report nonresults, such as the failure to identify any competing stimuli, as such information is pertinent to identifying or eliminating variables that affect CSA outcomes (see Hagopian et al., 2015).

General Discussion

Results of the current study showed that CSAs have been used to identify stimuli that compete with a variety of topographies and functions of PB. The majority of CSAs were conducted for automatically maintained PB; however, CSAs have also demonstrated utility for socially maintained PB. Several terminological and procedural differences were identified during reference filtering and data analysis. In the included studies, leisure items and activities comprised the majority of the stimuli assessed, followed by food items and attention. Reductions in PB were not inversely related to levels of engagement or contact with stimuli, regardless of the function of PB or whether the authors nominated the stimuli as “matched/unmatched” to the stimulation thought to be produced by automatically maintained PB. In all reviewed CSA applications, the authors defined competing stimuli on the basis of “low” rates of PB in the CSA, but none defined “low” relative to the no-stimulus control condition. Competing stimuli were subsequently delivered noncontingently as a stand-alone treatment or as a supplemental treatment component (e.g., during schedule thinning, when the reinforcers for PB were unavailable). Finally, when CSA validations were performed, CSAs were found to have good predictive validity. The PB reductions observed during the CSA were similar to those observed during extended analyses; the few instances of non-correspondence were not attributable to differences in participant characteristics or procedural variations in the CSAs.

Only two of the reviewed studies (DeLeon, Toole, et al., 2005; Jennett et al., 2011) described and tested refinements to CSA procedures. The majority of papers focused on subsequent treatment of PB using competing stimuli identified in the CSA. As previously discussed, DeLeon, Toole, et al. (2005) examined the relation between individualized sampling parameters (trial duration) and predictive validity of CSA results for two participants. Jennett et al. (2011) reported a case of automatically maintained PB for which no competing stimuli were identified when the stimuli were noncon-tingently available. The authors conducted two additional series in which prompts and blocking of automatically maintained PB were implemented; only when both procedures were in place were competing stimuli identified. Recently, Hagopian et al. (2020) replicated and extended this study by repeating the noncontingent access series after conducting the conditions in which engagement was prompted and PB disrupted. Relative to the initial free access condition, improvements were observed in all cases when the noncontingent access series was repeated. Although additional research needs to be conducted to determine if the effects maintain during extended analysis, the findings suggest that it may be possible to use the CSA to not only identify competing stimuli but also to establish them.

A common concern in clinical practice is the efficiency of assessment procedures. Although we could have calculated total assessment time based on reported trial duration and number of series conducted, those values would not have been especially informative, as the number of stimuli assessed varied across studies and trial duration did not appear to affect the outcome of the CSAs. However, future research on CSAs might consider evaluating parameters related to efficiency. One such parameter involves use of multiple-stimulus test trials, for which there are at least three potential uses. First, multiple-stimulus test trials might be conducted prior to isolated test trials as a stimulus screening procedure. As previously discussed, the procedure described by Ringdahl et al. (1997) did not allow for identification of the relative reductive effects of any given stimulus. It did allow for efficient identification of noncompeting stimuli and half of the stimulus sets evaluated did not reduce PB relative to the control. That is, in those cases, had isolated rather than multiple-stimulus test trials been conducted, it would have taken more time to determine that none of the stimuli reduced PB relative to the control. Thus, prior to conducting isolated test trials, a multiple-stimulus test trial could be conducted to identify stimuli that might be included or excluded from subsequent isolated test trials. This screening procedure could be especially useful when PB is automatically maintained and it might improve the efficiency of the augmented CSAs described by Jennett et al. (2011) and Hagopian et al. (2020).

Second, CSAs that include a no-stimulus control and several multiple-stimulus test trials could be used to evaluate the extent of reinforcer competition across sets or classes of stimuli (e.g., books, games, art supplies, figurines and accessories) rather than evaluating each stimulus individually (cf. Brogan et al., 2018). Identifying the extent to which an entire stimulus set reduces PB relative to a control may be particularly useful in school settings, where stimulus sets are made available at certain times (e.g., “centers”), or for individuals with relatively well-developed play skills who might engage with seemingly unrelated stimuli during imaginative play (e.g., the use of blocks to build houses for figurines).

Third, multiple-stimulus test trials might be useful if a CSA identified several competing stimuli, all of which would be concurrently available during treatment. Prior to treatment, it might be beneficial to briefly evaluate the extent of PB reduction in the presence of multiple competing stimuli as the number of stimuli available might constitute an increase in effort that could affect their reinforcing potency. Importantly, whether such evaluations should be termed CSAs will depend on whether competing stimuli are identified by PB reduction in their presence relative to their absence.

As previously discussed, it is a matter of technological precision to identify competing stimuli based on observed levels of PB relative to rates of PB observed in a no-stimulus control comparison. The purpose of including a control within the context of the CSA is to provide a (relatively confound-free) point of comparison for the test trials. It is possible that, in the absence of such a condition, percentage reduction in PB could be calculated using data from similar no-stimulus conditions that were conducted in another context (e.g., the ignore condition of a functional analysis). However, it would be necessary to account for potential confounds, such as the time between observations, and differences across the no-stimulus comparison condition and the CSA test trials, such as observation or trial duration, or experimental design. The extent of procedural differences between the test and control condition would determine where such assessments would fall along the continuum of experimental control (Kazdin, 2011), and the validity of such comparisons remains an empirical question. In any case, we maintain that the term “CSA” be restricted to assessments that quantify the extent of reinforcer competition relative to a no-stimulus control condition that is parametrically identical to the test condition. As CSA procedures continue to evolve, it is of utmost importance for researchers and practitioners to distinguish reinforcer preference and competition, as a matter of technological precision and conceptual continuity.