Assessing Parameters of Reinforcement on Efficiency of Acquisition: A Systematic Review

Abstract

Discrete-trial training (DTT) is a common method of instruction used in early intervention amongindividuals with autism spectrum disorder and related neurodevelopmental disorders and is an effective method of teaching different skills such as tacting, listener responding, and matching. Delivery of effective reinforcers is a key component of DTT. Although general recommendations have been made for effective reinforcement delivery in DTT, no review has synthesized the available research on the efficiency of various reinforcer parameters on efficiency of acquisition. The current systematic review assessed the efficiency of various reinforcer parameters on acquisition in DTT. Results were idiosyncratic, and a general paucity of repeated measures examining specific reinforcer parameters within and across studies was observed. In general, (1) maintaining high levels of treatment integrity, (2) delivery of tangible (i.e. leisure items) or edible reinforcers in comparison with contingent praise as a reinforcer, and (3) delivery of edible reinforcers in comparison with other topographies of reinforcement were the most successful reinforcer parameter manipulations and always resulted in more efficient skill acquisition. The results of this review provide clinicians with information regarding what reinforcer parameter manipulations may be more or less likely to support efficient acquisition. The present review also provides considerations and makes recommendations for future research.

Discrete-trial training (DTT) is an instructional method common in early intensive behavioral intervention (EIBI) for young learners with autism spectrum disorder (ASD) or related neurodevelopmental disabilities (e.g., Lovaas, 1987; McEachin et al., 1993; Sheinkopf & Siegel, 1998). Although not typically used as the sole instructional method, DTT alone and in combination with other types of instruction has shown to be an effective method of teaching new skills (Downs et al., 2008; Eikeseth et al., 2002; Howard et al., 2005; Remington et al., 2007; Smith et al., 2000). A discrete trial is a short-duration unit of instruction that is usually delivered by a single therapist (e.g., Carroll et al., 2013), teacher (e.g., Dib & Sturmey, 2007), or parent (Lafasakis & Sturmey, 2007) to an individual learner. Smith (2001) and Tarbox and Najdowski (2008) describe five components of discrete trials: (1) cues (i.e., antecedent stimuli that evoke or will be trained to evoke appropriate responding) (2) prompts (i.e., responses emitted concurrent with or following a cue that assist a learner in emitting a correct response; Wolery & Gast, 1984), (3) responses (i.e., behaviors emitted by the learner), (4) consequences (i.e., the stimuli that follow a response), such as praise, tangible reinforcement (i.e., leisure items or activities), edible reinforcement (i.e. consumable foods), or instructional feedback, and (5) intertrial interval (i.e., the duration between end of one trial and the beginning of the next; e.g., pacing; Neil et al., 2020). In general, the presence of these components remains fixed but delivery of each of them may vary across learners and therapists. Manipulating components 1, 2, 4, and 5 might affect responding (component 3) and the efficiency with which a learner acquires the response targeted in the discrete trial.

Efficiency is defined as total teaching time, number of trials to mastery of skills, or number of sessions to mastery (Reichow & Wolery, 2011). Several recent behavior-analytic research articles have manipulated components of instructional procedures for a predetermined set of acquisition targets to compare efficiency of acquisition using an alternating-treatments design (e.g., Carroll et al., 2013; Carroll et al., 2016; Carroll et al., 2018; Lang et al., 2014; Markham et al., 2020; Turan et al., 2012). These studies assess which manipulation supports the most rapid acquisition (i.e., a “race to the top”). Figure 1 depicts an example of a “race-to-the-top” study in which sessions to mastery across two sets of acquisition targets, one reinforced with reinforcer X, and one reinforced with reinforcer Y, are shown. When the intervention was implemented following the phase line, percent correct responses increase in both sets, but mastery criteria are met in fewer sessions with reinforcer x (white circles) than reinforcer y (black circles). Thus, reinforcer X serves as a more efficient manipulation in this hypothetical evaluation. These designs may show clear distinctions across treatment conditions if discrimination across conditions occurs. Despite some pitfalls in its setup (e.g., multiple-treatment interference [Kazdin, 2011], difficulties equating tasks), this design is a quick and efficient way to identify the variables that support most efficient target mastery.

Fig. 1.

Example of a “Race-to-the-Top” study design. Note. Hypothetical data

Although manipulation of any component of a discrete trial can influence rate of acquisition, component 4 (i.e., consequences) is important particularly because a response must contact reinforcement for it to be acquired or maintained. Manipulating parameters of reinforcement (e.g., reinforcer schedule, type, magnitude, delay, choice) can affect acquisition (e.g., Frank-Crawford et al., 2019; Joachim & Carroll, 2018; Johnson et al., 2017; Sy & Vollmer, 2012; Toussaint et al., 2016). For example, Toussaint et al. (2016) employed a multielement design to compare acquisition of tacts of common objects and intraverbal responses when participants were provided with either choice of reinforcer, no choice of reinforcer (yoked to match the choice of reinforcer condition), or no reinforcer and observed that acquisition was as or more efficient when a choice of reinforcer was provided. Thus, relatively simple changes to a necessary component of a skill-acquisition program can substantially influence efficiency of acquisition.

Nonetheless, a common mistake in implementing DTT is the use of inadequate reinforcers (Steege et al., 2007). Reinforcers may produce less efficient skill acquisition because of errors in reinforcer delivery (e.g., Carroll et al., 2013), increased delay to reinforcement (e.g., Majdalany et al., 2016), use of low-magnitude reinforcers (e.g., Paden & Kodak, 2015), and the method of reinforcer delivery (e.g., Frank-Crawford et al., 2019), among other reasons. Small adjustments to reinforcement parameters can reduce acquisition efficiency. For example, using a multielement design, Majdalany et al. (2016) observed that, in general, participants required more sessions to acquire tacts of images of different countries when reinforcement for correct responding was delivered following 6- and 12-s delays relative to when there was no delay to reinforcement.

Although general recommendations for effective reinforcement delivery in DTT exist (e.g., Lerman et al., 2016), a general synthesis of the breadth of possible reinforcer parameter manipulations that could be incorporated individuals’ DTT programs would be useful for researchers and clinicians alike. A comprehensive review might allow for general recommendations from which individualized treatments can be derived. Given that there appeared to be discrepancies between what occurs in clinical practice and within published research (Graff & Karsten, 2012), more comprehensive guides would be useful in disseminating information to clinicians. Furthermore, reviewing trends in reinforcement-based instruction is useful in it’s own right to better understand the types of reinforcers being used and their potential long-term usefulness. For example, praise may result in less efficient acquisition but enhance generality, and edible reinforcement may result in more efficient acquisition but lead to nutritional imbalance. The frequency with which different parameters of reinforcement are employed should be compared to their efficacy, efficiency, and long-term maintenance.

Recent reviews of DTT have focused exclusively on components such as error-correction procedures, task interspersal, additional target incorporation, and differential procedures (Cariveau et al., 2019; Nottingham et al., 2015; Rapp & Gunby, 2016; Vladescu & Kodak, 2010). These authors used methodology akin to a rapid review in that they reviewed a subset of specific journals or articles (see Watt et al., 2008, for a review). Rapid reviews are replicable, informative, and useful when expedited completion of the review is required. However, these reviews likely contribute to publication bias (i.e., the “file-drawer problem”) by omitting relevant studies published outside of premiere journals and grey literature (Ganann et al., 2010). The recommendations made by rapid reviews may be premature or less-informed than those provided by systematic reviews (Hailey et al., 2000). Therefore, the purpose of this review was to conduct a systematic review synthesizing extant research comparing various reinforcement parameters used in DTT on rates of acquisition. In particular, we examined research in which an alternating-treatment design was used to compare the effect of various reinforcer parameters on efficiency of acquisition (i.e., “race-to-the-top studies”) in children with ASD.

Method

Search Strategy and Inclusion/Exclusion Criteria

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist and flow diagram to aid in conducting this review (Page et al., 2021). The titles and abstracts of articles in the PsycINFO (EBSCO) and PubMed (Medline) databases were searched initially on May 19, 2021, using the terms (Acqui* OR "skill" OR Learn* OR Master*) AND (Instructi* OR "DTI" OR "DTT" OR "Discrete Trial" OR Train* OR "discrete-trial" OR teach*) AND ("Early Intervention" OR Autis* OR (Intellectual* AND Disab*) OR (Development* AND Disab*) OR (Mental* AND Retard*) OR "EIBI" OR "IDD") AND (reinforce* OR edible* OR "leisure" OR "token" OR magnitude* OR "quality" or delay*). We selected these terms to capture articles that focused on using reinforcement-based comparisons of skill acquisition among individuals with intellectual and/or developmental disabilities. The search was not restricted by year or by journal. Dissertations located in these online databases were also retrieved. Therefore, search terms were included to capture diagnostic classifications that are no longer used. Following the screening process (described below) the references of included articles were screened to identify additional studies for inclusion that may not have been returned from the database searches.

The titles and abstracts of all retrieved studies were screened by two trained screeners to determine eligibility for full-text screening. At least one of the first two authors screened each study. To meet criteria for inclusion, a study must have (1) included children (i.e., under the age of 15) or youth (ages 15–24; United Nations Department of Economic and Social Affairs, 2013 diagnosed with a neurodevelopmental disorder; (2) used a multielement or alternating-treatments design; (3) evaluated some aspects of reinforcement on response acquisition that occurred following completion of the task (i.e., the manipulation did not occur prior to task completion); (4) was conducted in an early-intervention or DTT context; (5) used a within-subject design; and (6) analyzed trials or sessions to mastery. Studies that (1) were nonexperimental; (2) included adults or typically developing individuals; (3) compared responding across experiments or groups; (4) manipulated instructional and reinforcement parameters concurrently (i.e., both antecedent and consequence manipulations were implemented together such that the effect of either could not be isolated); (5) did not manipulate parameters of reinforcement other than the presence of a reinforcer; or (6) did not meet all the inclusion criteria were excluded from this analysis. A study was marked for full-text screening if inclusion or exclusion could not be determined by screening the title and abstract alone. Dissertations (i.e., grey literature) were included if they met inclusion criteria. Hand searches were also conducted to verify that our search criteria was comprehensive. Two trained screeners and at least one author of this article read the methods section for each study included in the full-text review and determined eligibility for data extraction using the same inclusion and exclusion criteria.

Searches of both databases yielded a total of 2,046 studies and were imported into Covidence™ (Veritas Health Innovation, n.d.) for screening. Covidence is an online software tool that offers streamlined services to aid in conducting systematic reviews. Of the total number of articles, 271 articles were removed as duplicates. An additional 1,720 studies were excluded during title/abstract screening for not meeting inclusion criteria. Of the remaining 55 full-text studies assessed for eligibility, 29 were removed because inclusion criteria were not met once the studies were read in their entirety. Therefore, full-text screening resulted in identification of 26 studies to be included in the review. The citations of each of the included articles were searched to determine other articles that merited inclusion (i.e., a “snowball search”), and eight additional studies were found. Figure 2 shows the PRISMA diagram for the present review, which summarizes this process.

Fig. 2.

PRISMA diagram

Interrater Reliability

Interrater agreement of article inclusion during title and abstract screening and the full-text review was calculated by dividing the number of articles in which inclusion/exclusion was agreed upon by the number of agreements and disagreements and multiplying by 100%. Every article was screened by the first and second author. Interrater agreement was 92%. In cases in which there was a disagreement on whether the article should be included, at least two of the authors discussed the article until they were able to reach a unanimous agreement.

Interrater agreement of data extraction was conducted by identifying the reinforcer parameter category based on the author description of the reinforcer parameter manipulation, and sessions or trials to mastery for each evaluation for each subject. These data were extracted independently by the first and second authors for 33% of the included articles. Interrater agreement was calculated by dividing the number of articles in which the reinforcer parameter manipulation or the number of sessions/trials to mastery was agreed upon exactly, and divided by the number of agreements and disagreements ad multiplying by 100%. Reviewers discussed any disagreement and came to a consensus before including the data. Interrater agreement for reinforcer parameter category was 100%, and for sessions/trials to mastery was 93%.

Data Extraction

Study Characteristics and Subject-Specific Data

Data on article characteristics extracted included year of the published study, number of subjects, journal, and category (or categories) of reinforcer parameters. Subject-specific data included diagnosis and age (if provided). We next collected data on included acquisition targets, how researchers equated for target difficulty, and a brief narrative of the reinforcer parameter manipulations assessed in each study.

Reinforcer Parameter Categories

We then devised a categorization scheme of reinforcer parameters by grouping the manipulations described in each study into relevant thematic categories. All categories and the studies we assigned to each category are shown in Table 1. For the purposes of the current review, differential reinforcement with extinction (Diff-EXT) was defined as reinforcement contingent on correct, independent responses, with no reinforcement following prompted correct or incorrect responses. Differential reinforcement without extinction (Diff no-EXT) is defined as (1) reinforcement contingent on correct, independent responses, and reinforcement of a lower magnitude or quality following prompted correct responses; or (2) reinforcement contingent on every correct, independent response (i.e., fixed-ratio 1 schedule), and a thinner schedule of reinforcement for prompted correct responses (i.e., fixed-ratio 3 schedule). Nondifferential reinforcement (non-Diff) was defined as reinforcement contingent on both correct, independent responses and correct, prompted responses. Although this definition of “nondifferential reinforcement” may be somewhat of a misnomer, this definition follows the way in which “differential reinforcement” is applied in skill-acquisition literature. In a differential-outcomes procedure, a specific reinforcer is delivered following specified responses in the presence of a specified discriminative stimulus (McCormack et al., 2021). Comparisons of different types of social reinforcement were defined as those that compared acquisition using various forms of social interaction as reinforcement. Topographically different reinforcers are comparisons of reinforcers from distinct topographical classes. These manipulations could have met the definitions for other categorizations such as differential outcomes or differential reinforcement but were not characterized as such by the authors of those studies; thus, “topographical” is used as a catch-all term for the remaining manipulations. Immediate versus delayed reinforcement was defined as the delivery of a reinforcer following a response with no programmed delay versus the delivery of a reinforcer after some programmed delay following a correct response. Assessments of reinforcer magnitude included studies that compared reinforcer quantity, size, or access time on skill acquisition. Studies included in this review under the accumulated versus distributed reinforcement category compared skill acquisition when reinforcement was delivered as a larger magnitude following a given number of responses (i.e., accumulated) and in lesser magnitudes following smaller units of responding (i.e., distributed). The compound versus single reinforcer category was defined as the delivery of edible reinforcement plus praise after unprompted responses and praise only following prompted responses versus the delivery of edible reinforcement alone after unprompted responses. The studies in the choice of reinforcer versus no choice category compared skill acquisition when either subjects or experimenters selected the reinforcement to be delivered, and the selection made by either subject or experimenter occurred after the correct response was emitted. The omission errors category was defined as a comparison of various levels of treatment integrity such that reinforcement was sometimes not delivered following correct responding.

Table 1.

Categorization scheme

Reinforcer Parameter Manipulation	n	Citations
Differential Reinforcement or Differential Schedules of Reinforcement (Diff-EXT vs. Diff no-EXT or Nondifferential; Diff no-EXT vs. Nondifferential)	7	Campanaro et al., 2019; Cividini-Motta & Ahearn, 2013; Fiske et al., 2014; Johnson et al., 2017; Karsten & Carr, 2009; Pence & St. Peter, 2015; Touchette & Howard, 1984
Differential Outcomes	5	Addison, 2006; Chong & Carr, 2010; Eldevik et al., 2020; McCormack et al., 2017; McCormack et al., 2021
Types of Social Reinforcement	5	Clausen et al., 2007; Pacitto, 2019; Polick et al., 2012; Stevens et al., 2011; Weyman & Sy, 2018
Topographically Different Reinforcers (Reinforcer Type)	5	Joachim & Carroll, 2018; Kang et al., 2013; Kodak et al., 2013; Koop et al., 1980; Leaf et al., 2014
Immediate vs. Delayed Reinforcement	4	Carroll et al., 2016; Grindle & Remington, 2004; Majdalany et al., 2016; Sy & Vollmer, 2012
Reinforcer Magnitude	4	Boudreau et al., 2015; Campanaro et al., 2019; Johnson et al., 2017; Paden & Kodak, 2015
Accumulated vs. Distributed Reinforcement	3	Frank-Crawford et al., 2019; Keen & Pennell, 2015; Kocher et al., 2015
Compound vs. Single Reinforcers	3	Boudreau et al., 2015; Campanaro et al., 2019; Johnson et al., 2017
Choice of Reinforcer vs. No Choice	2	Northgrave et al., 2019; Toussaint et al., 2016
Omission Errors	1	Carroll et al., 2013

One study may be coded into more than one category

Each included study could be grouped into one or more categories, based on the reinforcer parameter manipulations included in the study. For example, Johnson et al. (2017) assessed the efficiency of delivering compound reinforcers, an increasing reinforcer magnitude, and delivering nondifferential reinforcement on efficiency of acquisition. Thus, we placed this study in the differential reinforcement, reinforcer magnitude, and compound versus single reinforcers categories.

Reinforcer Parameter Evaluations and Efficiency of Acquisition

Next, we coded the number of (1) subjects exposed to each reinforcer parameter manipulation across all included studies in the parameter manipulation category and (2) evaluations of each reinforcer parameter manipulation. We recorded both number of subjects and evaluations per category as one subject may have experienced one, two, or several evaluations of the reinforcer parameter manipulation when the experimental design included a multiple baseline across target sets or replication conditions. Outcomes were coded by determining the effect of each reinforcer parameter on efficiency of acquisition across both number of subjects and number of evaluations. We used both visual analysis and author report to determine efficiency of acquisition by counting the number of sessions required to achieve the mastery criteria set by the authors. Outcomes were coded on a subject-by-subject basis and an evaluation-by-evaluation basis to prevent artificially inflating or deflating results.

Results

Study Characteristics

A total of 34 studies including 114 subjects were included for analysis. Almost half of published studies were from the Journal of Applied Behavior Analysis (n = 15). No other journal had more than two published studies included in the review. The majority of the studies included were published between 2010 and 2021 (n = 28).

Table 2 shows study-specific characteristics. Each citation with the number of subjects included, their ages and diagnoses, whether the study included replications or maintenance probes, and the results of replications and maintenance probes are displayed as the table columns. The majority of subjects were 9 years of age or younger (n = 90), with most of those individuals being between the ages of 3 and 5 (n = 52). Most participants had a diagnosis of ASD with or without comorbidities present (n = 87). Thirteen of the 34 studies included replications of the intervention, and of those 13 studies there were only 4 in which the most efficient reinforcer parameter manipulation remained the same across each replication for each subject. Seven studies reported maintenance probes. Levels of correct responding during maintenance probes varied within and across participants.

Table 2.

Study-specific characteristics

Citation	# Subjects	Subject Age Range	Diagnoses	Replication Attempted?	# Subjects for whom Successful Replication?	Maintenance Probes?	Levels of Correct Responding in Maintenance Probes by Subject
Addison, 2006	4	3–6	Autism, IDD	Yes	1 of 4	No	-
Boudreau et al., 2015	3	7–10	Autism	No	-	No	-
Campanaro et al., 2019	3	7–9	ASD	Yes	1 of 3	No	-
Carroll et al., 2016	2	4–5	Autism	Yes	1 of 1	No	-
Carroll et al., 2013	2	4–5	ASD	No	-	No	-
Cividini-Motta & Ahearn, 2013	3	12–16	ASD	No	-	No	-
Chong & Carr, 2010	3	4–4	Autism	No	-	No	-
Clausen et al., 2007	3	2–3	Autism	No	-	No	-
Eldevik et al., 2020	8	3–9	ASD	No	-	No	-
Fiske et al., 2014	3	5–8	Autism	No	-	No	-
Frank-Crawford et al., 2019	4	5–24	ASD, ADHD	Yes	2 of 3	No	-
Grindle & Remington, 2004	5	5–10	ASD	No	-	Yes	Luke & Robby: high Dave, Jake, & Teddy: moderate
Joachim & Carroll, 2018	4	4–9	ASD	Yes	1 of 1	Yes	Gabe: low Chad: variable Garrett: high
Johnson et al., 2017	3	8–10	ASD	No	-	No	-
Kang et al., 2013	3	3–8	ASD	No	-	No	-
Karsten & Carr, 2009	2	3–5	Autism	Yes	0 of 1	No	-
Keen & Pennell, 2015	3	3–4	ASD	No	-	No	-
Citation	# Subjects	Subject Age Range	Diagnoses	Replication Attempted?	# Ss for whom Successful Replication?	Maintenance Probes?	Levels of Correct Responding in Maintenance Probes by Subject
Kocher et al., 2015	3	16–17	ASD	Yes	0 of 2	No	-
Kodak et al., 2013	2	4–6	ASD	No	-	No	-
Koop et al., 1980	5	18–21	Autism, Encephalopathy, Down Syndrome	n/a	-	No	-
Leaf et al., 2014	3	4–5	Autism	No	-	Yes	Penny, JD, & Chris: high
Majdalany et al., 2016	3	5–5	ASD	No	-	No	-
McCormack et al., 2017	3	7–12	ASD	No	-	No	-
McCormack et al., 2021	4	5–10	IDD	Yes	1 of 3	Yes	Gabe: high Edgar: high/moderate Benji: chance levels
Northgrave et al., 2019	2	5–9	ASD	Yes	1 of 1	No	-
Pacitto, 2019	3	7–8	ASD, ADHD	No	-	No	-
Paden & Kodak, 2015	4	4–5	Autism	No	-	No	-
Pence & St. Peter, 2015	3	6–8	DD	No	-	No	-
Polick et al., 2012	2	3–4	Autism	Yes	1 of 2	Yes	Shaun & Brad: mostly high
Stevens et al., 2011	2	6–15	PDDNOS	Yes	0 of 2	Yes	Joey: high Alan: mostly high
Sy & Vollmer, 2012	8	3–8	DD	Yes	4 of 6	No	-
Touchette & Howard, 1984	3	6–13	Severely retarded	No	-	No	-
Toussaint et al., 2016	3	3–5	Autism	Yes	2 of 2	No	-
Weyman & Sy, 2018	3	19–22	Autism/IDD	No	-	Yes	Ophelia: moderate, low Mason: high, moderate

Some studies included greater than the listed number of subjects; however, not all subjects may have met inclusion criteria for the systematic review. IDD = intellectual and developmental disability. DD = developmental disability. # Ss for Whom Successful Replication = the number of subjects for whom the replication phase of the acquisition assessment resulted in the same reinforcer parameter manipulation supporting the most efficient acquisition across all replications, out of the total number of subjects that experienced a replication in the study. However, Sy and Vollmer (2012) conducted several experiments with some of the same participants; thus, each experiment was counted as a separate series of replications. Levels of Correct Responding in Maintenance Probes by Subject: general levels of correct responding, based on both author report and visual analysis, for each subject that experienced maintenance probes in the study

Table 3 shows how authors of each study equated for target difficulty. Sixteen studies (a sum of the first and second row) equated for syllable length (for verbal targets). In 7 of those 16, the authors also ensured that no similar sounds were present within each target set. For seven studies, authors did not report any rationale for equating for target difficulty. Authors reported conducting echoic assessments of the targets with the participants in six studies. In four studies, authors reported confirming that there were no shared properties across targets. Authors in three studies reported equating targets by number of letters. In three different studies, authors stated that targets were selected based on functioning level and a standardized assessment of skills such as the VB-MAPP. In one study in which multistep motor tasks were used, authors reported equating the number of steps required to complete the tasks. In an additional four studies, authors simply stated that “attempts were made” to equate for difficulty but did not expand upon this. Lack of reporting of explicit criteria used to equate for difficulty across many of the included studies presents a concerning confound: differential mastery of target sets may have occurred due to the differential ease of targets in one set as opposed to another set, and not due to the reinforcer parameters implemented.

Table 3.

Equating for target difficulty across studies

Attempts to Equate for Target Difficulty	n	Citations
Equated for syllable length AND dissimilar sounds in a target set	7	Campanaro et al., 2019; Carroll et al., 2016; Joachim & Carroll, 2018; Johnson et al., 2017; Kodak et al., 2013; Northgrave et al., 2019; Paden & Kodak, 2015;
Equated for syllable length	9	Addison, 2006; Cividini-Motta & Ahearn, 2013; Frank-Crawford et al., 2019; Keen & Pennell, 2015; Majdalany et al., 2016; Pacitto, 2019; Pence & St. Peter, 2015; Polick et al., 2012; Toussaint et al., 2016
Echoic assessment (where relevant) conducted to ensure proper enunciation of targets, or of words of equivalent syllable length	6	Carroll et al., 2016; Joachim & Carroll., 2018; McCormack et al., 2017; McCormack et al., 2021; Northgrave et al., 2019; Pence & St. Peter, 2015
No shared physical properties/formal dissimilarity	4	Campanaro et al., 2019; Grindle & Remington, 2004; Johnson et al., 2017; Touchette & Howard, 1984
Similar number of letters across targets	3	Cividini-Motta & Ahearn, 2013; Frank-Crawford et al., 2019; Weyman & Sy, 2018
Targets selected based on assessments such as VBMAPP of ABLLS	3	Campanaro et al., 2019; Chong & Carr, 2010; Clausen et al., 2007
Equated for number of steps required for motor task completion	1	Koop et al., 1980
“Attempts were made to equate for difficulty” or similar statement	4	Carroll et al., 2013; Fiske et al., 2014; Kang et al., 2013; Stevens et al., 2011
No rationale stated	7	Boudreau et al., 2015; Clausen et al., 2007; Eldevik et al., 2020; Karsten & Carr, 2009; Kocher et al., 2015; Leaf et al., 2014; Sy & Vollmer, 2012

Some studies met criteria for multiple methods of equating for target difficulty

Data Analysis

Table 4 shows the data analysis for each category of reinforcer parameters based on a head count of the number of subjects for whom the italicized manipulation was more efficient at any point. Because many of the studies included in the systematic review were designed as an alternating treatments design embedded within a multiple-baseline-across-targets design, the italicized manipulation may have been assessed across several replications. Thus, if the manipulation was more efficient for a subject in one leg of a multiple-baseline design but not as efficient in a replication using new targets, then the italicized manipulation was marked as more efficient for that subject at any point and would be a tally in the “more efficient” column. For example, Polick et al. (2012) compared descriptive and general praise on efficiency to acquisition. The assessment was set up as an alternating-treatments design embedded within a multiple-baseline-across-targets design. The subjects were exposed to three manipulations comparing general and descriptive praise. For both subjects, descriptive praise (the italicized manipulation in Tables 4 and 5) was more efficient for two out of the three manipulations. Thus, each subject would count as a tally under the “more efficient” column in Table 4. Efficiency was determined by the total number of trials/sessions required to reach mastery criteria. How efficiency was determined differed across studies because mastery criteria were defined by each author. Within the topographically different reinforcers category, the types of reinforcers may have been compared against each other within a given study. For example, Leaf et al. (2014) compared tangible reinforcement, edible reinforcement, social reinforcement, and feedback alone on efficiency of acquisition.

Table 4.

Subject-by-subject efficiency analysis

Reinforcer Parameter Manipulation	Intervention	# Subjects	More Efficient per Subject (at any point)	Never More Efficient per Subject	% More Efficient All Subjects
Differential Reinforcement or Differential Schedules of Reinforcement	Diff-EXT vs. Diff no-EXT or Nondifferential; Diff no-EXT vs. Nondifferential	25	21	4	84%
Differential Outcomes (DO)	DO vs. No DO	19	15	4	78.9%
Immediate vs. Delayed Reinforcement	Immediate vs. Delayed	18	14	4	77.8%
Reinforcer Choice	Choice vs. No choice	5	2	3	40.0%
Reinforcer Magnitude	High magnitude vs. Low magnitude	13	6	7	46.2%
Compound vs. Single Reinforcers	Compound reinforcers vs. Single reinforcer	9	5	4	55.6%
Errors of Omission	No errors vs. Omission errors	3	3	0	100%
Accumulated and Distributed Reinforcement	Accumulated reinforcement vs. Distributed reinforcement	10	9	1	90.0%
Topographically Different Reinforcers (Reinforcer Type)	Preferred edible reinforcer vs. Other	3	3	0	100%
	Preferred tangible reinforcer vs. Other	7	1	6	14.2%
	Token reinforcer vs. Other	4	2	2	50.0%
	Preferred social reinforcer vs. Other	6	0	6	0.0%
	Edible reinforcer/ Extended praise vs. Minimal praise	5	4	1	80.0%
	Tangible/Edible reinforcer vs. Praise	2	2	0	100%
Types of Social Reinforcement	Descriptive praise vs. General praise	5	5	0	100%
	Enthusiastic praise vs. Neutral praise	10	7	3	70.0%

Table 5.

Evaluation-by-evaluation efficiency analysis

Reinforcer Parameter Manipulation	Intervention	# Evaluations	More Efficient per Evaluation	No Diff	Less Efficient per Evaluation	% More Efficient All Subjects	% More Efficient for All Subjects When a Difference Existed
Differential Reinforcement or Differential Schedules of Reinforcement	Diff-EXT vs. Diff no-EXT or Nondifferential; Diff no-EXT vs. Nondifferential	35	23	6	6	65.7%	79.3%
Differential Outcomes (DO)	DO vs. No DO	30	17	3	10	56.7%	62.9%
Immediate vs. Delayed Reinforcement	Immediate vs. Delayed	37	19	17	1	51.4%	95%
Reinforcer Choice	Choice vs. No choice	8	4	4	0	50.0%	100%
Reinforcer Magnitude	High magnitude vs. Low magnitude	13	6	0	7	46.2%	46.2%
Compound vs. Single Reinforcers	Compound reinforcers vs. Single reinforcer	18	9	0	9	50.0%	50.0%
Errors of Omission	No errors vs. Omission errors	3	3	0	0	100%	100%
Accumulated and Distributed Reinforcement	Accumulated reinforcement vs. Distributed reinforcement	16	11	0	5	68.8%	68.8%
Topographically Different Reinforcers (Reinforcer Type)	Preferred edible reinforcer vs. Other	3	3	0	0	100%	100%
	Preferred tangible reinforcer vs. Other	4	1	3	0	25.0%	100%
	Token reinforcer vs. Other	5	3	1	1	60.0%	75.0%
	Preferred social reinforcer vs. Other	6	0	3	3	0.0%	0.0%
	Edible reinforcer/ Extended praise vs. Minimal praise	12	8	3	1	66.7%	88.9%
	Tangible/Edible reinforcer vs. Praise	2	2	0	0	100%	100%
Types of Social Reinforcement	Descriptive praise vs. General praise	14	8	3	3	57.1%	72.7%
	Enthusiastic praise vs. Neutral praise	15	7	2	6	46.7%	53.8%

Table 5 shows the data analysis for each category of reinforcer parameters. From left to right, the columns display (1) the general category of each reinforcer-parameter manipulation; (2) the specific reinforcement parameters manipulated; (3) the total number of datasets for each specific manipulation; (4) the number of datasets for which the first manipulation listed in column 2 (italicized) was more efficient than the comparator; (5) the number of datasets for which there was no difference in efficiency between manipulation; (6) the number of datasets for which the first manipulation listed in column 2 (italicized) was less efficient than the comparator; (7) the percentage of datasets for which the first manipulation listed in column 2 was more efficient than the comparator (i.e., dividing the number of evaluations for which the intervention was more efficient by the total number of datasets for a given intervention and multiplying by 100); and (8) the percentage of datasets for which the first manipulation listed in column 2 was more efficient than the comparator when there was a difference between comparators (i.e., dividing the number of evaluations for which the intervention was more efficient by the number of datasets for which the intervention was both more and less efficient and multiplying by 100, in other words, excluding datasets from the “No Difference” column). It is important to note that the percentage of subjects for whom the italicized manipulation was more efficient only shows that the manipulation was best during at least one manipulation. If the subject was exposed to replications (for example, studies for which the subjects were exposed to the independent variable manipulation in a multielement design with additional with replications across sets in a multiple baseline design), the manipulation was not always most efficient.

Several general response patterns emerged. First, some reinforcer parameters supported more efficient acquisition across every evaluation for every subject. Perfect treatment integrity (as opposed to programmed omission errors) led to more efficient acquisition 100% of the time. In addition, within the “reinforcer type” category, edible items compared to other reinforcers, and tangible plus edible items combined when compared to contingent praise led to more efficient acquisition 100% of the time when compared to delivery of other types of reinforcers. However, it is worth noting that these reinforcer parameters categories contained fewer studies than the others.

Several reinforcement parameter manipulations resulted in more efficient acquisition, on average, at both the subject and the evaluation level despite not being the most efficient manipulation 100% of the time. Use of differential reinforcement for correct responses resulted in more efficient acquisition than nondifferential reinforcement. In addition, accumulated reinforcement was more efficient for most of the datasets at any point, and 68.8% of all evaluations whether there was a difference in efficacy between comparators. One study in the “reinforcer type” category compared the efficiency of what the authors described as “extra” reinforcement (extended social reinforcement plus delivery of edible items) to minimal praise and was more efficient than delivery of other reinforcer types across most datasets and evaluations. Descriptive praise was more efficient across datasets and total evaluations when a difference in comparators existed.

Another response pattern to emerge was that of more efficient acquisition under a given reinforcer parameter category only when a difference across comparators existed. Differential outcomes manipulations and immediate reinforcement, as opposed to nondifferential outcomes and delayed reinforcement, resulted in more efficient acquisition for most subjects and for most evaluations when considering only datasets for which there was a difference in efficacy across comparators; however, these manipulations were only more efficient in supporting acquisition in 56.7% and 51.4% of all evaluations. These results show that on average there was little difference in these manipulations across all evaluations, but that when a difference existed, differential outcomes and immediate reinforcement were the more efficient manipulation compared to nondifferential and delayed reinforcement. In addition, reinforcer choice (i.e., participant choice vs. no choice) was more efficient for 100% of total evaluations when considering only datasets for which there was a difference in efficacy between comparators; however, it was the more efficient manipulations across subject and all evaluations only at chance levels (~50%). Within the “reinforcer type” category, delivery of preferred tangible items, compared to delivery of other items, were rarely more efficient across datasets and across all evaluations; however, they were observed to be more efficient for 100% of total evaluations when considering only datasets for which there was a difference in efficacy between comparators. Likewise, in the same category, delivery of tokens was more efficient than delivery of other items only at chance levels at the dataset- and evaluation-level; but was more efficient for 75% of total evaluations when considering datasets with a difference in efficacy between comparators.

Some reinforcer parameter manipulations resulted in more efficient acquisition at only chance levels. High-magnitude reinforcement and compound reinforcers supported more efficient acquisition than low-magnitude or single reinforcers, respectively, at chance levels across subjects and manipulations. This indicates that increasing the magnitude or quality of reinforcement contingent on a response will not necessarily increase efficiency of acquisition. Enthusiastic praise, when compared to neutral praise, was efficient for most datasets at any point, but was more efficient across evaluations with and without differences in efficacy between comparators at only chance levels.

Discussion

In the current systematic review, we assessed the effects of various reinforcer parameters on efficiency of acquisition using the increasingly common multielement, “race-to-the-top” research design. In general, idiosyncratic differences were observed for most reinforcer parameter manipulations, both within and across subjects’ data. However, several general response patterns emerged which inform key takeaways. First, the most robust findings were found for interventions manipulating integrity errors, tangible/edible reinforcer delivery compared to contingent praise, and edible reinforcement delivery (compared to other topographies of reinforcer). In these cases, 100% of subjects and evaluations showed enhanced efficiency of target mastery when treatment was implemented with 0% errors of omission, and when edible reinforcement was delivered. It is perhaps unsurprising that errors of omission slow mastery. It is also important to note that each of these categories only contained one study; thus, the generality of the results may be questionable. Nevertheless, these results hold important implications for practitioners. Clinicians should focus on maximizing treatment integrity when implementing interventions. In addition, providing edible reinforcers or edible and tangible reinforcement (when compared to contingent praise) was shown to increase efficiency of acquisition. Although delivery of edible reinforcers may not always be ideal, in circumstances in which efficiency of acquisition is a concern, results of this review indicate that this may be the most effective reinforcer manipulation to make. Although future research is needed to verify whether these reinforcer parameter manipulations maintain robust results across additional studies, these results hold important implications for clinicians developing DTT programs among children and youth with neurodevelopmental disorders.

Several reinforcer parameter manipulations were more efficient on average than their comparison counterpart but were not more efficient 100% of the time. However, these manipulations may still be important for clinical interventions. For example, differential reinforcement of correct, unprompted responses, descriptive praise as opposed to general praise, and the delivery of “extra reinforcement” beyond that of praise may be effective in enhancing efficiency of acquisition. Because these manipulations do not take much time or added materials to implement, they may be an important tool in the arsenal of a clinician. In addition, Campanaro et al. (2019) implemented a brief differential reinforcement assessment prior to evaluating differential versus nondifferential reinforcement. The authors determined whether magnitude, quality, or schedule was the most salient differential reinforcer manipulation, and included that in the subsequent analysis. Brief efficiency analyses may help identify which reinforcer parameter manipulation will be most effective for a given learner prior to designing a full-length acquisition program.

Accumulated reinforcement resulted in more efficient acquisition than distributed reinforcement across most evaluations. Accumulated reinforcement, beyond providing uninterrupted access to a given reinforcer, may also serve to increase fluency (as there are no breaks between responses), and reduce disruptions to total teaching time (DeLeon et al., 2014). Therefore, administering a DTT session using an accumulated reinforcement arrangement may result in more efficient acquisition due to both the manipulation of the reinforcer parameter and through elimination of disfluent teaching trials. More research is needed to uncover how reinforcer parameter manipulations such as accumulated reinforcement may affect all components of a discrete trial and how these separately affected components support acquisition.

The difference in values between the seventh and eighth columns in the evaluation-by-evaluation efficiency analysis in Table 5 warrants discussion for several reinforcer parameter manipulations. Again, the eighth column differs from seventh column in that efficiency is calculated by dividing the number of evaluations in which the manipulation was more efficient by the sum of the number of evaluations that were both more and less efficient for the given manipulation. The “no difference” column was not included in the calculation for the eighth column. For about half of the manipulations (reinforcer magnitude, compound vs. single reinforcers, errors of omission, accumulated vs. distributed reinforcement, preferred edible reinforcer, preferred social reinforcer, and tangible/edible reinforcer), percent efficiency remained the same across both calculations. Percent efficiency for the remaining manipulations increased when a difference existed (i.e., the calculations provided in the eighth column), indicating that although there were idiosyncrasies in the efficiency of each manipulation, in general the manipulations were more efficient, or no difference was observed. Manipulations such as immediate reinforcement versus delayed reinforcement, differential outcomes as opposed to nondifferential outcomes, and delivery of tangible or tokens in the “reinforcer type” category compared to delivery of other topographies of reinforcement all showed increases in efficiency when a difference existed. The opportunity cost for implementing these reinforcer parameter manipulations with clients is therefore low; although they may lead to no difference in rates of acquisition, they may also lead to more efficient acquisition of targets.

Finally, several reinforcer parameter manipulations resulted in little or no increases in efficiency across the included studies for that category. High-magnitude reinforcers compared to lower-magnitude reinforcers, delivery of compound reinforcers rather than single reinforcers, and delivery of enthusiastic praise instead of neutral praise all supported more efficient acquisition at around chance levels. Though more research is needed to verify these results, at least one surprising outcome is that increasing magnitude or quality of reinforcement contingent on a response did not necessarily increase correct responding. These chance-level results may in fact underscore the importance of individualizing treatments. For some learners, these manipulations may be valued, although for others, this may not be the case. Implementing brief reinforcer assessments prior to instruction may help to indicate whether this is the case (Boudreau et al., 2015; Campanaro et al., 2019).

There are several other important aspects of the included studies themselves, unrelated to the reinforcer parameter manipulations, that warrant discussion and highlight areas for continued research. First, most studies did not specify how difficulty was equated across targets. This reveals a concern with replicability that behavior analysts usually attempt to rectify with operational definitions and detailed descriptions of procedures. This limitation also inhibits the ability to specify the extent to which efficiency of mastery is a function of the given intervention, or if perhaps certain targets were less effortful to complete. Cariveau et al. (2020) reviewed methods used to equate target sets in studies using adapted alternating treatment designs and found that approximately half of the reviewed studies attempted to equate for difficulty. It is possible that some of the included studies in the present review and of that of Cariveau et al. (2020) did indeed equate targets for difficulty but did not state those methods in the article. Future research comparing various DTT manipulations on acquisition should include methods for equating for target difficulty and operationalize those methods in-text.

Second, only one third of the studies included in the current review embedded replications of the experimental manipulation for at least one participant. Across these 13 studies, replication of results failed in at least one manipulation in 9 of the studies; in other words, for nine studies, the same reinforcer parameter manipulation that supported the most efficient acquisition in evaluation 1 was not the manipulation that supported the most efficient acquisition in evaluation 2 (sometimes 3). For this reason, the across-subjects analysis (Table 4) likely overestimates the results of the reinforcer parameter manipulation. It is interesting that three of the studies that included replications stated no rationale in-text for how targets were equated for difficulty (Karsten & Carr, 2009; Kocher et al., 2015; Sy & Vollmer, 2012). For two of those studies (Karsten & Carr, 2009; Kocher et al., 2015), replication of results failed for each participant. For Sy and Vollmer (2012), replication was successful for four out of six participants. The remainder of the studies that attempted to replicate findings included some measure of equating for target difficulty. Because the methods of equating for target difficulty are so diverse, it is difficult to say whether these methods had any effect on whether replication succeeded or failed; however, placing a priority on equating for target difficulty to the greatest extent possible in future research may help researchers to confirm that if replications do fail, it is not due to possible differences in target difficulty across target sets. To that end, future research that employs multielement designs to assess the most efficient manipulation on acquisition should consider implementing replications, so long as it is feasible, as often the most efficient parameter may change across manipulations. This is important for clinicians because several reinforcer parameter manipulations may in fact help support acquisition and could be rotated or implemented when it is most convenient; in addition, the effect of certain manipulations may be washed out across replications and thus may not be necessary to implement in DTT.

Fienup and Carr (2021) conducted a review of research on mastery criteria in DTT and made a timely call for additional research in this area. Maintenance data ensure that the interventions that supported most efficient mastery also maintained high rates of correct responding over time and can help to determine whether mastery criteria implemented in a given intervention was sufficiently robust. About one quarter of studies included in the current review conducted maintenance probes. The paucity of maintenance measures is relevant, because without such measures, we cannot be sure that the results from the assessment were durable over time. Future research might assess appropriate parameters for determining true skill acquisition. Maintenance data may provide support for these criteria. Indeed, for those studies in which maintenance probes were conducted, high rates of correct responding were observed with approximately half of participants’ reported data. Researchers in this area should prioritize conducting maintenance probes in future as a demonstration of maintenance of correct responding and should consider modifying future mastery criteria if probes fail to demonstrate maintenance.

Finally, EIBI, DTT, and procedures related to applied behavior analysis are often touted as “gold-standards” for children with ASD (e.g., McPhilemy & Dillenburger, 2013). Given the increase in the number of certified behavior-analytic practitioners and large percentage of those working with individuals with individuals with developmental disabilities (73% working with individuals with ASD as of April 2021; Behavior Analyst Certification Board, n.d.), it is important that these practitioners are developing interventions based on research-based best practices. In terms of identifying best practices of various reinforcer arrangement for increasing skill-acquisition, much additional research is warranted. Although many clinical behavior analysts are likely able to (and should) assess parameters of reinforcement that work best for each individual client, additional research might inform these clinical assessments. The current review should serve as a prompt for future researchers to replicate the methods of the present studies to determine whether robust results can be produced consistently.

Some limitations of the procedures of the current systematic review exist and warrant discussion. Although systematic reviews are more comprehensive and reduce bias associated with reviewing a specific subset of articles or journals by reviewing several databases, it is still possible that not all relevant articles were captured with the search terms used in this review. Furthermore, research articles were excluded from the current systematic review if they did not state explicit mastery criteria. It is interesting that several articles met this exclusion criterion; otherwise, these studies would have otherwise been included in the review. In addition, we only focused our review to reinforcer parameters in “race-to-the-top” studies that used alternating treatments designs. Future systematic reviews might expand the search criteria to create a more comprehensive account of effective components of DTT on acquisition.

The categorization scheme used in this review may also be limiting. Although it was based on common dimensions of reinforcers (i.e., magnitude, delay, compound reinforcers) and descriptions provided by authors (e.g., differential reinforcement, differential outcomes), the distinction between these categorizations was sometimes tenuous. The categories were based on author description, rather than how a naïve observer might characterize the intervention in the absence of any formal description. For example, a compound reinforcer was defined in all included studies as the addition of praise to a tangible reinforcer (which all authors described as an increase in reinforcer quality). The addition of praise to a tangible reinforcer could be seen as differential reinforcement in that additional reinforcers are provided following a correct response. Likewise, differences in types of social reinforcement could be characterized as differences in quality in that the same vocal utterance presented in either a “neutral” and “enthusiastic” tone produces differences in reinforcing efficacy. Future research should evaluate other methods of categorizing parameters of reinforcement to determine additional possible commonalities.

The current systematic review was initiated as a first step towards synthesizing the literature of reinforcer parameter manipulations in discrete-trial instruction. The purpose of this synthesis was a practical one: to identify reinforcer parameter manipulations that support efficient acquisition and make recommendations that can inform clinical intervention. We found that, although efficiency of acquisition using a given reinforcer parameter often varied across datasets and across manipulations, some reinforcer parameter manipulations were overwhelmingly effective (i.e., avoiding omission errors, delivery of edible reinforcement, delivery of tangible plus edible reinforcement when compared to contingent praise). In addition, some parameter manipulations, such as differential reinforcement and accumulated reinforcement, may not lead to most efficient acquisition 100% of the time but often may help support more frequent acquisition of targets. We also found that several of the reinforcer parameters manipulated rarely or never support more efficient acquisition. These results are highly relevant to researchers and practitioners when designing DTT interventions. However, the results of this review raised several questions that warrant consideration. Many reinforcer manipulations that resulted in the most robust differences have not been replicated. In addition, idiosyncrasies with individual participants’ results, the noticeable lack of methods used to equate for target difficulty, and the paucity of maintenance probes prevent us from making sweeping recommendations. Another barrier to identifying generalities within reinforcer-parameter-manipulation categories is the idiosyncratic nature of each study design within each category. This may not a limitation given that most clinical manipulations in the field of behavior analysis are necessarily idiosyncratic and individualized. Nevertheless, the idiosyncrasies inherent across studies warrant caution when deriving recommendations from the results of this review. The current review revealed gaps in the literature and proposed several important questions for future researchers to address: to replicate many of the included studies, to include technological definitions of method, including ways in which targets are equated for difficulty and mastery criteria is selected, and to prioritize inclusion of maintenance probes.

Data Availability

Not applicable

Declarations

Conflicts of Interest

All authors declare that they each have no relevant conflicts of interest related to the preparation of this article.

Ethics Approval

Because the article was a synthesis of prior research, the article does not contain any studies conducted by the authors with human participants.

Consent to Participate

Not applicable

Consent for Publication

Not applicable

Concept Development

All authors contributed to study conception, analysis, and manuscript preparation. Design of search terms and data extraction were performed by Sarah Weinsztok and Kissel Goldman. The first draft of the article was written by Sarah Weinsztok. All authors commented on previous versions of the article and read and approved the final manuscript.