A Survey of Procedural-Fidelity Data Collection in Behavior-Analytic Practice

Abstract

Evaluating the extent to which applied-behavior-analytic interventions are carried out accurately (i.e., procedural fidelity) is important for quality control, data-based decision making, and facilitating optimal consumer outcomes. This study explored several questions related to procedural fidelity in practice by distributing a survey to behavior analysts who currently supervise or provide applied-behavior-analytic services in any setting and with any population. Specifically, we were interested in learning more about who, how, how often, for what, why, and where behavior analysts assess procedural fidelity in practice. The results from 203 behavior analysts who completed the survey revealed that behavior analysts and their supervisees were most likely to collect fidelity data using checklists while directly observing behavior-analytic services in various settings. The most common barriers to collecting fidelity data in practice were a lack of resources, no requirement to do so by employers, and limited supervision time.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40617-024-00995-1.

Procedural fidelity is the extent to which an intervention is implemented as designed (Gresham, 1989). Assessing the procedural fidelity of interventions is an important component of internal validity. In practice, procedural-fidelity data should be used by board certified behavior analysts (BCBA1) to make programming decisions and support behavior-change agents (CASP, 2024). When behavior changes in the direction necessary to achieve socially meaningful difference, procedural-fidelity data help bolster a BCBA’s determination that the intervention, not confounding variables, is likely responsible for the changes in the target behavior. This outcome supports the continued use of the intervention and implementation with other behavior-change agents, in other settings, and with other behaviors. When behavior is not improving in the direction necessary to achieve socially meaningful change, it may be that the intervention is not a good fit or is not being implemented correctly. Procedural-fidelity data allow a BCBA to determine whether the intervention needs to be revised or removed (i.e., when fidelity is high) or whether the behavior-change agent needs additional training (i.e., when fidelity is low).

Several reasons to measure procedural fidelity in practice emerge from descriptive, correlational, and experimental research on procedural fidelity. Descriptive analyses of procedural fidelity in the application of behavioral principles by behavior-change agents (e.g., paraprofessionals, teachers, behavior technicians) show that they make errors when implementing interventions to address behavior-reduction and skill-acquisition goals in school and clinic settings (e.g., Arkoosh et al., 2007; Bergmann et al., 2023; Breeman et al., 2020; Carroll et al., 2013; Colón & Ahearn, 2019; Donnelly & Karsten, 2017; Foreman et al., 2021; Kodak et al., 2018). For example, Carroll et al. (2013) reported that paraprofessionals and teachers implemented discrete-trial instruction (DTI) with children with autism spectrum disorder (ASD) with fidelity ranging from 21% (i.e., contingent tangible) to 74% (i.e., establish ready behavior). Correlational studies shed light on the relation between the fidelity of interventions and consumer outcomes (e.g., Dib & Sturmey, 2007; DiGennaro et al., 2007; Downs et al., 2008; Wood et al., 2007). That is, as interventions are implemented with higher fidelity, outcomes for consumers improve and vice versa (Brand et al., 2019). For example, DiGennaro et al. (2007) found that students were off-task less often when special education teachers implemented a function-based intervention package with higher fidelity. Finally, experimental analyses of programmed-fidelity errors conducted with translational and applied populations show that the type (e.g., errors of commission [adding components to an intervention], errors of omission [failing to implement prescribed components]) and degree of errors (e.g., 33%, 50%, 67% fidelity) affect the efficacy and efficiency of behavior-reduction and skill-acquisition procedures (e.g., Bergmann et al., 2017, 2021; Breeman et al., 2020; Carroll et al., 2013; Colón & Ahearn, 2019; Donnelly & Karsten, 2017; Foreman et al., 2021; Kodak et al., 2018; St. Peter Pipkin et al., 2010). For example, Carroll et al. manipulated the fidelity of DTI procedures based on their descriptive analysis, and the fidelity errors affected whether or how quickly the participants acquired the skills. Taken together, the extant research on procedural fidelity illuminates how crucial measuring procedural fidelity is to applied-behavior-analytic research and practice.

Historically, reviews of behavior-analytic literature published in a variety of journals have revealed that most data-based articles do not report procedural-fidelity assessment (see review in Falakfarsa et al., 2022) and the reasons for not assessing fidelity vary (St. Peter et al., 2023). Recently, however, assessing and reporting procedural fidelity in research has become more common (e.g., Bergmann et al., 2023; Han et al., 2023), and several behavior-analytic journals now require authors to include procedural-fidelity data in their submission (e.g., Education and Treatment of Children, n.d.; Preas et al., 2024). The behavior of researchers related to measuring and reporting procedural fidelity can be examined via systematic reviews of published literature, but there is no parallel method for the behavior of practitioners. Assessing procedural-fidelity measurement by BCBAs in practice is of paramount importance as these professionals often operate in applied contexts with direct consumers of applied behavior analysis (ABA) or in the development of ABA professionals.

Surveys can be beneficial for learning more about the behavior of BCBAs in practice. For example, through surveys the field gleaned useful data about a variety of topics related to ABA, including components of early intensive behavioral intervention programs (Love et al., 2009), knowledge of special education law (Vladescu et al., 2022), data-collection practices (Morris et al., 2022), preference assessments (Graff & Karsten, 2012), staff training and supervisory practices (Blackman et al., 2023; DiGennaro Reed & Henley, 2015; Sellers et al., 2019), training in the assessment and intervention of severe problem behavior (Colombo et al., 2021), training in compassionate care (LeBlanc et al., 2020), and use of treatments for people with ASD (Marshall et al., 2023; Schreck et al., 2016). To our knowledge, only two surveys have specifically queried ABA practitioners about procedural fidelity in practice.

Fallon et al. (2020) asked BCBAs who supervised caregiver-implemented ABA services with their children with ASD about their training in procedural fidelity, how often they collected fidelity data during supervision, and how they collected fidelity data. Fallon et al. reported that nearly all of the 314 respondents (i.e., 97.1%) received some training related to the importance of collecting procedural-fidelity data and data-collection methods, and 77.7% of the participants indicated that they thought their training was sufficient. Despite nearly all respondents (i.e., 99.6%) agreeing that procedural fidelity is a “key component of intervention success” (p. 5), 55.4% said that they collected procedural-fidelity data for 30% or fewer supervision sessions. With regard to how procedural-fidelity data were collected, an overwhelming majority of participants (i.e., 89.8%) said that they used direct observation (i.e., observing the behavior-change agent implementing the procedures live or from a recording) to assess procedural fidelity. Approximately 70% of respondents said they used an indirect assessment (i.e., self-report, interview, or permanent products) to assess fidelity for at least 10% of sessions (Fallon et al., 2020). A greater reliance on direct observation is promising, given that indirect assessments can be inaccurate and unreliable, especially without additional training (Fallon et al., 2018; Gresham et al., 2017).

The data collected by Fallon et al. (2020) represented an advancement in the knowledge of practice, but the sample was limited to BCBAs supervising in-home ABA intervention implemented by caregivers with their children with ASD. Thus, recruiting participation from additional supervising BCBAs would permit an assessment of other practice areas and settings. Dueker and Desai (2023) surveyed certified and non-certified behavioral practitioners regarding their attitudes toward procedural fidelity, their training related to procedural fidelity, and potential barriers to collecting fidelity data in practice. Dueker and Desai received 209 completed surveys. Nearly half of the respondents were BCBAs, and most respondents worked in clinical settings (i.e., 43.9%) for companies that served more than 30 clients (i.e., 72.5%). Respondents strongly agreed that procedural fidelity was important to the success of their clients, but they were more neutral when asked if procedural-fidelity measures are essential components in their programs. With regard to training, respondents agreed that they received training in calculating procedural fidelity and agreed that procedural fidelity is important in practice as well as research. When asked about barriers to using procedural-fidelity measures in practice, the respondents agreed that they had experience calculating procedural fidelity, that it is not time-consuming to collect procedural fidelity, and that their employer encouraged collecting fidelity data. The respondents in Dueker and Desai felt that they had the requisite skills and resources to collect procedural-fidelity data in practice; however, the survey did not provide information regarding how or how often practitioners collected these data.

The surveys by Fallon et al. (2020) and Dueker and Desai (2023) provided insight into the assessment of procedural fidelity in practice. However, there is still much to learn about who, how, how often, for what, why, and where practitioners collect and analyze procedural-fidelity data in practice. Although Fallon et al. asked respondents whether they used direct observation or one of several indirect assessments, they did not include additional details about measurement systems. For example, the researchers did not ask whether practitioners collected data for each trial or component of an intervention or if they used a rating scale. Direct observation is recommended for fidelity assessment (Gresham et al., 2000), but there are multiple measurement systems that can be used to assess fidelity during the observation. Not all fidelity measures will yield the same fidelity estimates, and different fidelity estimates could lead to different supervisor behavior (e.g., modify the intervention, continue as is, conduct additional training; Bergmann et al., 2023). As previous literature indicated, the collection of procedural-fidelity data and the way it is calculated may serve as moderating variables for intervention efficacy in practice and supervision. Therefore, the purpose of the current study was to survey BCBAs who are providing and/or supervising ABA services on their procedural-fidelity practices, including the method of, frequency of, purpose of, and barriers to assessing procedural fidelity in practice. To extend Fallon et al., participants were not limited by setting, specialty, or practice.

Method

Recruitment and Informed Consent

We recruited BCBAs who were currently supervising or providing ABA intervention with any population (e.g., ASD, brain injury, developmental disabilities, employees) and in any setting (e.g., clinics, homes, schools, organizations). Recruitment emails were distributed via the Behavior Analyst Certification Board’s mass email service on July 25, 2022, and August 1, 2022. Emails were also sent to the Association for Behavior Analysis International’s special interest groups who provided permission to recruit (i.e., Behavioral Gerontology, Naturalistic and Developmental Behavioral Interventions, and Rehabilitation and Independent Living). The recruitment information was also posted on various university department Facebook pages and personal social media profiles. The final response was recorded on October 5, 2022, and the survey was closed thereafter.

The recruitment message informed recipients that the purpose of the study was to understand procedural-fidelity data collection in the practice of ABA and identify potential barriers to collecting these data in practice. The recipients were informed that participants would not be excluded based on race, gender, or ethnic composition and that the survey would be administered in English via Qualtrics. Instructions stated that the survey would be anonymous, take approximately 10 min to complete, include questions about fidelity data collection and demographics, allow questions to be skipped, and did not include any compensation for completion. Recipients who were interested in participating clicked on the link embedded in the recruitment message and were instructed to read the informed-consent notice, which was approved by the university’s human subjects institutional review board (UNT IRB-21–687).

If the participant acknowledged that they read the consent notice and agreed to participate, Qualtrics advanced to two questions to confirm eligibility. The first question was designed to filter out respondents without basic knowledge of procedural fidelity. Before answering the question, the participants read:

Treatment integrity2 (also known as procedural integrity, procedural fidelity, treatment fidelity, and implementation integrity) is the extent to which intervention protocols are administered consistent with how they have been designed. For example, if a procedure requires the interventionist to administer a response prompt during every trial, treatment-integrity data measure the extent to which the interventionist provided the prompt during every trial. Although treatment integrity goes by several other names, we will be using the term treatment integrity throughout this survey. Specifically, we’ll be asking about treatment integrity in the context of applied behavior analytic services (ABA services) across different populations and settings.

Then, the survey advanced to the question, “Treatment integrity is also known as (you may select more than one answer):” with the choices: interobserver agreement, procedural fidelity, procedural implementation, and implementation reliability. To participate in the survey, the participant had to select procedural fidelity and could not select interobserver agreement. Thirty-one respondents selected “interobserver agreement” and did not advance. The second question was designed to confirm eligibility and asked participants if they were currently providing or supervising ABA services. To advance, the respondent needed to indicate that they were supervising and/or providing services. If they selected “neither,” Qualtrics advanced to the end. Ten respondents were unable to participate following this question. After the exclusion criteria were applied, a total of 203 participants completed the survey.

Survey

Participants continued to the 32-question survey that included questions about the participant’s demographic information, their training histories related to procedural fidelity, how often procedural fidelity was assessed in practice, the methods of procedural-fidelity assessment, and barriers to assessing procedural fidelity in practice (see Supplemental Information for a copy of the survey questions and response options). Participants could skip all questions in the survey—thus, some questions had fewer than 203 responses—and provide multiple responses on some questions (denoted with an asterisk in Tables 2 and 3)—thus, some questions had more than 203 responses. On the basis of their answers, some participants were asked fewer than 32 questions (Qualtrics survey logic available in Supplemental Information).

Table 2.

Participant demographic information

Variable	n (%)	Variable	n (%)
Age (n = 146)		Service setting(s) (n = 307)*
25–29	29 (19.9)	Center/clinic-based	93 (30.3)
30–34	36 (24.7)	Home-based	75 (24.4)
35–39	28 (19.2)	Public school	47 (15.3)
40–44	16 (11.0)	Community-based	46 (15.0)
45–49	8 (5.5)	Private school	26 (8.5)
50–54	16 (11.0)	Other	20 (6.5)
55–59	6 (4.1)	Supervision of ABA services (n = 145)
60 +	7 (4.8)	Yes	128 (88.3)
Highest degree obtained (n = 145)		No	17 (11.7)
Master’s	119 (82.1)	Number of staff on supervision load (n = 128)
Doctorate	26 (17.9)	1–2	14 (10.9)
Area/discipline of highest degree (n = 145)		3–5	33 (25.8)
Behavior analysis	77 (53.1)	6–10	47 (36.7)
Special education	37 (25.5)	11 +	34 (26.6)
Psychology	21 (14.5)	Number of clients on caseload (n = 128)
General education	4 (2.8)	0–5	32 (25.0)
Other	6 (4.1)	5–10	40 (31.3)
Years of experience as BCBA/BCBA-D (n = 144)		10–15	26 (20.3)
0–5	80 (55.6)	15–20	11 (8.6)
6–10	31 (21.5)	20 +	19 (14.8)
11–15	16 (11.1)	Training on Procedural Fidelity (n = 167)
16 +	17 (11.8)	Yes	149 (89.2)
Current position(s) (n = 202)*		No	18 (10.8)
Clinical supervisor	68 (33.7)	Modality of Training (n = 479)*
Clinical director	32 (15.8)	University coursework	116 (24.2)
Consultant	32 (15.8)	Continuing education	98 (20.5)
Case manager	25 (12.4)	Practicum or fieldwork	96 (20.0)
Researcher	15 (7.4)	Independent study	94 (19.6)
Professor	14 (6.9)	On-the-job	69 (14.4)
Teacher	4 (2.0)	Other	6 (1.3)
Other	12 (5.9)
Population(s) served (n = 249)*
Children (0–17 years) with disabilities	137 (55.0)
Adults (18 + years) with disabilities	56 (22.5)
Children (0–17 years) of typical development	24 (9.6)
Individuals with brain injury	15 (6.0)
Elder adults (65 + years)	11 (4.4)
Adults (18 + years) of typical development	4 (1.6)
Other	2 (0.8)

A total of 203 surveys were completed. All questions could be skipped. Percentages are rounded to nearest tenth. Asterisks denote questions in which participants could select multiple options

Table 3.

Variables related to assessing procedural fidelity in practice

Variable	n (%)	Variable	n (%)
Data collector (n = 213)*		Interventions to reduce behavior (n = 266)*
Respondent	109 (51.2)	Reinforcement-based	104 (39.1)
Staff member respondent supervises	67 (31.5)	Antecedent-based	101 (38.0)
Staff member respondent does not supervise	19 (8.9)	Punishment-based	55 (20.7)
Respondent’s supervisor	14 (6.6)	Other	6 (2.3)
Other	4 (1.9)	Interventions to increase behavior (n = 355)*
Use of direct assessment (n = 122)		Discrete-trial instruction	93 (26.2)
Yes	121 (99.2)	Naturalistic teaching	93 (26.2)
No	1 (0.8)	Chaining	84 (23.7)
Observation method (n = 147)*		Shaping	83 (23.4)
Live observation	118 (79.7)	Other	2 (0.6)
Non-live video/audio recording	29 (19.6)	Training (n = 76)*
Direct assessment method (n = 302)*		Staff/teacher	42 (55.3)
Specific checklist	63 (20.9)	Parent/caregiver	32 (42.1)
Generic checklist	60 (19.9)	Other	2 (2.6)
Trial-by-trial component checklist (occurrence/nonoccurrence)	45 (14.9)	Assessment (n = 104)*
All-or-none component for complete session/program	43 (14.2)	Skill-based	35 (33.7)
All-or-none trial-by-trial checklist	28 (9.3)	Functional assessments or analyses	35 (33.7)
Rating scale	19 (6.3)	Preference	34 (32.7)
Frequency of trials or opportunities implemented correctly	18 (6.0)	Telehealth (n = 46)*
All-or-none for complete session/program	14 (4.6)	Staff	24 (52.2)
Tool in electronic data system	9 (3.0)	Parent/caregiver	21 (45.7)
Other	3 (1.0)	Other	1 (2.2)
Use of indirect assessment (n = 113)		Use of PF data in practice (n = 229)*
Yes	34 (30.1)	Feedback and training	105 (45.9)
No	79 (69.9)	Assess treatment effectiveness	78 (34.1)
Indirect assessment (n = 68)*		Insurance or quality assurance	39 (17.0)
Permanent products	27 (39.7)	Other	7 (3.1)
Supervisee interviews	17 (25.0)	Collect PF data in one setting (n = 162)
Supervisee self-reports	15 (22.1)	Yes	127 (78.4)
Lesson plan reviews	7 (10.3)	No	35 (21.6)
Other	2 (2.9)	Collect PF data across all settings (n = 112)
Procedures/activities with PF data (n = 333)*		Yes	88 (78.6)
Interventions designed to reduce behavior	110 (33.0)	No	24 (21.4)
Interventions designed to increase behavior	109 (32.7)	Settings without PF data (n = 34)*
Staff/caregiver training	44 (13.2)	Home/residential-based	12 (35.3)
Assessments	41 (12.3)	Clinic/center/hospital-based	6 (17.7)
Telehealth	29 (8.7)	School-based	9 (26.5)
		Community-based	7 (20.6)

A total of 203 surveys were completed. All questions could be skipped. Percentages are rounded to nearest tenth. Asterisks denote questions in which participants could select multiple options. PF procedural fidelity

The authors met via Zoom to design the survey questions and supporting materials. Fallon et al. (2020) was used to inform some of the questions, and the authors called upon their experiences in clinical application, research, and university settings to develop additional questions and examples. We created examples of direct-assessment methods (see Table 1) for Question 12 based on the literature (e.g., Carroll et al., 2013; Kodak et al., 2018) and our experiences assessing procedural fidelity in behavior-analytic research and practice. The examples accompanied the names of the methods in the survey and were meant to help respondents identify which measurement systems were akin to those used in their practice, even if they were referred to by a different name. The examples included a brief description of the measurement system (e.g., “A summary of how the therapist implemented each trial or opportunity of the intervention for the session”). The first author drafted the survey questions and measurement examples, and the co-authors reviewed the drafts and provided feedback. This process was completed multiple times until all authors were satisfied with the survey questions and format. Following approval from all the authors, the first author asked several students to test the survey to identify issues and determine the approximate time required to complete the survey. Thereafter, the survey was distributed via email.

Table 1.

Examples of direct assessment data sheets provided in the survey

The names used to refer to the direct-assessment methods were meant to draw attention to the level at which fidelity was assessed (i.e., whole session/program, trial, component), whether the measure was dichotomous (i.e., yes or no for all-or-none), and whether the method was customized or generic. An example of a specific checklist was not included because we did not reference a specific behavior-intervention plan. The images were examples and not meant to be exhaustive. Respondents could select multiple responses for this question

Data Analysis Plan

The distribution of participant responses to survey questions was used to assess trends in procedural fidelity. Response distributions for each question were derived from relative-selection percentages. Relative-selection percentages were calculated based on the total number of participants who selected a response divided by the total number of participants who answered the question and multiplied by 100. For multiple-response questions, relative-selection percentages were calculated based on the number of selections divided by the total number of responses to the question multiplied by 100.

Results

Respondent Demographics

We received 203 completed surveys. Not all participants chose to provide demographic data. We did not ask about sex, gender, race, or ethnicity. The majority of participants were younger than 40 years old (63.9%), credentialed at the master’s level (82.1%), held a degree in behavior analysis (53.1%), and had five or fewer years of experience as a BCBA (55.6%; see Table 2 for a summary of participant demographic information). The years of experience for our respondents were similar to those who participated in the survey by Fallon et al., (2020; 52.4% with 0–5 years). They were likely to occupy positions of clinical supervisors (33.7%), clinical directors (15.8%), consultants (15.8%), and case managers (12.4%). A smaller proportion of participants worked as researchers (7.4%), professors (6.9%), and teachers (2%; respondents could, and did, select multiple options for current positions). A majority of participants served individuals with disabilities, with 55% serving children and 22.5% serving adults. Although we did not ask about specific diagnoses, it is likely that many respondents provided and supervised ABA services for individuals diagnosed with ASD, given that 75.4% of BCBAs report ASD as their primary area of professional emphasis (BACB, n.d). Respondents worked in a variety of settings, with center/clinic-based (30.3%) and home-based (24.4%) service settings being the most common, and some respondents worked in multiple settings. Some respondents (6.5%) said they worked in settings other than those provided in the survey, including hospitals, residential settings or group homes, state institutions, childcare settings, nursing facilities, and universities. The majority of respondents supervised ABA services (88.3%). Respondents providing supervision were most likely to report that they supervised at least six supervisees (63.3%), and the caseload size varied from a few clients (0–5; 25%) to more than 20 clients (14.8%) with the majority of respondents’ client caseloads between 0 and 15 (76.6%).

Nearly all of the respondents (89.2%) reported having received training on procedural fidelity, and the training spanned various experiences, including university coursework, continuing education, practicum or fieldwork, independent study (i.e., reading articles, chapters, and books), on-the-job training, and other experiences (e.g., conducting research on procedural fidelity, teaching the material to others).

Variables Related to Procedural-Fidelity Assessment in Practice

Approximately half of the respondents (51.2%; see Table 3) indicated that they, the BCBAs who provide and supervise ABA services, are the individuals who collect procedural-fidelity data. Some respondents said that their supervisors collect fidelity data (6.6%). Respondents indicated that their supervisees (31.5%) and staff that they do not supervise (8.9%) collect fidelity data. When fidelity is assessed in practice, nearly all of the respondents (99.2%) indicated that they collect procedural fidelity data using direct-assessment methods. Direct observation is the mechanism by which data for direct assessments are obtained, and in the current study, most respondents observed live sessions in person or via a streaming feed (79.7%), and others used recorded video and audio clips (19.6%).

When procedural fidelity is monitored with direct assessments, the respondents use a variety of methods. The most commonly used methods from the options provided (see Table 1 for examples used in the survey) were a specific checklist (i.e., a component checklist that is customized for the consumer and/or program; 20.9%) and a generic checklist (i.e., a component checklist that does not include individual consumer modifications but may represent a “standard” intervention used by individuals at that practice; 19.9%). Some participants reported that they collect fidelity data with a trial-by-trial component checklist (i.e., every component is recorded at every opportunity to indicate whether it was implemented correctly at that moment; referred to by multiple names, including trial-by-trial in Suhrheinrich et al., 2020; occurrence/nonoccurrence in Bergmann et al., 2023 and Kodak et al., 2023; modified discrete-trials teaching evaluation form in Cook et al., 2015; 14.9%) or all-or-none component fidelity for the complete session or program (i.e., a dichotomous measure of whether each component was implemented correctly throughout the observation period; 14.2%).

Fewer than 10% of respondents indicated that they use the other direct-assessment methods included in the survey. Of note, only 9.3% of respondents said that they collect data by trial using an all-or-none trial-by-trial measure (i.e., each trial is marked as correct or incorrect depending on whether any errors occurred for any of the components; referred to as whole-session fidelity by Brand et al., 2018). Few respondents said that they use rating scales (i.e., provide a number to indicate how often a component was implemented correctly across the observation period or how well all of the components within a trial were implemented; 6.3%) or frequency of trials or opportunities implemented correct (i.e., tally the number of times a component is implemented correctly during an observation period; 6%). An even smaller proportion of respondents said that they use an all-or-none measure for the complete session or program (i.e., a single datum to reflect whether all of the components of a program were implemented correctly during the observation; 4.6%), a tool built into their electronic data collection system (we did not ask them to describe the tool although some respondents named the platform; 3%), or some other method (1%). We asked respondents who used direct assessments what percentage of supervision sessions included collecting procedural-fidelity data. Most of the 119 respondents who selected an answer said that they collected fidelity data with a direct assessment for 20% or fewer supervision sessions (63%, Fig. 1). Very few (8.4%) respondents said that they collect procedural-fidelity data using a direct-assessment method for more than 80% of supervision sessions.

Fig. 1.

Percentage of sessions with procedural-fidelity assessment. Note. A total of 119 participants provided responses to the direct-assessment question, and 34 participants provided responses to the indirect-assessment question. We did not define what constituted a “session” in the survey question

In addition to direct assessment via direct observation, BCBAs can assess procedural fidelity in practice by using indirect means. This is less common than direct assessment as only 30.1% of the participants (Table 3) said that they use any indirect assessments in practice. Just like with direct assessment, there are multiple methods for indirect assessment, and our results suggest that respondents are likely to use more than one method. The most frequently used method for indirect assessment was using a permanent product (i.e., observing aspects of the environment to gauge whether an intervention or procedure was implemented correctly; e.g., reviewing a completed data sheet with items ranked in order of selection for a multiple stimulus without replacement preference assessment; 30.1%). Respondents may use supervisee interviews (i.e., asking the behavior-change agent about their implementation of the procedures; e.g., “How many items did you include in the preference assessment?”; 25%) and supervisee self-reports (i.e., the behavior-change agent records, rates, or reports the degree to which they implemented the procedures correctly; e.g., the behavior-change agent writes a “ + ” to indicate that they conducted the preference assessment correctly; 22.1%). Respondents who use indirect assessment to evaluate procedural fidelity in practice were presented with a question to estimate the percentage of supervision sessions that include at least one of these methods. Most of the 34 respondents said that they used indirect assessments for 20% or fewer sessions (70.6%, Fig. 1). Very few (5.9%) respondents indicated that they collected procedural-fidelity data with an indirect assessment for more than 80% of sessions.

Regardless of the use of direct or indirect assessments, we asked respondents to select the activities and procedures for which they analyze procedural fidelity (Table 3). We found that respondents were more likely to collect data on behavior-change interventions (i.e., interventions designed to reduce behavior and interventions designed to increase behavior combined; 65.8%) than staff/caregiver training (13.2%), assessments (12.3%), and telehealth (8.7%). Thirty-three percent of respondents analyze the fidelity of interventions designed to reduce behavior and are more likely to assess the fidelity of reinforcement-based (e.g., differential-reinforcement procedures; 39.1%) and antecedent-based strategies (e.g., prompts, motivating operations manipulations; 38%) than punishment-based procedures (e.g., response cost, time-out; 20.7%) or other procedures (2.3%). Nearly one-third of respondents assess fidelity for interventions designed to increase behavior and assess fidelity for DTI (26.2%), naturalistic teaching procedures (26.2%), chaining procedures (23.7%), and shaping procedures (23.4%) at similar levels. Respondents who collected procedural-fidelity data during training were most likely to do so during staff/teacher training (55.3%) with parent/caregiver training not far behind (42.1%). If respondents evaluated the fidelity of assessments in practice, they were just as likely to assess skill-based assessments (33.7%), functional assessments/analyses (33.7%), and preference assessments (32.7%). Over half of the respondents who analyze fidelity during telehealth appointments collect data during staff sessions (52.2%), and just under half collect data during parent/caregiver sessions (45.7%).

When asked how procedural-fidelity data are used in ABA practice, some respondents selected multiple uses (Table 3). The respondents were most likely to endorse that these data are used to provide feedback and training to supervisees (45.9%) and to assess treatment effectiveness (34.1%). A smaller proportion of respondents (17%) indicated that procedural-fidelity data are used for insurance or quality-assurance purposes.

Respondents provided and/or supervised ABA in a variety of settings, and a large proportion of respondents (78.4%; see Table 3) indicated that they collect procedural-fidelity data in at least one setting, and a similar proportion said that they collect procedural fidelity data across all settings in which they work (78.6%). Nevertheless, there were respondents who reported that they did not collect procedural fidelity in at least one setting (21.6%). The service settings in which some respondents do not collect procedural-fidelity data include home/residence (35.3%), school (26.5%), community (20.6%), and center/clinic/hospital (17.7%). It could be that some settings present more barriers to assessing fidelity than others. Several respondents indicated that the settings in which they work are not conducive to collecting fidelity data (5.9%, Fig. 2) or that collecting fidelity is too intrusive for the setting (5.3%, Fig. 2). However, barriers related to the setting were selected less often than other barriers to procedural-fidelity assessment in practice.

Fig. 2.

Barriers to analyzing procedural fidelity in practice. Note. Respondents could select multiple options when answering this question

The most commonly selected barrier was related to a lack of resources (e.g., equipment, staff, time; 28.9%, Fig. 2); related barriers of limited supervision time and their caseload being too heavy were endorsed by 13.2% and 10.5% of respondents, respectively. One-fifth of the respondents indicated that procedural-fidelity assessment was not required by their place of employment. Very few respondents said that they did not assess procedural fidelity in their practice because data were not useful to them (1.3%).

Discussion

In total, we received 203 completed surveys. We could not calculate a response rate given the method of survey distribution and the use of anonymous links. In 2022, there were 59,976 BCBA certificants (BACB, n.d). Therefore, we sampled only 0.3% of the overall population of people who hold a BCBA credential. Although this sample size is similar to previous surveys on procedural fidelity in practice (e.g., Dueker & Desai, 2023), these respondents may not represent the larger body of practicing individuals. Additionally, it is likely that respondents in our study and other surveys on procedural fidelity are more likely to be completed by BCBAs who are familiar with procedural fidelity, consider it in their practice, and think it is important. A large proportion of the participants indicated that they had received training related to procedural fidelity, so it is possible that the responses given by respondents in the current study may not be indicative of patterns of behavior of all BCBAs due to self-selection bias. Therefore, the results of the survey should be interpreted with those limitations in mind.

Respondents in the survey by Fallon et al. (2020) agreed that they received sufficient training related to assessing and monitoring procedural fidelity in their practice. Respondents in Dueker and Desai (2023) agreed that they were taught how to calculate fidelity in their training program but were neutral about whether they could use more training. We did not ask about the perceived quality of the training experiences of respondents in the present survey. Nearly 90% of respondents in our survey said they received training on procedural fidelity, but it could be that the training was insufficient or did not help them integrate procedural-fidelity data into practice. That is, training on the importance of procedural fidelity in research and practice may not lead to increases in fidelity assessment if the training does not include content on how to create measurement systems, interpret fidelity data, and use fidelity data in treatment decisions.

Who Collects Procedural-Fidelity Data?

The respondents in the survey said that they or their direct supervisees were the most likely people to collect procedural-fidelity data in their practice. Practice guidelines such as those provided by the Council for Autism Service Providers (CASP, 2024) list monitoring procedural fidelity as a direct-supervision activity specifically assigned to behavior analysts in a tiered treatment-delivery model; however, it could be beneficial for others, like supervisees, to collect fidelity data. Training supervisees to collect fidelity data could have multiple benefits, including assessing procedural fidelity for more sessions; giving the supervisee opportunities to practice an important task-list item (BACB Task List H-6; BACB, 2017) while earning unrestricted hours, which must comprise 60% of their hours (BACB, 2024); and preparing supervisees to become supervisors after certification.

Regardless of who is collecting procedural-fidelity data—a BCBA, a supervisee pursuing fieldwork, a teacher, a paraprofessional, or caregivers—they should be trained to ensure accurate and reliable data collection. Because a BCBA may have limited time to work directly with each data collector, it may be beneficial to use asynchronous training (e.g., video modeling; Marano et al., 2020). The training should teach observers to discriminate between correct and incorrect performance based on the operational definitions in a protocol, and the reliability of their observations should be verified through periodic checks, as observer drift can occur.

Observer drift is important to monitor, but there are other variables that may affect the accuracy of observers’ procedural-fidelity data. One variable is the level of fidelity itself. For example, Aguilar et al. (2023) found that the reliability of an observer’s data may decrease as fidelity decreases. Another variable is whether the observer is required to provide feedback to the implementer on their performance. Performance feedback, especially verbal feedback provided by a supervisor, is commonly used to provide ongoing training to staff (Blackman et al., 2023). Nevertheless, the requirement to provide performance feedback to a behavior-change agent could influence whether fidelity data are recorded correctly. For example, Matey et al. (2019) showed that the accuracy of participants’ data on the dependent variable decreased when the participants were required to provide feedback to the confederate, and Matey et al. (2021) found that the data collected were less accurate when the confederate’s responses to feedback were negative compared to more neutral responses. A follow-up study by Matey et al. (2024) found that data accuracy increased if the confederate’s performance improved following feedback. Because multiple variables such as those described above could influence an observer’s data collection, reliability checks for procedural fidelity should be incorporated into supervision sessions, and a supervisor or colleague should check a BCBA’s fidelity data collection as well.

How are Procedural-Fidelity Data Collected?

Direct Observation and Assessment

Direct assessments obtained via direct observations are recommended for fidelity data collection and analysis (Gresham et al., 2000). Thus, it was promising to see that 99.2% of the respondents in the current study used direct assessments, and the majority (79.7%) obtained these data via live observations either from in-person sessions or streamed video. This proportion exceeded the one reported by Fallon et al., (2020; 88.8%). During these direct observations, BCBAs are most likely to use a checklist to assess procedural fidelity. Although there are multiple ways to describe a checklist (e.g., Barton et al., 2018; Kodak et al., 2023; Morris et al., 2024), a checklist includes the procedure broken down into steps or components that are operationally defined (i.e., task analysis; Gresham et al., 2000), and the observer marks whether the component was implemented correctly (i.e., dichotomous variable with only two options per cell [i.e., yes or no, correct or incorrect]; Barton et al., 2018; Kodak et al., 2023; see Morris et al., 2024, for an example of a checklist that included dichotomous and continuous variables with different dimensions measured; see Aguilar et al., 2023, for a checklist that included component frequency). When we combine the direct assessment options that we provided to include all checklists that are broken down by component and involve dichotomous outcomes (i.e., specific, generic, and all-or-none component fidelity checklists), we see that these are used by over half of the respondents (55%). One of the reasons for this relative preference might be that checklists may be reasonably efficient and less likely to mask component errors than other fidelity measurement systems (Bergmann et al., 2023; referred to as all-or-none by component by the authors).

Respondents indicated that they are just as likely to use a generic checklist (i.e., a general checklist used for a program that does not include components that have been modified for an individual’s specific programming) as a specific checklist (i.e., customized to the client and/or program). Whereas a generic checklist may include the component “implements error-correction procedure” (i.e., generic for all error-correction procedures used across clients), a specific checklist may include the component “re-presents the target stimulus along with an immediate vocal model, allows 3 s for an echoic, and re-presents the target with an independent opportunity to respond” (i.e., specific to a client’s re-present until independent error-correction procedure). A potential barrier to procedural-fidelity assessment in practice is the time that it takes to create individualized fidelity data sheets, so using a generic checklist may reduce this barrier. Nevertheless, the time it takes to create a specific checklist may be worth the investment. Data from specific checklists provide more information about the implementation of components that are designed to address a client’s needs and retain the spirit of individualized interventions rather than one-size-fits-all approaches to ABA. Additionally, if a specific checklist incorporates dimensions other than accuracy, the most critical intervention components may be captured (e.g., the duration for the access-to-reinforcer component, the latency for the prompt-delay component; see Morris et al., 2024, for a tutorial). Other benefits of specific checklists for fidelity could be improvements in detecting errors and providing more personalized positive and corrective feedback, but these benefits are tentative and worthy of investigation.

One of the reasons that BCBAs may gravitate toward using checklists in practice is that they yield data on individual components of an intervention. That is, checklists can be used to identify procedural components that are being implemented well and those that need improvement. Thus, component data can help with decisions about effective treatment components and staff training. However, dichotomous (all-or-none by component in Bergmann et al., 2023; all-or-nothing in Kodak et al., 2023) checklists do not inform the BCBA about when or how many errors occurred for each procedural component, nor do dichotomous checklists permit the detection of incremental change (i.e., the behavior-change agent has to implement the component correctly on all opportunities before the observer marks that component correct). Sensitivity to incremental decrements and improvements in fidelity may be important for the BCBA evaluating the effects of their supervision, and incremental improvement could serve as a conditioned reinforcer for both the BCBA and the behavior-change agent.

A direct-assessment method that is more sensitive to incremental change is trial-by-trial component fidelity (i.e., collecting data on whether each procedural component was implemented correctly on each opportunity), which is considered the gold standard for fidelity assessment (Gresham et al., 2000). Although this method is likely the most robust fidelity data collection method (see Han et al., 2023, for a depiction of possible aggregation and calculations using this method; see Brand et al., 2018, for an alternative option that considers sequence errors with a Markov Transition Matrix), only 14.9% of survey respondents said they use this method to measure fidelity. There are likely multiple variables that account for the relatively infrequent use of this method, but the use of live or streamed sessions for direct observation may play a role. Studies that employed this measurement system typically use video recordings for data collection (e.g., Bergmann et al., 2023; Breeman et al., 2020; Cook et al., 2015; Suhrheinrich et al., 2020; cf. Carroll et al., 2013, and Kodak et al., 2018), and it can be challenging to collect these data without pausing and rewinding a video. Additional studies are needed to determine whether the trial-by-trial component method can be completed during live supervision sessions and still yield accurate and reliable estimates of fidelity.

Indirect Assessment

Compared to direct assessments, a much smaller proportion of respondents (30.1%) said that they use some form of indirect assessment to evaluate procedural fidelity in practice. Of the respondents who use indirect assessments, the most used method is permanent-product review (39.7%), which involves observing the environment after the procedure has been conducted to determine the extent to which it was implemented correctly. For example, a BCBA using permanent-product review to assess a teacher’s implementation of the Good Behavior Game would examine the record form to determine whether the teacher wrote down the winning team’s reinforcer, tallied instances of inappropriate behavior, and calculated group scores (Gresham et al., 2017). Compared to the respondents in Fallon et al. (2020; i.e., 66.5%), the respondents in our study were less likely to use permanent products. This could be because the respondents in Fallon et al. supervised caregivers’ implementation of ABA in their homes, whereas participants in our study could practice in any location and with any population.

Although permanent-product review can be advantageous because fidelity can be assessed without observing implementation, this can also be problematic as there is no guarantee that the program was implemented correctly by the behavior-change agent when the permanent product(s) is(are) intact. For example, a behavior-change agent may be tasked with using backward chaining to teach a resident in a group home how to make their favorite food, chocolate pudding. Although the chocolate pudding may be in the fridge when the observer checks, it is possible that the behavior-change agent completed all of the steps of the chain for the resident, used total-task presentation instead, skipped ingredients/steps, or something else. Another limitation is that permanent-product review can only capture procedures or components that create a lasting, observable change in the environment. When Gresham et al. (2017) compared permanent products to other assessment methods, they could only assess three of the seven components of the Good Behavior Game with permanent products. Including a subset of components could be an issue because of potential blind spots regarding fidelity and estimates that may not align with direct assessment via direct observation. In Gresham et al., permanent-product review had the lowest mean fidelity and largest spread of scores (i.e., M 67%, SD 34%) compared to data obtained via direct observation (i.e., M 79%, SD 16.4%) or self-report (i.e., M 97%, SD 6.3%).

Of the respondents in the current study who use indirect assessments, 25% use supervisee interviews and 22.1% use self-report. Supervisee interviews can be structured or unstructured, but they generally involve a supervisor asking how the supervisee implemented the behavior-change procedure across a specific period (e.g., daily, weekly). Using interviews may not lead to improvements in fidelity (Noell et al., 2005). Similar to interviews, self-reports of procedural fidelity involve the behavior-change agent reporting, rating, or recording the extent to which they implemented the procedures. Because a behavior-change agent may score their performance similarly across multiple sessions, self-reports may be “extremely reliable” (Gresham et al., p. 118); however, behavior-change agents are likely to overestimate their fidelity. Compared to the sample in Fallon et al. (2020), respondents in the current study were much less likely to use supervisee interviews (71.6% vs. 25%) and supervisee self-reports (67.9% vs. 22.1%). Again, these differences could be related to the practice restrictions for participant recruitment in Fallon et al. In a variety of practice settings, it could be convenient to use interviews and self-reports with supervisees to assess fidelity, but the outcomes from Noell et al. (2005) and Gresham et al. (2017) suggested that supervisors should refrain from using interviews and self-report measures as a primary measure of fidelity. It is possible that fidelity values collected via interviews and self-reports do not accurately describe the supervisee’s performance, especially given the potential for conditions that may increase the likelihood of inaccurate reporting. For example, supervisees reporting to their fieldwork supervisor may be inclined to say that they implemented the behavior-change procedures with high fidelity to appear competent, avoid criticism or reprimands, avoid additional training demands, lower the likelihood that the supervisor will not sign off on their hours, or make it seem like their supervisor’s supervision is effective. If interviews or self-reports are used to monitor procedural fidelity, they should be incorporated into packages that include direct assessments.

How Often Are Procedural Fidelity Data Collected?

Although there is no defined minimum requirement for the proportion of sessions that should include fidelity assessment, recommendations for research quality generally suggest that the standard for fidelity should be at least as rigorous as standards for interobserver agreement (Ganz & Ayres, 2018; What Works Clearing House, 2022). Practice guidelines suggest frequent fidelity checks, especially when staff are first hired, begin working with a new client, implement complex protocols, or provide intervention to clients with challenging behavior (CASP, 2024). Even without a specific guideline for the proportion of sessions that should include procedural-fidelity assessment, it should be conducted frequently enough to obtain values that are representative of the true fidelity of an intervention. In their comparison of self-report, permanent-product, and direct-observation measures of teacher fidelity implementing the Good Behavior Game, Gresham et al. (2017) evaluated how often fidelity needed to be measured to yield reliable estimates. Gresham et al. found that all three observation methods were reliable when fidelity was measured twice weekly for 5 weeks. The Good Behavior Game is designed to be implemented each school day, so the results of Gresham et al. suggest that fidelity should be assessed for at least 40% of sessions to produce a reliable estimate. In our survey, few respondents collect fidelity data via direct assessment (13.4%) or indirect assessment (14.7%) for at least 40% of supervision sessions. The majority of respondents reported that they measure fidelity on 20% or fewer sessions with direct (63%) or indirect (70.6%) assessments. Given these data, BCBAs may not be collecting fidelity data often enough to yield a reliable estimate of fidelity. Additional research is needed to determine how often fidelity needs to be assessed in practice to obtain reliable estimates.

For Which Procedures are Procedural-Fidelity Data Collected?

Reviews of procedural-fidelity assessment in the literature typically required that an article include the manipulation of at least one independent variable (i.e., an experiment) to be included (e.g., Han et al., 2023; McIntyre et al., 2007), but few reviews reported the type of procedure and likely excluded non-intervention procedures that are critical to behavior intervention (e.g., assessments; see Falakfarsa et al., 2022, and Preas et al., 2024, for exceptions). Therefore, we asked respondents to provide information about which practice activities they collect procedural-fidelity data. Respondents were most likely to collect procedural-fidelity data for behavior-change interventions and were just as likely to collect data when the interventions were designed to reduce behavior as those that were designed to increase behavior. When using behavior-change interventions to reduce behavior, procedural-fidelity assessment was more likely to occur with reinforcement-based and antecedent-based interventions than punishment-based interventions. Several parametric analyses on fidelity errors committed during differential-reinforcement procedures (i.e., differential reinforcement of alternative behavior, e.g., Foreman et al., 2023; Jones et al., 2023; St. Peter Pipkin et al., 2010) and punishment-based procedures (i.e., response cost, time out; e.g., Foreman et al., 2020; St. Peter et al., 2016) demonstrated that fidelity errors can affect the outcomes for participants. Thus, BCBAs should seek to measure the fidelity of all behavior-reduction procedures. It is possible that a number of the respondents who collect procedural-fidelity data for behavior-reduction programs are more likely to use reinforcement- and antecedent-based procedures than punishment-based procedures; thus, it may not be the case that fidelity is considered less often for punishment-based procedures. Future surveys could consider asking respondents if they use any punishment-based procedures with their clients and program survey logic to advance to a question about whether fidelity data are collected.

For interventions designed to increase behavior, respondents were just as likely to assess fidelity across DTI, naturalistic teaching, chaining, and shaping procedures. The diversity of contexts in which respondents reported collecting fidelity data contrasts with the distribution of contexts in most published studies. Although there are examples of fidelity measurement during naturalistic procedures in the literature (e.g., Suhrheinrich et al., 2020), much of the descriptive and experimental analyses of fidelity have focused on DTI (e.g., Bergmann et al., 2023; Breeman et al., 2020; Carroll et al., 2013; Kodak et al., 2018). To our knowledge, only one study has examined the effects of errors on learner behavior during behavior chains (Donnelly & Karsten, 2017). Thus, BCBAs should collect procedural-fidelity data on all types of skill-acquisition procedures, and additional research on procedural fidelity needs to be conducted to reflect the various teaching arrangements and procedures used in practice.

Respondents were less likely to collect fidelity data for assessments conducted in practice. Because reviews of procedural-fidelity reporting in behavior-analytic literature have largely focused on experiments or interventions, it is unclear how often fidelity data are collected on assessments. An exception is Falakfarsa et al. (2022), in which the authors coded assessment and interventions. They found that few studies consisted of an assessment alone (i.e., 6.3% of studies reviewed), but of those few studies, about one-third reported fidelity. Studies that included both an assessment and intervention phase were unlikely to report fidelity for both phases; that is, these studies were likely to include fidelity for the intervention phase only (i.e., n = 50, 89.3%). Respondents who assessed the fidelity of assessments were equally likely to collect procedural-fidelity data during skill-based assessments, functional assessments or analyses, and preference assessments. To date, descriptive analyses of procedural fidelity in practice have focused on interventions to reduce or increase target behaviors, so the fidelity of assessments is a blind spot. Nevertheless, multiple staff and caregiver training studies on strategies to teach and improve the fidelity of assessments suggest that different training experiences (e.g., instructions only, behavioral skills training) can produce different levels of fidelity, and individuals conducting the assessment can engage in errors (e.g., Graff et al., 2012; Marano et al., 2020; Pence et al., 2014; Wacker et al., 2013; Wallace et al., 2004). To our knowledge, the procedural fidelity for skills-based assessments has been reported for the PEAK Comprehensive Assessment (Dixon, 2019) only (M = 98.7%; range 89.6–100% in Sutton et al., 2022) but not for other assessments such as the Verbal Behavior Milestones and Placement Program (Sundberg, 2014) or the Assessment of Basic Language and Learning Skills (Partington, 2007). Considering how crucial assessments are to identifying consumer needs, evaluating progress, and reporting to third-party payors, the fidelity with which assessments are conducted should be evaluated.

During the COVID-19 pandemic, the use of telehealth for providing ABA services increased as providers rapidly transitioned from face-to-face to remote service delivery (Awasthi et al., 2021; Crockett et al., 2020; Pollard et al., 2021). A small proportion of respondents in our study collect fidelity data during telehealth interventions, and data are collected during staff- and caregiver-implemented services. Fallon et al. (2020) asked respondents who supervised caregiver-implemented sessions but did not include a telehealth option. Given the increased use of telehealth to deliver ABA interventions during the COVID-19 pandemic, future researchers should assess fidelity measurement for telehealth sessions.

Why are Procedural-Fidelity Data Collected?

On the basis of the results of the survey, BCBAs are most likely to collect procedural-fidelity data in practice to provide supervisees with feedback and training and to assess treatment effectiveness. These responses align with a BCBA’s ethical responsibilities to monitor supervisees’ performance and provide feedback (Standard 4.08; BACB, 2020) and the need to continuously evaluate behavior-change interventions (Standard 2.18; BACB, 2020). Procedural-fidelity data, especially when collected on specific components (Bergmann et al., 2023; Cook et al., 2015), can be helpful to trainers and supervisors. Component-level fidelity could help supervisors provide specific performance feedback to their supervisees or conduct re-training on only the components that need improvement rather than all components of the procedure, which should be more efficient. If the same components are implemented with errors across multiple staff or clients, the BCBA could develop or revise training materials as an antecedent intervention to prevent those fidelity errors in the future.

Using fidelity data as part of a BCBA’s assessment of treatment effectiveness is important to avoid false positives and false negatives (Gresham et al., 2000; Kodak et al., 2023). A false positive occurs when a BCBA assumes that the intervention as prescribed is responsible for changes in the dependent variable, but the intervention is not implemented with adequate fidelity. A false negative occurs when a BCBA assumes that an intervention is ineffective, but it could be if it were implemented with higher fidelity. Although the BACB Code of Ethics (BACB, 2020) specifies that behavior analysts need to continually evaluate the effects of a behavior-change intervention, it does not specifically require an assessment of procedural fidelity. Additionally, the Code of Ethics does not specify whether a BCBA should select appropriate data-collection measures for fidelity, ensure the reliability of procedural-fidelity data, nor incorporate procedural-fidelity data in Standard 2.17, “Collecting and Using Data” (BACB, 2020). Guidance and requirements in these areas could promote increased assessment of fidelity in practice.

In the current study, relatively few participants (17%) said that they assess procedural fidelity in practice to satisfy funder requirements or for quality assurance. The authors of this paper who supervise ABA services funded through medical insurance or other payors (e.g., public schools) have never been asked to provide fidelity data during the authorization process with various insurance companies nor during progress meetings or contract renewals with other payors. Similarly, in Dueker and Desai (2023) and Fallon et al. (2020), the participants disagreed with statements that funders required them to report procedural-fidelity data. Requiring fidelity measures as one marker of quality could improve outcomes for clients and substantiate the need for medically necessary ABA interventions.

Where are Procedural-Fidelity Data Collected?

The majority of respondents (78.4%) collect procedural-fidelity data in at least one setting, and many (78.6%) collect fidelity data across all settings in which they provide ABA services. However, there are respondents who do not collect fidelity data in at least one setting. These settings included home/residence, school, community, and center/clinic/hospital settings. Consumers should receive ABA interventions in environments wherein socially significant changes are meaningful (e.g., community, school, home), but it can be challenging to collect procedural-fidelity data in some settings. There could be multiple reasons why some settings are not conducive to collecting fidelity data. One reason is that BCBAs may spend less time in some environments providing supervision than others and may need to complete other tasks during this time. For example, a behavior-change agent providing services in an autistic child’s home 40 h per week may be supervised by a BCBA for as few as 2 h each week. During that time, the BCBA may observe only a handful of the clients’ programs and need to accomplish several tasks under healthcare insurance billing code 97155 (i.e., “Adaptive behavior treatment with protocol modification administered by a physician or other qualified health care professional, which may include simultaneous direction of technician, face-to-face with one patient;” ABA Coding Coalition, 2023), including modifying protocols, training on new programs, and observing the client’s behavior. These competing tasks may leave little room for assessing procedural fidelity, even though fidelity is important for protocol modification and training.

Another reason that fidelity is not assessed in some settings is that it would be too intrusive. This may be the case if the presence of other individuals in the therapeutic environment would compromise the outcomes of the intervention or may be limited for privacy or practicality. For instance, it may be that the presence of additional staff may be stigmatizing for the client if this makes them appear discrepant from their peers. Further, a caregiver may not be comfortable having an observer in the environment when their child is bathing or toileting, or it may be impractical for a BCBA to be present when working on sleep-related issues. Nevertheless, setting-related barriers should be addressed to the greatest extent possible to assess fidelity for at least some proportion of sessions. Indirect assessments such as permanent products or self-reports may be appropriate options. If direct observation is necessary or preferred, the BCBA may make use of recording equipment that is approved for data collection by all parties in the environment and is used, stored, and destroyed per ethical guidelines (Standard 2.03; Standard 2.11; BACB, 2020).

What are the Barriers to Collecting Procedural-Fidelity Data in Practice?

Setting-related barriers described above could help explain why the majority of the respondents (63% for direct assessment and 70.6% for indirect assessment) in the survey analyze procedural fidelity for less than 20% of sessions, but identifying other barriers to fidelity assessment in practice could generate possible solutions. Barriers may include limited time or resources, a lack of support or requirement from employers, and limited education or specific training on how to use fidelity-measurement tools and interpret fidelity values (Cochrane et al., 2019; Fallon et al., 2020; Sanetti & DiGennaro Reed, 2012). In our survey, 28.9% of respondents selected a lack of resources (e.g., equipment, staff, time) as a barrier to collecting fidelity data in their practice, and 13.2% selected limited supervision time. In addition to competing tasks during supervision described above, it can be effortful to analyze fidelity in practice. Interestingly, respondents in Dueker and Desai (2023) agreed that obtaining useful fidelity data only takes a few minutes and were neutral when asked if fidelity is time-consuming to measure. However, a recent comparison of measures of fidelity by Bergmann et al. (2023) found that all of the measures exceeded the duration of the DTI session when observers collected fidelity data. To support providers in the collection of fidelity data in their practice, procedural-fidelity methods that are quick and produce data that are accurate and reliable need to be developed and evaluated. For example, Likert scales may be easier to use than some direct-assessment measures yet be reliable and correlated with trial-by-trial measures (Suhrheinrich et al., 2020). Enthusiasm for Likert scales may need to be tempered, however, based on issues with overestimating fidelity and masking component errors (Bergmann et al., 2023). Another possibility is that providers do not have data-collection systems in place to reduce the effort associated with procedural-fidelity data collection in sessions, so a tutorial by Morris et al. (2024) could help address that need in practice.

Although 20.4% of respondents indicated that fidelity data are not currently required by their employer, a large proportion of respondents (79.6%) did not identify this as a barrier. For providers who do not have a requirement to collect fidelity data, it could be even more challenging to prioritize this task, especially when combined with the limited resources noted above. That is, when a supervisor is present at a session with staff or caregivers, there are often competing contingencies pulling the supervisor in different directions based on client needs. This may be especially true if BCBAs carry large caseloads or are expected to complete most of their case management and supervision activities as billable hours. Requirements to analyze procedural fidelity as part of the BCBA’s supervision activities are recommended (CASP, 2024) and should be considered by employers.

Limitations

The current study provides insights into BCBA’s procedural-fidelity assessment in ABA practice, but the study includes several limitations. Although we sent out emails using the BACB listserv, special interest group listservs, and social media to try to sample broadly, our sample represents a very small proportion of practicing BCBAs. We did not include important demographic questions (Jones et al., 2020), so we are unsure about how our sample represents the racial, ethnic, cultural, and linguistic diversity of behavior analysts. We only permitted individuals who currently served as supervisors or providers of ABA to participate, and 10 people were unable to proceed following this question. It is possible that those individuals could have answered questions based on their experience and contributed to the survey. Moreover, we required that participants pass a comprehension check before advancing, and this resulted in 31 people exiting the survey. The use of a comprehension question as a filter was designed to exclude participants who consider procedural fidelity synonymous with interobserver agreement, an issue described previously (e.g., Vollmer et al., 2008). These questions were meant to ensure that we recruited participants who had experience with procedural fidelity, but our recruitment and filtering procedures could have exacerbated the selection bias and skewed our results. Future researchers should include additional demographic questions (see Blackman et al., 2023, for an example) and review exclusionary criteria carefully.

Previous surveys on procedural fidelity (Cochrane et al., 2019; Dueker & Desai, 2023; Fallon et al., 2020) included questions with Likert or rating scales. We did not ask the participants to indicate how much they agreed or disagreed with different statements in the survey. Additionally, we allowed participants to select multiple responses on many questions. These decisions prevented us from computing correlations or inferential statistics, which limited our comparisons and statements about the influence of training, experience, settings, and other variables on the method and frequency of procedural-fidelity assessment. Future researchers should consider rating scales and programming surveys such that multiple responses can be gathered across questions rather than within a question.

Another issue with the survey is that we asked respondents to recall and estimate how frequently fidelity is assessed by using a proportion of sessions. However, we did not define “session” within the question. Thus, a respondent could consider a “session” to be a day, single supervision observation, implementation of one program in one day, or other possibilities. Different interpretations of what constitutes a session could significantly alter responses to questions about frequency. It could be beneficial to provide some additional parameters to help respondents estimate fidelity by session. For example, Fallon et al. (2020) specified that a session was a “one-to-one consultation” (p. 18). Future studies should specify what is meant by “session” when asking how often fidelity is assessed.

In this survey, we asked respondents about specific fidelity-assessment tools and provided examples. The tools were based on literature (e.g., Bergmann et al., 2023; Cook et al., 2015; Suhrheinrich et al., 2020) and the authors’ experiences, but it is possible that the descriptions were unfamiliar to some or most respondents. It is also possible that our examples (see Table 1) looked quite different from how the data-collection tools are designed in practice or were overly focused on trial-based skill-acquisition procedures. In the future, researchers may acquire exemplars of tools used in practice or conduct a descriptive assessment. Additionally, more uniformity in what constitutes a checklist and the difference between measurement systems and calculations in procedural fidelity would be helpful. Although behavior analysts are taught to develop conceptually systematic behavior-change procedures and assessments, to describe the procedures technologically, and to analyze the effects of dependent variables, the measurement of the implementation of the independent variable remains in a “relatively rudimentary status” (Collier-Meek et al., 2018, p. 518). Researchers and practitioners should bring the skills they use to define, measure, and analyze dependent variables to bear on the independent variable to enhance the sophistication of procedural-fidelity assessment.

Conclusion

We asked BCBAs about their experiences collecting procedural fidelity in practice. We received completed surveys from 203 BCBAs, and we learned that nearly all of the respondents received training on procedural fidelity, and most collected procedural-fidelity data with direct-assessment methods. Respondents indicated they were likely to analyze procedural fidelity across a variety of skill-acquisition and behavior-reduction programs. However, the majority of the respondents collected data for 20% or fewer sessions. Additionally, there were some settings in which they did not assess procedural fidelity, perhaps due to limited resources or challenges associated with those settings. When fidelity data were collected, they were most often used to train staff and evaluate treatment outcomes. Respondents’ reported barriers to collecting fidelity included access to resources such as time and staff, no requirement to collect fidelity data by their employers, and limited supervision time. To help combat these barriers, increased access to tutorials and resources that include data-collection systems that can be (a) used during observations (e.g., Morris et al., 2024), (b) lead to meaningful data for staff training and treatment evaluations, and (c) produce reliable estimates of “true” procedural fidelity would be worthy areas of future dissemination.

Supplementary Information

Below is the link to the electronic supplementary material.

Authors’ Contributions

All authors designed the survey. The first author obtained IRB approval and managed survey distribution. The first and second author analyzed the results. The first author wrote the first draft of the manuscript. All authors revised and approved the final manuscript.

Funding

N/A.

Data Availability

All data obtained from the survey are included in the manuscript.

Declarations

Ethics Approval

This study was approved by the human subject institutional review board at the University of North Texas. The study was performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Conflicts of Interest/Competing Interests

Samantha Bergmann serves on the editorial board for the journal. Jason Vladescu serves as an associate editor for the journal and is the incoming Editor-in-Chief.