An Investigation of BCBA Exam Pass Rates as a Quality Indicator of Applied Behavior Analysis Training Programs

Abstract

The number of colleges and universities offering behavior analytic training has grown rapidly over the last decade and it is time for our profession to evaluate the rigor of such trainings and to gain quality control. In this study we set out to investigate first-time pass rates on the Board Certified Behavior Analyst (BCBA) examinations across Verified Course Sequences (VCSs) using publicly available data between 2013 to 2020. We evaluated outcomes related to overall first-time pass rates; the small number of programs contributing the greatest number of BCBA exam candidates; and the difference in first-time pass rates by mode of instruction and ABAI accreditation. We discuss the use of first-time pass rate data as a quality indicator and provide some recommendations based on the outcomes evaluated.

In 2015, the journal Behavior Analysis in Practice hosted a discussion about quality indicators of training programs in applied behavior analysis (ABA). The impetus for this discussion was an article published by Dixon et al. (2015a) that examined the research publication productivity of faculty teaching in applied behavior analytic (ABA) training programs. Dixon et al.’s (2015a) article sparked a debate that resulted in several published responses (e.g., Ahearn et al., 2015; Arena et al., 2015; Bydenburg & Diller, 2016; Carr & Nosik, 2015; Critchfield, 2015; Detrich, 2015; Hayes, 2015; Iwata, 2015; Maguire and Allen, 2015; Pritchard & Wine, 2015; Reeve, 2015; Wilczynski, 2015; Wilder et al., 2015), a follow-up response from the authors (Dixon et al., 2015b), and a replication 3 years later (Blair et al., 2018).

Some authors acquiesced to the broader points made by Dixon et al. (2015a) and others disagreed with the premise that research productivity is a relevant factor in determining the quality of ABA training programs. Many of these authors suggested alternative indicators for determining the quality of the field’s training programs (e.g., job analysis studies, postgraduate publications and presentations, and employer and consumer surveys), including an analysis of the Behavior Analyst Certification Board (BACB)’s exam first-time pass rates for candidates completing coursework within ABA training programs (Ahearn et al., 2015; Arena et al., 2015; Carr & Nosik, 2015; Critchfield, 2015; Dixon et al., 2015a, 2015b; Iwata, 2015; Maguire & Allen, 2015; Pritchard & Wine, 2015).

In general, a training program’s first-time pass rate on a BACB exam indicates how successful the program was at preparing its students to pass an exam intended to measure minimal competence to independently practice ABA. It is important to note that this value does not account for the myriad of variables that could contribute to an individual candidate passing or failing a BACB exam on their first attempt. For example, some training programs primarily host students who work full-time jobs and may have less time to dedicate to their studies. Instead, a training program’s first-time pass rate is a summative value that, in general, informs prospective students about their likelihood of passing a BACB exam based on the past performance of its graduates. Given the emphasis placed on certification by employers, funders, and licensure boards, a program's ability to produce graduates capable of passing a BACB certification examination is an important outcome measure. The inability to achieve this credential means most of these students would be unable to independently practice behavior analysis.

The interest in BACB first-time pass rate data is not new. In 2017, Shepley et al. found that on-campus mode of instruction and ABAI accreditation were statistically significant variables correlated with higher pass rates based on published data from 2014. Guinness and Turner (2021), examining a more recent, larger data set (2015–2019), found similar statistically significant differences between campus and hybrid modes of instruction when compared against distance programs. These same authors also reported weak, but statistically significant, negative correlations between program size and pass rates, indicating an association between larger programs and lower first-time pass rates. In this article, we conducted a series of exploratory descriptive analyses using the BACB publicly available first-time pass rate data from 2013 to 2020 to determine if there were any relevant trends, within and across ABA training programs, that would help shed light on the quality of the training being offered.

The BACB

The BACB is a nonprofit corporation responsible for issuing the most widely recognized professional credentials for applied behavior analysts. The BACB currently offers two credentials and one distinction at the professional level and one certification at the paraprofessional level. The Board Certified Behavior Analyst (BCBA) credential and the Board Certified Behavior Analyst-Doctoral™ (BCBA-D) distinction, are held by graduate-professionals capable of independently practicing ABA.1 The second professional credential, the Board Certified Assistant Behavior Analyst (BCaBA), is an undergraduate-level certification held by professionals capable of practicing ABA under the supervision of qualified BCBA or BCBA-D certificants. Finally, the Registered Behavior Technician™ (RBT) certification is held by paraprofessionals responsible for the direct implementation of behavior analytic services under the ongoing supervision of qualified BCaBA, BCBA, or BCBA-D certificants.

The BACB is not the only professional credentialing body for behavior analysts, but it is currently the most prevalent and, it can be argued, the most relevant for providers practicing ABA.2 Since the BACB’s establishment in 1998, there has been significant growth in the number of providers seeking out a BACB credential (Deochand & Fuqua, 2016).3 As of July 1, 2020, there were 40,338 certificants holding a BCBA/BCBA-D credential, 4,365 certificants holding a BCaBA credential, and 76,825 providers holding a RBT credential (BACB, 2020a). Over the last 2 decades, BACB credentials have taken on increased importance in the field. Reimbursement for ABA health-care services, especially when treating those diagnosed with autism spectrum disorders, is often dependent upon a provider holding one of these credentials. Furthermore, with a few exceptions, providers must hold a BCBA and BCaBA credential in good standing to be licensed to practice behavior analysis in over 30 states (Carr & Nosik, 2017).4 These requirements have no doubt contributed to the almost 2,000% increase in job demand for providers holding a BCBA/BCBA-D credential between 2010 to 2020 (BACB, 2019a, 2021a).

The BACB Exam

Acquiring a BACB credential requires passage of an examination based off the content from the most current edition of the organization’s Task List (BACB, 2012a, 2013a, 2017a). The BACB’s Task Lists are updated every few years with the input of various stakeholders (e.g., credentialed providers, university faculty) in the profession. The Task Lists contain items spanning a variety of topical areas deemed relevant to the practice of behavior analysis by the professional community, including, but not limited to, principles, methods, assessments, interventions, and ethics.

The BACB uses the modified Angoff method to establish cut scores on their BCBA and BCaBA certification exams (BACB, 2020b). This criterion-referenced method is commonly used to set legally defensible exam cut scores that meet the standards of accrediting bodies like the BACB’s accrediting body, the National Commission for Certifying Agencies (NCCA). The modified Angoff method involves asking panels of trained subject matter experts (SME)5 to review exam items (questions) and rate the probability that a minimally competent candidate (MCC) would answer the items correctly (Assessment Systems, 2018). An MCC is best conceptualized as a candidate that possess the minimum knowledge necessary to perform at a level equitable with the credential they are being tested on. Candidates pass a BACB exam if they answer enough questions to meet or surpass the score determined by the SMEs. For example, at the time of writing, BCBA candidates need to score a 400 out of 500 (80%) to pass the BCBA exam (BACB, 2020b). It is important to note that passing a BACB examination only indicates that a candidate has met minimal levels of competence. In other words, the exam is not designed to assess the behavior analytic clinical skills or expertise of an exam candidate.

Certain eligibility requirements must be met before an individual can qualify to sit for a BACB examination. These requirements vary depending upon the type of BACB credential but can be split across three categories: degree, coursework, and supervised fieldwork. The exact qualifications necessary to sit for each certification examination are periodically updated by the BACB and details can be found on their website: www.bacb.com.

Verified Course Sequence Designation and Accreditation

Between 2001 and 2017, the BACB offered an Approved Course Sequence designation to undergraduate and graduate training programs containing course sequences that allowed students to meet the coursework eligibility requirements for their BCaBA and BCBA exams (BACB, 2012b). This designation was publicly shared with consumers through BACB-issued “Approved Course Sequence” (ACS) seals that were designed to be published on training program websites and marketing materials. In 2017, the BACB changed the title from “Approved Course Sequence” to “Verified Course Sequence” (VCS), presumedly to reflect the purpose of this designation more accurately (BACB, 2017c). In particular, the VCS designation is granted to training programs that meet certain coursework requirements, content hours, and faculty standards. Program coordinators apply for this designation by submitting course syllabi and faculty information for review (ABAI, 2018). It is important to note that the VCS designation is not a statement about the rigor or quality of a training program. Instead, it is best understood as a mechanism to inform prospective students about a training program’s potential to assist the student in meeting coursework eligibility requirements for the BACB’s BCaBA or BCBA certification exams. In 2019, the BACB transferred all VCS operations to the Association for Behavior Analysis International (ABAI) in recognition of its role in overseeing training standards (i.e., accreditation) in the field. It is important to note here that accreditation is completely different from verifying course content. In addition to review of syllabi for specific content hours and faculty credentials, accreditation is a peer-review process guided by standards and involves assessment of all coursework, the overall curriculum, faculty, research, experiential learning, administration, outcome measures of the overall program, and more (ABAI Accreditation Board, 2021). At present, programs that contain a VCS may apply for accreditation from ABAI, but it is not required for eligibility to take a BACB exam.6 Programs advertise to potential students that they offer training that meets the BACB’s coursework eligibility standards through ABAI’s VCS program designation.7 As of May 2019, there were 592 VCSs listed in ABAI’s directory, with the majority housed within the United States (n = 424; ABAI, 2019).8

In January 2012, the BACB (2012c) announced that training programs containing a VCS could begin publicly sharing pass rate data if six or more of their students had taken their examination in the last 3 years. Two years later, the BACB went a step further and began publishing BCBA and BCaBA examination first-time pass rate data for VCSs on their own website (BACB, 2013b). The most recent of these publicly available data sets include the total number and first-time pass rates of candidates per BCBA VCS between 2016 and 2020 (BACB, 2021b). At the time of writing, the BACB has now shared first-time pass rate data for 7 years (2013–2020) spanning the third and fourth editions of their task lists (BACB, 2005a, 2012a).9

By publishing these data sets, the BACB is sharing an important outcome measure to potential consumers of training programs hosting a VCS.10 It is important to note that the VCS pass rate data sets publicly shared by the BACB are incomplete. In particular, the BACB does not report data for a VCS in their first 4 years of operation and in general will not report data if a VCS had fewer than six first-time candidates in a given year.11 However, even with incomplete data sets, important trends may be identified that could partially capture the state of training in ABA. The purpose of this study was to analyze these data and identify relevant trends within and across graduate-level BCBA VCS training programs.12 Our exploratory analyses included the total number of first-time exam candidates completing a VCS; the percentage of candidates per VCS; the differences in first-time pass rates by mode of instruction; the percentage of candidates by overall VCS first-time pass rate; the potential impact of ABAI accreditation on first-time pass rates; the number of years a VCS was in operation; and the changes in first-time pass rates within individual VCS across time.

Method

Data Sample

To complete this study, we gathered published annual report data for the years 2016 to 2020 from the BACB website (BACB, 2019b, 2020e). These data included the total number of first-time candidates that took the BCBA certification examinations and whether or not the candidate attended a VCS. In addition, we obtained publicly available first-time pass rate data on the BCBA certification examinations from VCSs for the years 2013 through 2020 (BACB, 2018, 2019c, 2020b, 2020c, 2021b). These published data sets from the BACB included the names of the institutions hosting a VCS, their VCS identification number, and mode of instruction. They also included the number of first-time candidates and the first-time pass rates by year for each VCS that was in operation for at least 5 years.13

We combined and transcribed data from multiple BACB first-time pass rate publications into a single table in Microsoft Excel for analysis. Within the first-time pass rate publications, data are typically presented within a single year. For example, the first-time pass rate and number of first-time candidates for a single VCS might be shared for 2019. However, in instances when a single value for a VCS was reported across more than 1 year (e.g., 2019 and 2020), we assigned the value to the later year (e.g., 2020). In rare instances where data from overlapping years did not match across published data sets, we used the data from the most recent publication.14 It should be noted that these accommodations resulted in overall annual first-time pass rates that sometimes differed by 1 or 2 percentage points from those published by the BACB. The overall trends observed would not be affected by such small percentage point differences.

We rechecked values that were entered into Microsoft Excel (n = 5,122) independently against the values published by the BACB to ensure reliability. This check identified 1 error and 22 omissions that occurred during data entry. Following this check, a graduate student working with the first author independently reconstructed the data set using the original source files. We then compared the values against those constructed initially using comparison formulas in Excel. We found a 98.5% match between the data sets for programs reporting first-time pass rates. We analyzed and corrected any disagreements by revisiting the source data.15

Within the published data sets, there were 374 BCBA VCSs listed. However, of these VCSs, 40% (n = 151) had no reportable first-time pass rate data values to analyze. This is likely because these VCSs were in their first 4 years of operation or had fewer than six candidates that took their certification exam during a given year. For this study, we included only BCBA VCSs (n = 223) that had at least 1 reported year of first-time pass rate data.

Calculated Values

The BACB’s published annual report data from 2016 to 2020 included the total number of first-time BCBA certification exam candidates. An estimated number and percentage of first-time candidates that completed their coursework eligibility requirements from a VCS was calculated using these data in combination with the BACB’s published VCS first-time pass rates for the same years. This was accomplished by subtracting the total number of first-time candidates by the number of BCBA VCS candidates for 2016 through 2020. It is worth noting that these values are an underestimation as the BACB does not report the number of first-time candidates when a VCS had fewer than six candidates in any given year.

We calculated the total number of first-time candidates passing the BCBA exam by year and VCS by multiplying the first-time pass rates by the total number of candidates and rounding the product.16 We then calculated the total number of candidates failing the exam by year and VCS by subtracting the number of candidates passing by the total number of candidates in a given year. From these calculations, we determined the total number of first-time candidates and the total number of first-time candidates passing and failing the exam across VCS and year. Dividing the total number of first-time candidates by the first-time passing candidates produced an overall first-time pass rate by VCS across reported years. We calculated the total number of first-time VCS candidates per year by summing the total number of reported first-time candidates across VCS during a given year. We determined the percentage of overall annual first-time pass rates per year by dividing the sum total of all reported first-time candidates passing the BCBA exam against the total number of first-time candidates during a given year.

We conducted several other calculations to aid in the analysis of overall correlations. For example, we calculated the percentage of exam candidates per VCS by dividing the total number of candidates by VCS by the total number of candidates across all VCS. In addition, we determined the average number of first-time candidates per year by dividing the total candidates by the number of reportable years of data per VCS between 2013 and 2020. For example, if a VCS had 4 years of reportable data and a total of 100 first-time candidates, the average first-time candidates per year for that VCS was 25. Likewise, we analyzed the differences in first-time pass rate by mode of instruction by creating pivot tables in Microsoft Excel and examining overall VCS first-time pass rates17 and total candidates by mode of instruction. Moreover, we sorted and grouped VCS by their overall first-time pass rates to determine the percentage of candidates by VCS first-time pass rates. In addition, the first-time pass rates of ABAI accredited programs were determined by identifying and grouping VCS housed within ABAI accredited programs (https://accreditation.abainternational.org). Programs were included in the list as long as they held ABAI accreditation at some point during the 2013–2020 period. Furthermore, years of operation by VCS was determined by counting the number of years when first-time pass rate data was published for a VCS and the years when fewer than six candidates from a program took an exam. When first-time pass rate data overlapped across years, all overlapping years were included in the count. The first 4 years a VCS was in operation was not included in the count as the published data did not indicate when a VCS was established. Moreover, VCS that were shown to be in existence for over 8 years were excluded, because it was unclear how long these programs may have been in operation beyond the 8-year time frame (e.g., 8, 10, 15 years) making it difficult to make relevant comparisons. Finally, we calculated the slope for programs reporting 2 or more years of data to evaluate change in first-time pass rate within a VCS.

Results

We analyzed the BCBA first-time pass rate data across and within VCS across topical areas. We examined the potential impact of several variables during the 2013 to 2020 reporting period. This included the total number of first-time exam candidates completing a VCS; the percentage of candidates per VCS; the differences in first-time pass rates by mode of instruction; the percentage of candidates by overall VCS first-time pass rate; the correlation of ABAI accreditation on first-time pass rates; the number of years a VCS was in operation; and the changes in first-time pass rates within individual VCS across time.

Number and Percent of VCS Exam Candidates

We compared the total number of exam candidates against the number of exam candidates who completed their coursework eligibility requirements from a recognized VCS between 2016 and 2019. We found that almost all first-time BCBA candidates (99.97%) completed their coursework requirements from a VCS.

Overall First-Time Pass Rate and Total Number of VCS Candidates

In Figure 1 we depicted the overall annual pass rates and total number of BCBA candidates, respectively, across all VCSs between 2013 to 2020. BCBA certification exams offered in 2013 and 2014 were based on the BACB Third Edition Task List (BACB, 2005a) and between 2015 and 202 were based on the BACB Fourth Edition Task List (2012a). The transition to a new task list in 2015 corresponded with an increase in coursework eligibility requirements for both certification exams (BACB, 2012c). This increase in requirements may help explain the drop in level of first-time BCBA candidates between 2014 and 2015.

Figure 1.

The First-Time Pass Rates and Number of First-Time VCS Candidates Sitting for the BCBA Certification Examinations between 2013 and 2020. Note. BCBA certification exams in 2013 and 2014 were based on the BACB Third Edition Task List and in 2015 to 2020 were based on the BACB Fourth Edition Task List.

In Figure 1, first-time pass rate levels in 2013 and 2014 were equivalent at 59% and 60% respectively, with an 82% increase in the total number of BCBA VCS candidates from 2,089 to 3,809 across the years. First-time pass rates for BCBA VCS candidates between 2015 and 2020 was relatively stable with an overall first-time pass rate of 64% and a range of 63%–67%. During this same period, the total number of BCBA VCS candidates increased by 142% from 2,615 candidates to 6,338 at its peak in 2019.

Percentage of BACB Exam Candidates per VCS

In Table 1, we depicted the percentage and cumulative percentage of first-time BCBA certification exam candidates by number of VCSs. According to these data, BCBA exam candidates (n = 32,929) between 2013 and 2020 came from 223 VCSs. Of these, a single VCS was responsible for 15% of the candidates; two VCS were responsible for a quarter of exam candidates; nine VCSs were responsible for half of all exam candidates; and half of the VCSs (n = 94) were responsible for 95% of exam candidates. Of the 20 VCSs contributing the most BCBA exam candidates, only one VCS was housed within an ABAI accredited program between 2013 and 2020.18

Table 1.

Percent and Cumulative Percent of First-Time BCBA Certification Exam Candidates by Number of VCS between 2013 and 2020

Note. Gray shaded rows highlight important trends.

Differences in First-Time Pass Rate by Mode of Instruction

In Table 2 we provided the number of BCBA VCSs, percentage of first-time candidates, and first-time pass rates across modes of instruction. Data are only included in the table if a VCS had a least 1 year of published first-time pass rates between 2013 and 2020 (n = 223). VCS were categorized by the most recent reported mode of instruction. For example, if a VCS mode of instruction was listed as “Hybrid” in 2019 but changed to “Online” in the 2020 data set, then the most recent category was applied to that VCS. These data indicate that of the 223 VCSs included in the analysis, 39% (n = 86) were categorized as “Campus,” 14% (n = 31) were categorized as “Online,”19 27% (n = 61) were categorized as “Both,”20 and 20% (n = 45) were categorized as “Hybrid.” According to these data, 15% (n = 4,770) of BCBA exam candidates came from a Campus VCS; 56% (n = 18,523) came from an Online VCS; 20% (n = 6,674) came from a VCS categorized as Both; and 9% (n = 2,962) came from a Hybrid VCS. The first-time pass rates for candidates from Campus (73%) and Hybrid (73%) VCSs were over 10 percentage points higher than the first-time pass rates for candidates from Online (60%) and Both (63%) even though only 23% (n = 7,732) of first-time candidates graduated from these programs.

Table 2.

The Number of BCBA VCS, Percent of VCS, First-Time Candidates, and First-Time Pass Rates Across Modes of Instruction

Mode of Instruction	Number of VCSs	Percent of VCSs	Sum of Total First-Time Candidates (%)	Sum of Total First-Time Candidates Passing (Aggregated Pass Rate)
Campus	86	39%	4,770 (15%)	3,486 (73%)
Online	31	14%	18,523 (56%)	11,139 (60%)
Both	61	27%	6,674 (20%)	4,233 (63%)
Hybrid	45	20%	2,962 (9%)	2,150 (73%)
Total Number	223		32,929	21,009

Note. Data show the most recent reported modes of instruction. Only VCSs with at least 1 year of reportable data between 2013 and 2020 are included.

We conducted a chi-squaregoodness-of-fit test to determine if there was a statistically significant difference between the modes of instruction based on the number of first-time candidates passing the exam. We found that our statistic is statistically significant: χ²(3) = 34.52, p < .0001. The results of the chi-square analysis suggest that the mode of instruction (i.e., campus, hybrid, online, versus both) is associated with first-time candidates passing. The chi-square statistic only indicates that variables are associated. To determine how the variables are associated, we reviewed the crosstabs (see Table 3). The observed count is the observed frequency in a particular cell of the crosstabs table. The expected count is the predicted frequency for a cell under the assumption that there is no association between the modes of training and first-time pass rate. In this case, we set the expected first-time passing rate at 64% by dividing the total number of first-time candidates passing the exam over the total number of first-time candidates across modes. Therefore, the expected value was calculated by multiplying 64% by the total number of candidates who took the exam in that modality. In the cross tabulation, the more the observed versus expected values diverge from each other, the more the variables are associated with each other. The data suggests that assuming mode of instruction is not associated with first-time pass rates, the campus and distance learning modes diverge the most and in opposite directions, such that the campus pass rates are higher than expected whereas the distance pass rates are lower. Both campus and the hybrid modes did better than expected.

Table 3.

Chi-Square Goodness-of-Fit Test for VCS by Mode of Instruction

Mode of Instruction	Total First-Time Candidates	Total First-Time Candidates Passing	Expected First-Time Candidates Passing	χ2	df	Critical Valuefor p < 0.001	Chi-Squarep value
Campus	4,770	3,486	3043	34.5161	3	16.266	5.16742E-30
Online	18,523	11,139	11,818
Both	6,674	4,233	4,258
Hybrid	2,962	2,150	1,890
Total Number	32,929	21,009	64%

Differences in First-Time Pass Rate by Size of VCS

In Figure 2 we depicted the overall first-time pass rates on the BCBA exam by average number of first-time candidates across modes of instruction. These data suggest that regardless of the mode of instruction, most VCSs are relatively small in size as indicated by the average number of first-time candidates per year. In particular, between 2013 and 2020, the median number of average candidates per year was 11 per BCBA VCS. Moreover, for BCBA VCS, the median number of average candidates per year was 9 for Campus VCS, 12 for Hybrid VCS, 13 for Online VCS, and 13 for VCS categorized as Both.

Figure 2.

The Overall First-Time Pass Rate on the BCBA Exam by Average Number of First-Time Candidates Across Modes of Instruction. Note. The average number of first-time candidates was determined by dividing the total candidates by the number of reportable years of data per VCS between 2013 and 2020.

Percentage of Candidates by VCS First-Time Pass Rate

In Table 4 we outlined the number and percentage of BCBA candidates across VCS first-time pass rate ranges. We found that 53% (n = 118) of BCBA VCSs had an overall first-time pass rate of less than 70% and over 16% (n = 37) had an overall first-time pass rate of less than 50%. We found that only 6% (n = 5,160) of all VCSs BCBA first-time exam candidates finished a VCS with an overall first-time pass rate above 80%.

Table 4.

The Number and Percentage of BCBA Candidates across VCS First-Time Pass Rate Ranges between 2013 and 2020

VCS First-Time Pass Rate Range	Number of First-Time BCBA Candidates	Percent of First-Time BCBA Candidates (n = 32,929)	Cumulative Percent of First-Time BCBA Candidates	Number of BCBA VCSs	Percent of BCBA VCSs (n = 223)	Cumulative Percent of BCBA VCSs
100%	119	0.4%	0.4%	11	5%	5%
90%–99%	1,153	3.5%	3.9%	20	9%	14%
80%–89%	3,888	11.8%	15.7%	41	18%	32%
70%–79%	7,198	21.9%	37.5%	33	15%	47%
60%–69%	7,348	22.3%	59.8%	42	19%	66%
50%–59%	7,058	21.4%	81.3%	39	17%	83%
40%–49%	5,015	15.2%	96.5%	20	9%	92%
< 40%	1,150	3.5%	100.0%	17	8%	100%

First-Time Pass Rates of VCSs Housed Within ABAI Accredited Programs

Table 5 depicts the number and first-time pass rates for BCBA VCSs that were housed within graduate-level ABAI-accredited programs (n = 27) at some point between 2013 and 2020. Thirteen of these VCSs were categorized as Campus programs (48%), 7 as Both (26%), 5 as Online (19%), and 1 as Hybrid (4%). The overall aggregate first-time pass rate for VCSs housed within ABAI-accredited programs between 2013 and 2019 was 74%, which was 11 percentage points higher than the aggregate first-time pass rate for the 197 VCSs that were not housed within an ABAI-accredited program during that same period (88%).

Table 5.

Number and First-Time Pass Rate for BCBA VCS Housed within Graduate-Level ABAI Accredited Programs between 2013 and 2020

ABAI Accredited Program	Mode	Total First-Time Candidates	Total First-Time Candidates Passing	Overall First-Time Pass Rate	ABAI Accreditation Years
1	Campus	514	426	83%	2005-2020
2	Campus	190	127	67%	1994–2005, 2007–2012, 2014–2022
3	Campus	159	137	86%	2008–2024
4	Campus	150	107	71%	2014–2024
5	Campus	145	137	94%	2007–2024
6	Campus	113	94	83%	1993–2023
7	Campus	79	74	94%	2003–2025
8	Campus	73	69	95%	2010–2020
9	Campus	35	31	89%	2008–2024
10	Campus	34	28	83%	2000–2020
11	Campus	19	17	89%	2019–2024
12	Campus	7	1	14%	2003–2012, 2014–2025
13	Campus	6	5	83%	2002–2007, 2009–2019
14	Campus	N/A	N/A	N/A	2014–2019
15	Hybrid	42	32	76%	2013–2023
16	Both	312	199	64%	2001–2021
17	Both	238	200	84%	2011–2021
18	Both	218	192	88%	1999–2025
19	Both	208	184	88%	1998–2019
20	Both	153	113	74%	2011–2021
21	Both	88	63	72%	2013–2021
22	Both	44	40	91%	2014–2024
23	Online	907	489	54%	2009–2019
24	Online	51	26	51%	2000–2020
25	Online	45	28	62%	2019–2023
26	Online	9	6	67%	1995–2000, 2002–2021
27	Online	6	6	100%	2019–2024
Totals (Average)		3,845	2,830	(74%)

We conducted a chi-squaregoodness-of-fit test and found that accreditation is associated with first-time candidates passing (χ²(3) = 13.52, p < .0001) (Table 6). We set the expected first-time passing count based on the average pass rates of all candidates (i.e., 64%), and found that more candidates from accredited programs passed the exam than expected. On the other hand, considering the same expected pass rate, less candidates from nonaccredited programs passed the exam than expected.

Table 6.

Chi-Square Goodness-of-Fit Test for VCS Housed with Accreditated vs. Nonaccredited Programs

Accreditation	Total First-Time Candidates	Total First-Time Candidates Passing	Expected First-Time Candidates Passing	χ2	df	Critical Value for p < 0.001	Chi-Squarep value
Accredited	3,845	2,830	2,453	13.5199	1	10.828	5.67556E-16
Not Accredited	29,084	18,179	18,556

Years of Operation by VCS

Figure 3 depicts the overall first-time pass rates for BCBA VCSs by years of operation between 2013 and 2020. Only VCSs with 1–7 years of published first-time pass rate data were included (n = 184). VCSs with 8 years of published first-time pass rate data were excluded because they may have been in operation longer than 8 years and therefore a relevant comparison could not be made. No discernable trend in the number of VCSs across years is visible when analyzed in this way.

Figure 3.

The Overall First-Time Pass Rates for BCBA VCs by Years of Operation between 2013 and 2020. Note. Only VCSs with 1–6 years of published first-time pass rate data were included (n = 184). VCSs with 8 years of published first-time pass rate data (n = 39) were excluded because they may have been in operation longer than 8 years.

It is important to note that this figure does not show how long these VCSs may have been in operation prior to 2014. The figure also does not show the exponential growth in the number of new VCSs that have been registered over the last few years. As was mentioned earlier, the BACB published data sets list an additional 151 BCBA VCSs with no published first-time pass rate data.

Change in First-Time Pass Rate within VCS

Figure 4 depicts the change in first-time pass rates for BCBA VCSs that have at least 2 years of published data between 2013 and 2020 (n = 110). Sixty-nine percent (n = 76) of these programs demonstrated an overall increase in first-time pass rates, 28% (n = 31) demonstrated an overall decrease, and the reminder showed no change in first-time pass rates across years.

Figure 4.

Change in First-Time Pass Rates for BCBA VCSs. Note. Only VCSs with at least 2 years of published data between 2013 and 2020 are included (n = 110). Positive slope values indicate an increase in first-time pass rates over time (76 VCSs). Negative slope values indicate a decrease in first-time pass rates over time (31 VCSs). Data points falling on zero indicate no change in first-time pass rates (1 VCS). The farther from zero the steeper the change.

Discussion

Evaluating the quality of ABA training programs is important, especially as the demand for behavior analysts continues to rise and more colleges and universities incorporate ABA training in their curriculum. We set out to compare first-time BACB exam pass rates across BCBA VCSs to assess the quality of ABA training programs in hopes to encourage continued discussion, evaluation, and scholarship regarding quality indicators for ABA training. We found that since at least 2016, over 99% of first-time BCBA exam candidates have completed their coursework requirements from a VCS, which indicates the important role that VCSs have in training future BACB certificants.

Between 2013 and 2020, the overall first-time pass rate for BCBA exam candidates completing their coursework requirements at a VCS was 64%. In other words, a little less than two thirds of first-time VCS exam candidates met minimum standards, as measured by the BCBA exam, to independently provide ABA services. These data indicate that during this period a large percentage (36%) of BCBA VCS graduates did not meet minimum standards as determined by panels of experts in the field. It is notable that first-time pass rates across years for the BCBA credential were remarkably stable, falling within a range of 10 percentage points even as the number of candidates taking the exam increased significantly.

Between 2013 and 2020, few VCSs were responsible for the vast majority of BCBA exam candidates. For example, just 9 out of 223 VCSs were responsible for half of the BCBA exam candidates completing their coursework requirements at a VCS. Likewise, during this same period, just half of the VCSs accounted for 95% of BCBA exam candidates. In other words, a handful of ABA training programs have had an outsized influence on the field. These results have serious implications for the diversity of training backgrounds of credentialed providers.21

Between 2013 and 2020, the Campus and Hybrid BCBA VCSs only accounted for 23% of the first-time candidates despite accounting for 59% of the VCSs with reportable data. During this same period, both Campus (73%) and Hybrid (73%) VCSs had an aggregate first-time pass rate on the BCBA exam that was between 10 and 13 percentage points higher than the individual aggregate first-time pass rates of Both (63%) and Online (60%) VCS. This difference in instructional modality was statistically significant and confirms the recent findings shared by Guinness and Turner (2021). According to the BACB, the distinction between instructional modalities did not always exist. In 2011, the BACB shared first-time pass rate data showing that there was little to no significant difference in first-time pass rates for BCBA and BCaBA candidates graduating from a Campus VCS and an Online VCS (BACB, 2011). This no longer appears to be the case. Instead, it appears that mode of instruction may be an important variable that should be further scrutinized. Unfortunately, the VCS category known as “Both” presents a challenge in tracking the importance of this variable. At the time of writing, the ABAI VCS Directory (2019) showed 814 VCSs across the BACB fourth and fifth edition VCS Task Lists. (BACB, 2012a, 2017a). Of these VCSs, 24% are categorized as “Online” (n = 227), 28% are categorized under “On-Campus” (n = 269), 7% are categorized as “Hybrid” (n = 63), and 42% are categorized as “Both On-Campus and Online” (n = 401).22 In other words, because “Both” could reference the VCS programs online or campus offerings, the largest category of VCS may be obscuring mode of instruction as an important predictor of success on a BACB certification exam. Furthermore, the two categories of VCS, Campus and Hybrid, that appear to have produced the highest first-time pass rates between 2013 and 2020 now represent a much smaller portion of total VCSs.

Between 2013 and 2020, the vast majority of BCBA VCS appear to have been relatively small in size as indicated by the average number of first-time candidates per year. It is not surprising that the largest BCBA VCS was categorized as Online or Both. Unfortunately, correlations between the average size of a VCS and first-time pass rates are difficult to ascertain visually. For example, BCBA VCSs with 100 or more average candidates per year did not surpass an overall first-time pass rate of 72%. However, this sample size only includes nine VCSs and does not account for other variables that likely affect the number of students admitted and finishing a VCS (e.g., admission profile, retention rates, faculty-to-student ratios). Regardless, it appears a weak, but statistically significant, negative correlation between first-time pass rates and number of first-time candidates exists (Guinness & Turner, 2021; Shepley et al., 2017).

With a single exception, it appears that none of the VCSs producing the most candidates were housed within an ABAI accredited program, despite the accredited programs appearing to have an overall first-time pass rate that was 10 percentage points higher than nonaccredited programs. Further investigation is needed to determine whether this correlation is due to the mode of instruction at accredited programs or some other function of accreditation. Unfortunately, there are too few VCSs housed within ABAI accredited programs to make meaningful comparisons across modes of instruction. However, it is worth noting that the 13 Campus BCBA VCSs housed within ABAI accredited programs at some point between 2013 and 2020 had an aggregate first-time pass rate of 82%, whereas the aggregate first-time pass rates of the 86 Campus VCSs not housed within an ABAI accredited programs was 69%. Moreover, these findings support the results reported by Shepley et al. (2017) examining a smaller first-time pass rate data set from 2014.

Between 2013 and 2020, only 60% of first-time BCBA VCS candidates completed their coursework at a VCS with an overall first-time pass rate at or above 60%. In other words, a large percentage of BCBA candidates attended a VCS with an overall first-time pass rate that was lower than the national first-time pass rate average during this same time period.

For now, it appears that the number of years a VCS is an operation may not be a great predictor of first-time pass rate performance. Between 2014 and 2020, the BCBA data indicate a zero trend in first-time pass rates the longer a program was in operation. However, the range (57%–71%) and moderate variability in these data indicate caution when interpreting these findings. Likewise, the lack of data for programs in existence for more than 7 years also suggest caution when interpreting these results. Although years of operation may not be an important metric in determining performance on the BACB certification exams, it appears most BCBA VCSs improved their overall first-time pass rates internally across years of operation, as indicated in Figure 4. This result is not surprising as it seems logical that most programs would improve their training over time.

Recommendations to Improve the Quality of ABA Training Programs

In 2015, Critchfield clearly articulated why it is important that the field monitor the quality of the field’s training programs. Although a single quality indicator is limited and cannot capture the complexities inherent to all training programs, first-time pass rate data on BACB certification exams are relevant quality indicators for students considering a career practicing in the field. Unfortunately, based on this metric alone, it appears the field has a lot of room for improvement. Below are some recommendations to address some of the concerns raised by these results.

First, ABAI should start requiring VCSs to report more annual data. It would be ideal if this activity could begin immediately and be modeled after the self-study report used for programs pursuing ABAI accreditation (ABAI Accreditation Board, 2019a). Collecting this information would prepare training programs containing a VCS for the self-study required for eventual accreditation and will help the field identify variables that may be relevant to the training of applied behavior analysts. There are still numerous unknown variables that may correlate with first-time pass rates on a BACB certification examination (e.g., teacher-to-student ratio, number of adjunct instructors, practicum availability and practicum variability, admissions, student retention, degree vs. certificate programs, language/cultural barriers for exam takers, nonprofit vs. for-profit training programs). In addition, given the differences between trainees completing coursework requirements at campus and online VCSs, the category listed as “Both On-Campus and Online” should be split to indicate which students are completing a VCS on campus and which are completing the program online. Gathering these data could help the field better understand the factors that lead to more effective training outcomes.

Second, there need to be clear standards and oversight of candidates’ supervised fieldwork hours in addition to accreditation of the coursework they completed. High-quality supervised fieldwork provides candidates with opportunities to practice and apply what they have learned in didactic settings. The supervisor may tact a principle as it occurs (e.g., the supervisor may say, “notice the child is scrolling for an answer, which indicates the correct answer is not under stimulus control”). Or the supervisor may work collaboratively with the candidate to detect an ethical dilemma and troubleshoot solutions. Likewise, a supervisor may work with a candidate collaboratively to learn to identify the correct assessment(s) to conduct with a client. Such discussions and oversight from a supervisor can help the candidate become much more fluent with concepts and procedures of behavior analysis as well as ethical conduct. The BACB exam is an assessment of the candidates’ knowledge-based skills. The multiple-choice test does not provide any information regarding the candidates’ clinical skills. This is also why the supervised experience of candidates is so important. However, to date, there are few structured clinical guidelines for supervision that have been adopted or monitored by an objective party. It can be argued that practical experiences are as important as the didactic instruction candidates receive in their VCS. Yet, there is no publicly available data about the fieldwork experiences of BACB exam candidates. Gaining quality control over the supervision experience of candidates will be the next necessary step to ensure that candidates are well-prepared for the profession of behavior analysis.

Fourth, employers hosting trainees accumulating fieldwork hours at their work sites should consider implementing incentive plans to encourage enrollment in training programs with higher first-time pass rates. For example, employers offering tuition reimbursement to employees pursuing a BCBA credential could offer higher levels of reimbursement or better positions within the organization for employees attending an ABAI accredited program or VCSs with higher first-time pass rates.

Finally, the field must do better at promoting accreditation. This includes VCS faculty actively pursuing accreditation for their training programs and practica experiences. Accreditation involves conducting a self-assessment and submitting to a peer review. It is time-consuming and costly for training programs to participate in this process. Moreover, the requirement that accredited training programs offer a thesis or equivalent as well as supervised learning experiences may limit the amount and type of students programs could enroll. Some university programs may have a hard time convincing their college dean or other decision makers that accreditation is a worthwhile time and financial investment because it is not required for candidates to sit for the certification exam.

It is not yet clear why candidates graduating from ABAI accredited programs have higher first-time pass rates. However, there are important differences between the two types of training designations. One important variable is likely the difference in coursework hours required for a ABAI accredited program versus a VCS program. In particular, ABAI accredited programs require 405 hours of instruction for a master’s degree program in behavior analysis, whereas a BCBA VCS program under the fourth edition of the Task List only requires 270 hours. The required coursework content between ABAI accredited and ABAI VCS programs also differs in important ways. For example, ABAI accredited programs require training in the philosophical and experimental analysis of behavior (EAB) domains of a behavior science, as well as supervised experiential learning and thesis or equivalent components. These same areas are not required for fourth edition VCS programs; however, the fieldwork requirements likely translate somewhat to the practicum and intensive practicum fieldwork categories that were available until recently (BACB, 2021c). Moreover, the training required in the philosophical domain appears to have been incorporated into the new fifth edition VCS coursework requirements (ABAI, 2021).

Leadership in the field of behavior analysis, including accrediting and credentialing organizations as well as professional associations, should work together to identify barriers and incentivize and raise awareness of quality assurance, accountability, and the path to accreditation. For example, a survey of coordinators or directors of university training programs that are eligible for ABAI accreditation may be helpful for identifying barriers. To promote accreditation, our organizational leaders need to collaborate with each other and put all focus on gaining quality control.

ABAI has been accrediting training programs in the field since 1993 (Hopkins & Moore, 1993). It has the needed experience to develop standards, provide thorough evaluation, and oversee accreditation of coursework and behavior analytic training curriculum. Yet, as of December 2019, there were only 10 ABAI accredited doctoral programs, 23 ABAI accredited masters-level programs, and two ABAI accredited bachelor-level programs (ABAI Accreditation Board, 2019b). New contingencies need to be arranged to advance accreditation such as particular incentives only available to candidates who graduate from accredited programs or certain reliefs for the faculty and students of such programs.

Implementing these types of recommendations would not be easy, may take years, and would require careful collaboration amongst peers and across organizations. However, if the field is interested in promoting quality and rigor within its training programs, then all parties need to invest in the system specifically designed for this purpose.

Declarations

Conflict of Interest

The first author accepted the position of Vice President of Accreditation with The Council of Autism Service Providers (CASP) during manuscript revisions. In addition, he was employed part-time as the Accreditation Administrator for the Association for Behavior Analysis International (ABAI) from October 2015 to February 2018. He currently serves as a subject matter expert (SME) for the Behavior Analyst Certification Board (BACB) and is married to a member of the BACB leadership team. Lastly, during preparation of the manuscript the first author was a Verified Course Sequence (VCS) Coordinator for a graduate training program in applied behavior analysis. The second author does not currently have an affiliation with ABAI accreditation or the BACB. She has served as a BACB SME and on a few task forces in the past. At present, she a faculty member in a VCS program she founded in 2006 and directed until 2016. Lastly, she is the Chief Science Officer at Behavioral Health Center of Excellence, which is an accrediting organization.

Ethical approval

This project was submitted to the University of Louisville Human Subjects Protection Program and was deemed to not meet the “Common Rule” definition of human subjects’ research. As such the project did not require IRB review.

Informed consent

This project did not utilize any human subjects and relied on publicly available data sets.