Supervision Behaviors of Board Certified Behavior Analysts With Trainees

Abstract

The field of behavior analysis is growing rapidly, and high-quality supervision is essential to producing thoughtful and effective Board Certified Behavior Analysts. There is little empirical information about the behaviors that supervisors engage in to support trainees in developing critical skills. Therefore, our purpose in this study was to survey supervisors (n = 317) about their supervision practices to better understand how frequently they engage in recommended supervision practices and whether the frequency with which they engage in recommended practices is related to demographic characteristics. Our results suggest that there is wide variability in the extent to which individual supervisors engage in recommended practices, and that supervisors engage in practices more frequently for which there are concrete guidelines and supports in place (e.g., behavioral skills training). We discuss the implications of our results for supporting supervisors to engage in recommended practices and for future research.

Supervision is the primary means used to teach the skills required in a given profession (Storm & Todd, 1997). The supervision process necessitates a supervisor overseeing the work of a supervisee (Tyler & Tyler, 1997). The supervisory relationship entails a competent professional serving in the role of the supervisor for an untrained aspiring professional, the supervisee (LeBlanc & Luiselli, 2016). Throughout this process, the supervisee acquires and demonstrates competencies related to his or her profession (Watkins, 2012).

Behavior Analyst Certification Board (BACB) supervision is a process that occurs through several aspects of the responsibilities of a Board Certified Behavior Analyst (BCBA). A BCBA may provide supervision to individuals holding or pursuing one of the BACB certifications, including (a) trainees who are accruing supervised fieldwork hours in pursuit of BCBA certification; (b) Registered Behavior Technicians™ (RBTs), who are paraprofessionals certified in behavior analysis to provide direct behavior-analytic services to clients; and (c) Board Certified Assistant Behavior Analysts (BCaBAs), who are undergraduate-level practitioners certified in behavior analysis. In addition, BCBAs may also supervise noncertified practitioners who are implementing programming under the direction of the BCBA (e.g., paraprofessionals, job coaches, teachers, behavior technicians). In the last 5 years, the number of BCBA certificants has more than doubled, to 39,329 BCBAs as of April 2020 (BACB, 2020a). Given the rapid growth of the field, it is reasonable to suggest that the supervisory relationship between a supervising behavior analyst and a trainee is essential to ensuring that the next generation of behavior analysts is well trained and competent (Turner, Fischer, & Luiselli, 2016).

The BACB requires that trainees complete a certain amount of fieldwork experience hours under the supervision of a responsible BCBA. These fieldwork hours include opportunities for the trainee to practice the application of the competencies on the BACB Task List (BACB, 2019). The goal of this supervisory relationship is to prepare the trainee to apply minimum competencies with consumers in an applied setting (Sellers, Alai-Rosales, & MacDonald, 2016a). Some consequences of ineffective supervision include a trainee failing to demonstrate competency on the BACB certification examination, or a newly certified behavior analyst engaging in unprofessional or unethical behavior with a consumer (Sellers et al., 2016a). Therefore, it is critical that the supervision of trainees results in the mastery of minimum competencies on the BACB Task List.

Although there are some clear logistical requirements (e.g., number of fieldwork hours required, frequency of contact) from the BACB regarding trainees’ supervised fieldwork experience, there are fewer explicit parameters that describe the process and substance of the supervision process. Prior to initiating a supervisory relationship, the BACB requires that all supervisors complete an 8-hr supervision training. The supervision training must be based on the BACB’s Supervision Curriculum Outline 2.0 (hereafter referred to as the “Curriculum Outline”; BACB, 2018b). The Curriculum Outline provides learning objectives and tasks that are “important for creating an effective supervisory relationship” (BACB, 2018b, p. 2) in order to assist in the development of this supervision training. The specificity of the learning objectives in the Curriculum Outline has increased from Version 1.0 (BACB, 2012c) to Version 2.0 (BACB, 2018b), and though perhaps not intended to serve as a set of recommended supervision behaviors, these learning objectives likely function as such because they are used to create the required supervision training. Section 5.0 of the Professional and Ethical Compliance Code for Behavior Analysts (hereafter referred to as the “BACB Code”; BACB, 2014), outlines seven subsections of broad responsibilities for supervisors.

The upcoming Fifth Edition of the BACB Task List includes additional competencies related to supervision, including “Establish clear performance expectations for the supervisor and supervisee” and “Use a functional assessment approach (e.g., performance diagnostics) to identify variables affecting personnel performance” (Behavior Analyst Certification Board, 2017 p. 5). Coursework requirements based on the Fifth Edition Task List reflect these additional competencies, with 30 hr required in the area of Personnel Supervision and Management, compared to 10 hr under the Fourth Edition Task List (BACB, 2012b). Added task list items and coursework hours will benefit future BCBAs and their supervisees. However, currently, the preparation of trainees is in the hands of a generation of supervisors who did not receive explicit training in supervision (DiGennaro Reed & Henley, 2015). Therefore, it is unclear how current BCBA supervisors are delivering supervision to trainees because of the lack of supervision-related competencies in the task lists under which they were trained.

In recent years, the topic of supervision has received increasing attention in the behavior-analytic literature. In 2016, Behavior Analysis in Practice released a special edition on supervision practices. This edition contained articles on topics such as providing group supervision (Valentino, LeBlanc, & Sellers, 2016), ethical considerations (Sellers et al., 2016a), addressing barriers to supervision (Sellers, LeBlanc, & Valentino, 2016b), and recommended approaches to, and models for, supervision (Sellers, Valentino, & LeBlanc, 2016c; Turner et al., 2016). Other literature related to supervision practices in behavior analysis has described recommended processes and models for behavior-analytic supervision (e.g., Garza, McGee, Schenk, & Wiskirchen, 2018). Whereas this literature provides important discussion and recommendations, there are few empirical investigations related to the supervision of trainees (Sellers, Valentino, & LeBlanc, 2016c).

A recent study by Sellers, Valentino, Landon, and Aiello (2019) is a notable exception. Sellers et al. (2019) surveyed supervisors of trainees about their supervision practices to identify strengths and barriers to providing effective supervision. The survey tool was developed by individuals who engage in fieldwork supervision and consisted of dichotomous responses and follow-up Likert-type questions to indicate how frequently each practice was used (e.g., never, rarely, sometimes, always). A large proportion of respondents reported using contracts with supervisees, considering time constraints before adding supervisees, setting clear expectations for supervisees, using varied evaluation strategies with supervisees, and including ethics and literature in their supervision.

Sellers et al. (2019) also identified areas for improvement based on practices that were reported by a smaller proportion of respondents. Those areas included setting clear expectations for receiving feedback, evaluating the supervisory relationship, assessing and teaching skills in professionalism, and recruiting feedback about one’s own supervisory practices. Common barriers to providing high-quality supervision, according to respondents, included time, material costs, and a lack of resources and examples. The findings of Sellers et al. (2019) make an important contribution to the literature by identifying the extent to which supervisors are engaging in practices related to effective supervision.

We sought to extend the work of Sellers et al. (2019) by investigating the supervision repertoires of BCBA supervisors in connection with the BACB Code while evaluating for supervisor variables that indicate positive outcomes for the trainee. Specifically, we developed survey items based on recommended practices from the behavior-analytic literature on supervision and linked the items directly to the subsections of BACB Code Section 5.0 Behavior Analysts as Supervisors. Identifying individual supervision behaviors that support the high-quality supervision of trainees may allow BCBA supervisors to consider the critical minimum supervisor competencies required in the precertification supervisory relationship. Therefore, to explore the supervision repertoires of BCBAs with trainees, we developed the following research questions:

1. How often are supervisors reporting the use of recommended supervision behaviors with trainees?

2. Are there significant differences between the means of participant responses for subsections of BACB Code 5.0 and for a miscellaneous category?

3. Are there significant differences between the means of participant responses for subsections of BACB Code 5.0 and for a miscellaneous category relative to supervisor demographics?

4. Are there correlations between supervisors’ reported individual behaviors and certification outcomes?

Method

Participants

Prior to conducting the survey, we obtained approval from the governing institutional review board. The survey was distributed by the BACB mass e-mail service. We sent the survey link via e-mail to BACB certificants who previously elected to receive e-mail solicitations from the BACB. The number of individuals who received the survey is unknown because the BACB cannot guarantee delivery to certificants and certificants are also able to opt out of receiving e-mail solicitations from the BACB. As a result, we were able to calculate only an approximated response rate based on the number of certificants who were in the system at the time the survey was open (November 2018). The URL link was open for a 4-week period.

Development of the Survey Instrument

We used the EBSCO, Google Scholar, and ProQuest databases and the search terms “BACB certification,” “fieldwork experiences,” and “BACB supervision” to locate articles about behavior analysts as supervisors. Because there are limited publications on this topic, we included conceptual articles describing recommended practices for the supervision of trainees. We located eight articles from which we extracted 46 supervision practices recommended for professionals who are supervising trainees (see Table 1; some behaviors were included in several articles). We categorized the recommended supervision practices in the articles according to the BACB Code (BACB, 2014), placing any recommended supervision practices that were not explicitly identified with a subsection into a miscellaneous category.

Table 1.

References for Survey Item Development

Reference	Number of Recommended Practices
Behavior Analyst Certification Board (2012c)	15
Ellis & Glenn (1995)	4
LeBlanc, Heinicke, & Baker (2012)	4
Sellers, Alai-Rosales, & MacDonald (2016a)	11
Sellers, LeBlanc, & Valentino (2016b)	7
Sellers, Valentino, & LeBlanc (2016c)	5
Turner, Fischer, & Luiselli (2016)	15
Valentino, LeBlanc, & Sellers (2016)	6

Validation of the Survey Instrument

We used a two-part process consisting of (a) expert review and (b) content review to ensure a comprehensive and valid instrument. The purpose of validation was to gather information about whether the items on the survey measured the construct that we intended to measure (i.e., supervision practices).

Expert review

To collect evidence to support the validity of the survey, we obtained input from three scholars who are BCBA-Ds. To participate, we required that BCBA-Ds be actively engaged in fieldwork supervision and have at least one publication on supervision practices or be teaching graduate-level behavior-analytic courses. These BCBA-Ds examined the instrument for content validity, comprehensiveness, and any potential threats to the valid and reliable collection of information. The BCBA-Ds provided evidence of content validity through the assessment of the relevance of each item to the associated category. Their tasks included evaluating all survey questions and response options by subsection of BACB Code 5.0 in three categories: (a) relevance (i.e., the degree to which the question was applicable for the population of the study participants), (b) significance (i.e., the degree to which the question was suitable for the associated subsection of the BACB Code), and (c) clarity of the questions. The BCBA-Ds were asked to rate each item on the survey using a Likert scale for relevance of the supervision behavior to the supervision of fieldwork candidates and item clarity. Four open-ended questions prompted the BCBA-Ds to type in any survey items that needed to be clarified, removed due to duplication, added, or modified to include examples of the supervision behaviors. The expert review process resulted in changes to 5 of the 46 questions, including deleting 1 question that was redundant, providing examples of the targeted behaviors in parentheses, and rewording some items (e.g., using the term “set schedule” instead of “clear schedule”).

Content review

The content review group served as a similar cohort to those who participated in the study. They provided feedback on (a) their ability to assess the items based on their recent supervision behaviors, (b) the clarity of the survey instructions, (c) their ease of understanding the items and indicators, and (d) the length of time required to complete the survey.

Final version of the survey

The final version of the survey was divided into two sections with a total of 69 items. Section 1 consisted of 23 items focusing on demographic and employment information from respondents. These included (a) years in practice, (b) type of certification, (c) years as a BACB-approved fieldwork supervisor, (d) area of study, (e) primary job classification, (f) fieldwork supervisor preparation, (g) primary place of employment, and (h) allotted hours by employer to conduct supervision activities.

Section 2 assessed respondents’ perceptions of frequency of supervision practices with trainees. Participants responded to a series of 46 Likert-type questions relating to the frequency of different supervision behaviors. The survey median response time for completion was 11 min. All the items in Section 2 were randomly ordered by Qualtrics (2018). These supervision practices included Subsections (a) 5.01 Supervisory Competence (e.g., checking licensure requirements for a new area of practice), (b) 5.02 Supervisory Volume (e.g., having a set schedule for supervision), (c) 5.03 Supervisory Delegation (e.g., confirming a supervisee has a skill set prior to delegating a task), (d) 5.04 Designing Effective Supervision and Training (e.g., using behavior skills training), (e) 5.05 Designing Effective Supervision and Training (e.g., having a written supervision contract signed before starting supervision), (f) 5.06 Providing Feedback to Supervisees (e.g., having a written evaluation system to assess performance), (g) 5.07 Evaluating the Effects of Supervision (e.g., using an evaluation system to determine the effectiveness of supervision on the supervisee’s performance), and (h) miscellaneous recommended behaviors (e.g., sending out agendas for supervision meetings). To maintain a uniform structure, we created a Likert-type scale with anchors related to the percentages of opportunities that the respondent engages in the practice. The Likert-type scale was as follows: 1 = almost never (0%–20%), 2 = rarely (21%–40%), 3 = sometimes (41%–60%), 4 = usually (61%–80%), and 5 = almost always (81%–100%).

Data Analysis Procedures

We used descriptive statistics (i.e., means, medians, and standard deviations) to analyze the data pertaining to our first research question (how often supervisors report using recommended supervision behaviors with trainees). To analyze the data pertaining to our second research question (whether there are significant differences between the means of participant responses for the subsections of BACB Code 5.0 and the miscellaneous category), we used a single between-groups analysis of variance (ANOVA).

To answer our third research question (whether there are significant differences between the means of participant responses for the subsections of BACB Code 5.0 relative to supervisor demographics), we ran multiple one-way ANOVAs. This analysis allowed us to examine the mean of each subsection across 14 demographics: years in practice, years as a supervisor, area of study, job classification, place of employment, geographic region, supervision format, number of candidates, number of candidates in the past 12 months, allotted supervision hours, scheduled supervision hours, number of clients/consumers, who dictates schedule, and RBT supervision percentage. We selected these demographics because they are known groups that can be targeted for research and training purposes. We conducted a Tukey honestly significant difference (HSD) post hoc analysis (Tukey, 1953) for p values less than .05.

We used a Spearman correlation test (Spearman, 1904) to assess the results related to our fourth research question (whether there are correlations in certification outcomes relative to supervisors’ reported individual supervision behaviors). The Spearman correlation test allowed us to determine how individual supervision practices correlate with the percentage of candidates who passed the BACB exam for each supervisor. Spearman correlations do not make assumptions about normal distributions, and they account for discrete variables (Spearman, 1904).

Results

Respondents and Demographics

Three hundred fifty-one surveys were completed anonymously through Qualtrics. Three hundred seventeen (90.3%; 317 of 351) respondents met the criterion as a BACB supervisor for trainees. Using BACB-published certificant data, the overall survey response rate was 1.1% (351 of 32,008; BACB, 2018a). Two respondents did not disclose state information, bringing n from 317 to 315 (89.7 %; 315 of 351), which impacted only the response rate calculation. A total of 42 states were represented. The states not represented were Arkansas, Delaware, Idaho, Mississippi, New Mexico, Oklahoma, South Dakota, and Wyoming.

Table 2 contains respondent demographics. About 40% of respondents had been certified as a BCBA for less than 5 years (40.7%; n = 129 of 317). A majority of respondents had been supervising trainees for less than 5 years (60.90%; n = 193 of 317) and indicated that their area of study was behavior analysis (38.5%; n = 122 of 317). A majority (83%; n = 263 of 317) of respondents identified their primary job classification as “behavior analyst.” When asked about their place of employment, respondents selected other (34.7%; n = 110 of 317), home based (29%; n = 92 of 317), clinic based (27.1%; n = 86 of 317), and university (9.2%; n = 29 of 317).

Table 2.

Demographics of Survey Respondents

Item	n	%
Years certified
0–5	129	40.70
5.01–10	117	36.90
10.01–15	52	16.40
15.01–20	15	4.70
>20.01	4	1.30
Years as a supervisor
0–5	193	60.90
5.01–10	92	29.00
10.01–15	24	7.60
15.01–20	6	1.90
>20.01	2	0.60
Area of study
Behavior analysis	112	38.50
Education	94	29.70
Psychology	59	18.60
Other	24	7.50
Counseling	18	5.70
Job classification
Behavior analyst	263	83.00
Other	22	7.00
Professor	14	4.40
Psychologist	9	2.80
Researcher	5	1.60
Counselor	2	0.60
Teacher	2	0.60
Place of employment
Other	110	34.70
Home based	92	29.00
Clinic based	86	27.10
University	29	9.20

Supervision-specific demographics

Table 3 summarizes supervisor location, supervision format, supervision training, supervision resources, supervision protocol source, total number of trainees supervised in the past 12 months, total number of trainees supervised to date, employer-allotted weekly hours for supervision, and scheduled weekly hours for supervision. Supervision protocol source referred to the resource(s) respondents used to inform their supervision procedures (e.g., graduate school coursework, online continuing education training, company, BACB publications). This section of the survey contained multiple-choice questions with an additional open-ended response question. The number of choices varied by question, and the specific choices presented for each question are presented in Table 3. In some cases, respondents were directed to choose all applicable options.

Table 3.

Supervision-Specific Demographics

Item	n	%
Supervision location(s) (n = 389)
Agency	176	45.20
Individual private practice	129	33.20
University	52	13.40
School	20	5.10
Other nonspecified	6	1.50
Clinic	3	0.08
State agency	2	0.05
Community program	1	0.03
Supervision format (n = 317)
Individual supervision	170	53.6
Individual/group supervision	123	38.8
Intensive practicum	22	7.00
Group supervision	2	0.60
Supervision training (n = 757)
online continuing education	249	32.90
Conferences	215	28.40
Mentoring	171	22.50
Institution-based coursework
Nothing	1027	13.500.90
Other nonspecified	3	0.40
Literature	3	0.40
Company	3	0.40
Personal experience	2	0.30
BACB-required online training	2	0.30
Supervision resources (n = 748)
Training	183	24.40
Curriculum	124	16.50
Performance feedback	118	15.70
Office time	108	14.40
Monetary compensation	88	11.70
Administrative assistance	79	10.60
None	26	3.50
Other nonspecified	17	2.30
Meetings	3	0.40
Self	3	0.40
Mentorship	1	0.10
Supervision protocol source (n = 829)
Online continuing education	182	22.00
Graduate coursework	174	21.00
Mentor	159	19.20
Published supervision curriculum	149	18.00
Live CE	120	14.60
Company	12	1.40
Other nonspecified	9	1.10
Self	7	0.80
BACB publication	6	0.70
Literature	5	0.60
Cooper, Heron, & Heward textbook	2	0.20
None	2	0.20
Podcasts	1	0.10
Professional collaboration	1	0.10
Total number of trainees (n = 317)
1–3	245	77.30
4–7	61	19.20
8–11	8	2.50
12+	3	1.00
Number of trainees in past 12 months (n = 317)
1–3	215	67.90
4–7	73	23.00
8–11	15	4.70
12+	14	4.40
Allotted weekly hours for supervision (n = 317)
0	73	23.00
1–5	158	49.80
6–10	55	17.40
11–15	13	4.10
16+	18	5.70
Scheduled weekly hours for supervision (n = 317)
0	3	1.00
1–5	197	62.10
6–10	77	24.30
11–15	26	8.20
16+	14	4.40

One hundred seventy-six (45.2%) respondents indicated their current supervision location was agency based. Over half of the respondents (53.6%; n = 170 of 317) indicated that an individual supervision format was most commonly used with trainees.

Respondents were able to report all methods for initial training to prepare for trainee supervision; thus, the n was greater than the total respondents. For initial training preparation, respondents reported Internet-based continuing education (32.9%; n = 249), live conferences (28.4%; n = 215), mentoring (22.5%; n = 171), graduate coursework (13.5%; n = 102), other (1.8%; n = 13), and nothing (0.9%; n = 7). Supervision resources used to support respondents were ongoing training (24.4%; n = 183), supervision curricula (16.5%; n = 124), performance feedback (15.7%; n = 118), office time (14.4%; n = 108), monetary compensation (11.7%; n = 88), administrative assistance (10.6%; n = 79), none (3.5%; n = 26), and other (3.2%; n = 24).

When asked about supervision protocol source, respondents were able to select all items that applied; thus, the n was greater than the total respondents. The most frequently endorsed supervision protocol source was online continuing education (22%; n = 182). The remaining options used as a supervision protocol source were graduate coursework (21%; n = 174), mentorship (19.2%; n = 159), published supervision curriculum (18%; n = 149), live continuing education (14.6%; n = 120), and other (5.2%; n = 45). Over 75% of respondents indicated they have supervised a total of one to three trainees (77.3%; n = 245). Less than 5% of respondents selected eight or more total candidates (3.5%; n = 11). When respondents were asked to report the total number of candidates they supervised over the past 12 months, the majority indicated one to three (67.9%; n = 215).

Finally, respondents were asked to provide information on the total number of weekly hours allotted for supervision (e.g., preparation for contact, contact with a candidate, and post meeting tasks) by an employer versus the total number actually scheduled by the respondent. The overall distribution of allotted hours was 0 (23%; n = 73), 1–5 (49.8%; n = 158), 6–10 (17.4%; n = 55), 11–15 (4.1%; n = 13), and 16+ (21%; n = 5.7). The actual scheduled weekly hours used by the respondent for supervision-related tasks and actual contact with trainees were 0 (1%; n = 3), 1–5 (62.1%; n = 197), 6–10 (24.3%; n = 77), 11–15 (8.2%; n = 26), and 16+ (4.4%; n = 14).

Work responsibilities

Table 4 summarizes RBT monthly supervision, total consumer caseload size, and control of work schedule. One hundred eighty-eight respondents (59.3%; n = 181 of 317) indicated that they conducted RBT supervision as part of their normal work responsibilities. Of the respondents who did supervise RBTs, an equal number reported supervising RBTs for 5% and 10% of RBT direct service hours per month (17.4%; n = 55). When asked about consumer and client caseload size, respondents reported serving 12 or more consumers or clients (47.9%; n = 152 of 317) at one time in addition to supervising trainees. A majority of these respondents reported that an employer dictated control over their caseload size (54%; n = 170 of 317) versus caseload size being self-dictated (39%; n = 124 of 317).

Table 4.

Supervisor Work Responsibilities

Responsibility	n	%
RBT % monthly supervision (n = 317)
Do not supervise RBTs	129	40.70
5% of direct services	55	17.40
10% of direct services	55	17.40
15% of direct services	37	11.60
20% of direct services	41	12.90
Total consumers/clients served (n = 317)
Do not serve consumers	25	7.90
1–3	24	7.60
4–7	55	17.40
8–11	61	19.20
12+	152	47.90
Control of work schedule (n = 317)
Employer	170	54.00
Self	124	39.00
Do not provide direct services to consumers	22	7.00

Research Question 1: How Often Are Supervisors Reporting the Use of Recommended Supervision Behaviors With Trainees?

Table 5 summarizes the means for each behavior grouped by subsection of BACB Code 5.0 Behavior Analysts as Supervisors, as well as the overall means for all behaviors in each subsection. The miscellaneous category contains additional recommended behaviors clustered together. Respondents rated their perceived frequency of individual behaviors using the following Likert-type scale: 1 = almost never (0%–20%), 2 = rarely (21%–40%), 3 = sometimes (41%–60%), 4 = usually (61%–80%), and 5 = almost always (81%–100%). Figure 1 depicts the distribution of responses for each subsection of BACB Code 5.0 presented in descending order of means; subsections on the left represent better performance, whereas subsections on the right indicate a need for improvement. Subsection 5.05 Communication of Supervision Conditions had the highest mean, 4.78, and a median of 5 (almost always). The behavior in Subsection 5.04 Designing Effective Supervision and Training had a mean of 4.32 and a median of 5 (almost always). Subsection 5.06 Providing Feedback to Supervisees had a mean of 4.26 and a median of 5 (almost always). There were two exceptions to the median performance of 5 for the individual items within Subsection 5.06: documenting feedback (M = 3.87) and having a written evaluation system (M = 3.70).

Table 5.

Individual Behaviors Grouped by Subsection of the BACB Code

Behavior	M	Mdn	SD
5.01 Supervisory Competence	3.67	4	1.39
Literature for new competency area	4.04	4	1.01
Outside training area: credentialing required	4.46	5	0.90
Outside training area: training and supervision	4.53	5	0.85
Professional groups	2.91	3	1.54
5.02 Supervisory Volume	3.33	4	1.51
Supervision schedule	3.33	4	1.51
5.03 Supervisory Delegation	4.01	4	1.00
Confirm required skill set	3.94	4	1.06
Practice skill set	4.09	4	0.93
5.04 Designing Effective Supervision and Training	4.32	5	0.89
Behavioral skills training	4.32	5	0.89
5.05 Communication of Supervision Conditions	4.78	5	0.68
Performance expectations	4.58	5	0.81
Supervision termination clause	4.89	5	0.52
Written supervision contract	4.92	5	0.42
Review supervision contract	4.72	5	0.82
5.06 Providing Feedback to Supervisees	4.26	5	1.06
Document feedback	3.87	4	1.22
Immediate feedback	4.29	5	0.89
Instructions and demonstration	4.55	5	0.74
Positive and corrective feedback	4.74	5	0.52
Written evaluation system	3.70	4	1.26
5.07 Evaluating Effects of Supervision	3.48	4	1.50
Evaluate client performance	3.26	3	1.47
Evaluate supervisee performance	3.39	4	1.44
Supervision fidelity	2.77	3	1.54
Miscellaneous	3.90	4	1.37
60% fieldwork hours	3.81	4	1.12
Arrive on time	4.81	5	0.47
Attend conferences	4.25	5	1.01
Behavioral skills training case presentation	3.70	4	1.43
Continue professional relationship	4.09	4	1.05
Create group activities	2.48	2	1.55
Detect barriers to supervision	4.41	5	0.88
Discourage distractions	4.56	5	0.80
Discuss how to give feedback	4.57	5	0.80
Supervise groups	2.51	2	1.51
Include ethics	4.69	5	0.68
Maintain positive rapport	4.88	5	0.35
Make notes on meetings	3.55	4	1.39
Observe body language	4.42	5	0.88
Participate in peer review	3.33	3	1.41
Peer-evaluate	2.24	2	1.20
Return communications within 48 hr	4.83	5	0.44
Review literature	4.30	5	0.91
Schedule contacts	4.38	5	0.96
Schedule direct observations	4.65	5	0.83
Schedule standing supervision appointments	4.23	5	1.14
Seek mentorship	3.68	4	1.08
Self-assess interpersonal skills	4.51	5	0.86
Send agenda	2.66	3	1.45
Attend supervisory study groups	2.38	2	1.44
Take baseline	3.22	3	1.51

Fig. 1.

Box plot of the distribution of responses across BACB Code 5.0 subsections. The box plot for each category depicts the minimum, first quartile, median (black line), mean (black square), third quartile, maximum, and any outliers (circles) of responses for that category

The mean reported frequency of behaviors in Subsection 5.03 Supervisory Delegation was 4.01 with a median of 4 (usually). Behaviors in Subsection 5.01 Supervisory Competence fell toward the back of the analysis. The means for individual behaviors were 2.9 (rarely) to 4.5 (almost always) occurring. Respondents reported usually seeking training and supervision (M = 4.53) and any additional credentialing required (M = 4.46). The median responses for both of these items were 5 (almost always). Respondents also reported usually reviewing literature for a new competency area (M = 4.04, Mdn = 4) and rarely to sometimes participating in professional groups (M = 2.91, Mdn = 3).

The overall mean for behaviors comprising Subsection 5.07 Evaluating the Effects of Supervision was 3.48 (sometimes). Respondents reported sometimes evaluating client performance (M = 3.26, Mdn = 3), sometimes to usually evaluating supervisee performance (M = 3.39, Mdn = 4), and rarely to sometimes evaluating supervision fidelity (M = 2.77, Mdn = 3). Finally, related to Subsection 5.02 Supervisory Volume, respondents reported having a set schedule for supervision sometimes to usually (M = 3.33, Mdn = 4).

Research Question 2: Are There Significant Differences Between the Means of Participant Responses for Subsections of BACB Code 5.0 and for a Miscellaneous Category?

A one-way between-groups ANOVA was conducted to compare the effect of the subsection of the BACB Code on survey responses. There was a significant effect of the independent variable of supervision subsection on the dependent variable of supervision survey responses at the p < .05 level, F(7, 14558) = 137.267, p < .001. Post hoc comparisons using the Tukey HSD test (Tukey, 1953) indicated that a majority of the categories’ mean scores were significantly different from each other at the p < .05 level.

At the top of the performance range, the mean score for Subsection 5.05 (M = 4.78) was significantly higher than all other subsections. Next, Subsections 5.04 (M = 4.32) and 5.06 (M = 4.26) were statistically similar. Subsection 5.03 (M = 4.01) and the miscellaneous category (M = 3.90) were also similar to each other at the p < .05 level (p = .367). Following this pair, 5.01 (M = 3.67) was statistically different from all other subsections at the p < .05 level. At the bottom of the performance range, 5.07 (M = 3.48) and 5.02 (M = 3.33) were statistically similar to each other at the p < .05 level (p = .516).

Research Question 3: Are There Significant Differences Between the Means of Participant Responses for Subsections of BACB Code 5.0 and for a Miscellaneous Category Relative to Supervisor Demographics?

We used multiple one-way ANOVAs to evaluate the 14 demographic variables against each subsection of BACB Code 5.0 and the miscellaneous category for a total of 112 tests (full results are available from the first author). There was a significant difference between the means of 35 of the 112 possible combinations (31.2%) at the p < .05 level. The demographic variables that were most often significantly related to participant responses were employment and the supervision-specific variables of (a) scheduled time by a supervisor, (b) number of trainees, (c) allotted time by an employer, and (d) place of employment.

Specifically, these variables were significantly related to the frequency of reporting recommended supervision behaviors primarily in 5.01 Supervisory Competence, 5.02 Supervisory Volume, and the miscellaneous category. In terms of place of employment, respondents working in a university setting self-reported higher than those working in a home-based setting in those three subsections. For Subsection 5.03 Supervisory Delegation, all four variables were statistically different during the ANOVA, but the post hoc analysis did not reveal any of the subgroups with statistically different results. In addition, respondents with less than 2 years of supervisor experience self-reported lower scores in Subsection 5.01 compared to respondents with at least 5 years of experience.

Research Question 4: Are There Correlations Between Supervisors’ Reported Individual Behaviors and Certification Outcomes?

A Spearman correlation test (Spearman, 1904) was used to determine how individual supervision practices correlate with the reported percentage of candidates who passed the BACB exam. Rho (ρ; Spearman, 1904) is a correlation coefficient that represents how closely the data align with the line of best fit. A ρ of 1.00 represents a perfect positive correlation. A ρ of −1.00 represents a perfect negative correlation. A ρ of 0.00 represents no correlation. Table 6 shows the seven individual behaviors (15.2%) that were significantly correlated to higher pass rates. Though these correlations were statistically significant (i.e., unlikely to be due to chance), they were all relatively weak, indicating that the relationship between the behaviors and pass rates may not be practically significant. Subsection 5.01 Supervisory Competence had two individual behaviors positively correlate to a higher pass rate (ρ = .353, .350). Subsection 5.03 had one individual behavior positively correlate to a higher pass rate (ρ = .177). Subsection 5.07 had a single negative correlation (ρ = −.160; e.g., the more the supervisor engages in the behavior, the lower the reported pass rate of trainees). The miscellaneous category had three individual behaviors positively correlate to higher pass rates (ρ = .186, .196, .159)

Table 6.

Significant Correlations Between Individual Items and Reported Supervisee Pass Rate

Item	ρ	p value
5.01 Outside training area: credentialing requirements	.353	.027
5.01 Outside training area: training and supervision	.350	.029
5.03 Practice skill set	.177	.015
5.07 Evaluate client performance	−.160	.028
Misc. Include ethics	.186	.010
Misc. Review literature	.196	.007
Misc. Attend conferences	.159	.028

Discussion

The current and expected growth rate of the number of BCBAs (BACB, 2020a) requires a careful examination of how future generations of aspiring behavior analysts are trained to work with consumers. Providing effective supervision experiences is an ethical and professional obligation of the BCBAs supervising these aspiring behavior analysts. The aims of this study were to examine the demographics of supervisors, to evaluate reported frequencies of various supervision practices, to assess whether reported frequencies differed based on the subsections of the BACB Code or supervisor demographics, and to evaluate whether there was a relationship between the reported frequency of supervision practices and trainee certification outcomes.

Our study is the first to examine supervisor demographics and employment variables in the field of applied behavior analysis. Only 0.90% of respondents in our study reported no prior supervision training, with most respondents reporting that they received training through online continuing education (32.9%) and relatively few reporting that they received training through their graduate coursework (13.5%). In previous research (DiGennaro Reed & Henley, 2015), most BCBAs (66.3%) indicated that they did not have access to training on effective supervision practices for direct care staff through their current place of employment. A direct comparison of our results with the results of DiGennaro Reed and Henley (2015) regarding access to supervision training is complicated by different time frames (i.e., access to supervision training in the current place of employment vs. access to supervision training at any point from any source) and different study populations (i.e., direct care staff vs. trainees). Nonetheless, it is encouraging that nearly all respondents in our study reported accessing some form of supervision training at some point during their coursework, fieldwork hours, or professional practice. A number of changes over the last several years may have increased the availability of resources and continuing education related to effective supervision, including additional supervision items on the Fourth Edition Task List (BACB, 2012b), the required 8 hr of supervisor training (BACB, 2012a), the Curriculum Outline (2012c), the requirement for a minimum number of supervision-specific continuing education units each recertification cycle (BACB, n.d.), and additional scholarly articles on the topic.

Most respondents reported being allotted, and using, 1–5 hr per week for supervision. It is encouraging to see that the reported time allotted by employers and used by respondents is relatively consistent (i.e., 1–5 hr). However, it is alarming that 23% of respondents reported no allotted time for supervision because these respondents could be beyond their work capacity. Further, most survey respondents reported working with 12 or more clients (i.e., 47.9%) in addition to supervising one to three trainees (i.e., 77.3%). Therefore, it is unclear whether respondents are maintaining a reasonable work volume to comply with Subsection 5.02 Supervisory Volume because there is no published guidance on how many trainees one supervisor should support at one time while also working with clients/consumers. Given the myriad, individualized contextual variables that impact how many trainees a supervisor can effectively supervise at any given time (e.g., number of clients, intensity of client services/needs, other job responsibilities), it may not be possible or desirable to identify a specific number of trainees that any one supervisor can support. However, future research could investigate the time associated with supervising a trainee by asking BCBAs to document their time spent completing supervision activities that occur before, during, and after supervision (Turner et al., 2016). An important implication of this finding is that employers should also be cognizant of the time needed to effectively supervise trainees and establish internal policies that support their employees in engaging in ethical and effective supervision.

In terms of the frequency with which respondents reported engaging in recommended supervision practices, our results indicate that there is wide variability. Respondents selected a range from 2 (rarely) to 5 (almost always) relative to individual behaviors. Respondents’ overall average was just below 4 (usually) as a composite score across all 46 behaviors; however, the average composite score for individual respondents ranged from just above 2 (rarely) to just below 5 (almost always). This variability in individual supervision practices likely produces an inconsistent quality of supervision for trainees.

Our results also indicate that respondents almost always engage in several recommended behaviors related to supervision, several of which are related to the use of a contract. Frequently reported behaviors include having a written contract with a supervision termination clause, reviewing the written contract, arriving on time, and responding to contacts within 48 hr. These findings are similar to those of Sellers et al. (2019), who reported that a large proportion of respondents used a contract to establish the expectations of their supervisory relationship. There are several potential explanations for these behaviors being reported at relatively high frequencies. First, several of the behaviors are related to the supervision contract, and these behaviors typically occur once per supervisee. Second, the same contract can be used for multiple supervisees. Third, the BACB provides specific information about what needs to be included in a supervision contract, along with a template (BACB, 2012c, 2018b). Fourth, compared to other behaviors on the survey, these behaviors are relatively low effort and have clear criteria for evaluation. For example, a respondent could easily evaluate the frequency with which they respond to contacts within 48 hr, whereas evaluating the frequency with which they participate in professional groups may be more complex.

We also identified recommended practices that respondents reported using less frequently. These low-performing items primarily include behaviors related to group supervision practices (e.g., engaging in group supervision, designing group activities, setting up supervisory study groups). Researchers have suggested that having group supervision supports the trainee in developing a well-rounded professional repertoire and the interpersonal skills necessary for effective clinical work (Valentino et al., 2016). It is possible that these behaviors were reported at a lower frequency than others because supervisors may not have access to enough supervisees to conduct group supervision. This potential explanation is supported by our finding related to employment demographics; supervisors with more trainees also self-reported significantly higher averages for a subset of behaviors related to group supervision (e.g., creating group activities and conducting group supervision) than supervisors with fewer trainees. Future research could evaluate supervisors’ perceived value of group supervision activities, as well as perceived barriers that prevent the supervisor from engaging in the behavior. Given the importance of group supervision, employers and BCBA supervisors may consider how to facilitate access to group supervision opportunities, even if only intermittently. For example, supervisors in a company could be paired to create shared groups, or supervisors across two or more service agencies could collaborate locally or via teleconference to create group supervision opportunities.

When looking at the practices organized by subsection of BACB Code 5.0, we found that there were statistically significant differences between mean ratings by subsection. Behaviors comprising Subsections 5.04 Designing Effective Supervision and Training, 5.05 Communication of Supervision Conditions, and 5.06 Providing Feedback to Supervisees were all self-reported at significantly higher frequencies relative to other subsections, such as 5.01 Supervisory Competence, 5.02 Supervisory Volume, 5.03 Supervisory Delegation, and 5.07 Evaluating Effects of Supervision. For example, responses related to Subsection 5.07 Evaluating the Effects of Supervision indicated that respondents are rarely to sometimes engaging in behaviors that support compliance with this part of the BACB Code. Failing to systematically evaluate the effects of supervision may lead to the delivery of ineffective supervision and the production of unqualified practitioners (Sellers, Alai-Rosales, & MacDonald, 2016a; Sellers, Valentino, & LeBlanc, 2016c). Based on the poorer performance in these subsections of the BACB Code, university training programs, employers, and authorized continuing education (ACE) providers may consider enhancing their training and resources relevant to these subsections of the BACB Code to ensure supervisors are supported to engage in effective and compliant supervisory practices.

The Curriculum Outline (2012c), upon which the required 8-hr supervision training must be based, detailed specific learning objectives for several of the behaviors in Subsections 5.04, 5.05, and 5.06. These specific learning objectives that related to what constitutes compliance with these subsections of the BACB Code may have helped ACE providers generate training content that assisted supervisors in performing better relative to other subsections (i.e., 5.01, 5.02, 5.03, 5.07). The updated Curriculum Outline (2.0; BACB, 2018b) includes more learning objectives and considerations related to effective supervisory relationships across the subsections of the BACB Code. Therefore, training based on the updated Curriculum Outline may assist supervisors in performing these behaviors more frequently. Future research could explore the impact of the updated Curriculum Outline and whether it improves performance relative to other subsections of the BACB Code.

Another aim of our study was to evaluate whether any demographic or employment variables impact the self-reported frequency of supervision behaviors. Four employment variables were significantly related to the frequency of reporting recommended supervision behaviors: (a) place of employment, (b) allotted hours, (c) scheduled hours, and (d) number of trainees. It is not surprising that respondents with more allotted time for supervision, and who allocate more scheduled time to supervision, engage in higher frequencies of recommended supervision behaviors. However, additional research is needed to determine the contingencies in a university setting that support a higher reported frequency of these supervision behaviors in these three categories relative to respondents working in home-based settings. For example, respondents in home-based settings may have diverse client needs that impact scope of competence or unpredictable client logistics that impact the time available for consistent supervision meetings.

Notably, we found that respondents who had been certified for less than 2 years consistently self-reported lower frequencies of engaging in recommended behaviors in 5.01 Supervisory Competence than respondents who had been certified for at least 5 years. This is concerning, given that this subsection relates to recognizing and practicing within one’s scope of competence and scope of practice. As scope of competence has received more attention in the behavior-analytic literature recently (e.g., Brodhead, Quigley, & Wilczynski, 2018), newer practitioners may become aware of the importance of practicing and supervising within their competence. However, our finding supports the BACB’s upcoming change to require supervisors to hold their certification for 1 year before supervising trainees (BACB, 2019).

In exploring potential relationships between reported supervision behaviors and reported pass rates of supervisees, we found seven statistically significant, but relatively weak, correlations. Two of the behaviors in 5.01 Supervisory Competence, checking credentialing requirements for new areas of practice and seeking additional training and supervision for new areas of practice, were positively correlated to a higher reported BACB exam pass rate. Interestingly, evaluating client performance had a weak negative correlation with higher pass rate. This finding means that the higher self-reported frequency of evaluating client performance by the supervisor resulted in a lower self-reported BACB exam pass rate by their supervisees. Given the respondent demographics, a majority of the supervisors may be primarily providing services to clients/consumers. Therefore, this result could be interpreted as supervisors in fact spending more time handling client/consumer issues as part of “evaluating client performance” and less time on other fundamental aspects of supervision (e.g., evaluating supervisees, delivering performance feedback). As these results are preliminary, and correlations were not strong, future research may continue to investigate which supervision behaviors are related to higher supervisee pass rates.

Limitations

This study has several limitations worth noting. Our results are based on supervisors’ self-report, which is subjective, and there is a potential for response bias. However, we did operationally define the anchors of our rating scale with percentage of opportunity (e.g., never = 0%–20% of opportunities) in an attempt to obtain comparable reporting across respondents. The dissemination of an online survey presents challenges to ensuring the invitation to participate is received. We are unsure of how many potential respondents did not receive the mass e-mail due to spam filters.

We also encountered challenges in our analysis related to having an adequate sample size and a representative sample. Although we were able to analyze the geographic distribution, ideally, the total sample size would have been closer to 10% of BCBAs and BCBA-Ds (e.g., 3,000). The smaller sample size limits the generalization of the results to all BCBAs and BCBA-Ds who are supervising trainees. Relatedly, it is possible that a large proportion of our respondents may have been from the same large organization or agency, so it is unclear how representative our data are of the field at large. Finally, the statistical analysis across the eight categories could be improved if the miscellaneous behaviors were all assigned to a distinct subsection of BACB Code 5.0 Behavior Analysts as Supervisors. Future research should address these limitations by assigning miscellaneous behaviors to subsections of Section 5.0 Behavior Analysts as Supervisors and attempting to increase the supervisor response rate. Assigning the miscellaneous behaviors would require subject matter experts to confer and agree to which subsection each individual behavior should be assigned. Alternatively, researchers could anchor practices to the categories in the Curriculum Outline 2.0 to identify the frequency of recommended practices from that document.

Conclusion

The quality of precertification supervision is of extreme importance to the integrity of behavior analysis. In this study, a majority of respondents reported they did not receive graduate coursework or comprehensive competency-based assessment on the critical skills required to deliver precertification supervision. Most individuals in this study reported relying on online continuing education events or conferences in order to get information on supervision for trainees. The primary reason for the lack of specific supervision skills and resources may be the relatively new development of this certification (i.e., 1998). However, due to the exponential growth rate and the predictions for ongoing growth (Behavior Analyst Certification Board, 2020b), it is necessary for the field to consider additional safeguards for the responsible supervision of future generations. Additional research, scholarly resources, and policy development around the supervision of future practitioners will allow the field to grow in a responsible and effective manner.

Funding

This research was conducted without external funding.

Compliance with Ethical Standards

Availability of data and material

Raw data may be obtained by contacting the corresponding author.

Conflict of interest

The authors declare no conflicts of interest.