An Evaluation of the Impact of Supervision Intensity, Supervisor Qualifications, and Caseload on Outcomes in the Treatment of Autism Spectrum Disorder

Abstract

Ample research has shown the benefits of intensive applied behavior analysis (ABA) treatment for autism spectrum disorder (ASD); research that investigates the role of treatment supervision, however, is limited. The present study examined the relationship between mastery of learning objectives and supervision hours, supervisor credentials, years of experience, and caseload in a large sample of children with ASD (N = 638). These data were retrieved from a large archival database of children with ASD receiving community-based ABA services. When analyzed together via a multiple linear regression, supervision hours and treatment hours accounted for only slightly more of the observed variance (r ² = 0.34) than treatment hours alone (r ² = 0.32), indicating that increased supervision hours do not dramatically increase the number of mastered learning objectives. In additional regression analyses, supervisor credentials were found to have a significant impact on the number of mastered learning objectives, wherein those receiving supervision from a Board Certified Behavior Analyst (BCBA) mastered significantly more learning objectives. Likewise, the years of experience as a clinical supervisor showed a small but significant impact on the mastery of learning objectives. A supervisor’s caseload, however, was not a significant predictor of the number of learning objectives mastered. These findings provide guidance for best practice recommendations.

Applied behavior analysis (ABA) is a well-established framework for the treatment of autism spectrum disorder (ASD; Eldevik et al., 2009; Reichow, 2012; Reichow et al., 2012). ABA-based treatment is conducted at a high intensity, typically between 30 and 40 h/week, for multiple years, often beginning in early childhood (Eldevik et al., 2009; Reichow et al., 2012). While a strong consensus exists that ABA is an effective treatment for ASD, evidence also indicates a good deal of variance in individual response to treatment (Eldevik et al., 2010; Howlin et al., 2009).

Several factors have been suggested to have an effect on ABA treatment outcomes. Some factors are specific to the individual at the start of treatment; for instance, younger age (Ben-Itzchak & Zachor, 2011; Eldevik et al., 2012; Flanagan et al., 2012; Granpeesheh et al., 2009; Harris & Handleman, 2000; Makrygianni & Reed, 2010; Perry et al., 2011; Virués-Ortega et al., 2013), higher IQ (Ben-Itzchak & Zachor, 2007; Eikeseth et al., 2002, 2007; Eldevik et al., 2006; Eldevik et al., 2010; Eldevik et al., 2012; Harris & Handleman, 2000; Hayward et al., 2009; Magiati et al., 2007; Magiati et al., 2011; Perry et al., 2011; Remington et al., 2007; Smith et al., 2010), lower severity of ASD symptoms (Ben-Itzchak & Zachor, 2011; Eldevik et al., 2012; Perry et al., 2011; Remington et al., 2007; Smith et al., 2000), greater adaptive skills (Eldevik et al., 2010; Flanagan et al., 2012; Magiati et al., 2011; Makrygianni & Reed, 2010; Perry et al., 2011; Remington et al., 2007; Sallows & Graupner, 2005), stronger language skills (Ben-Itzchak & Zachor, 2011; Eldevik et al., 2006; Magiati et al., 2007; Magiati et al., 2011; Sallows & Graupner, 2005), and greater social skills (Ben-Itzchak & Zachor, 2007; Sallows & Graupner, 2005) have been associated with superior outcomes. Other factors are treatment specific; for example, greater treatment intensity (Eldevik et al., 2010; Granpeesheh et al., 2009; Makrygianni & Reed, 2010; Remington et al., 2007), longer treatment duration (Luiselli et al., 2000; Makrygianni & Reed, 2010), and greater overall intervention time (Virués-Ortega, 2010; Virués-Ortega et al., 2013) have been shown to have a positive impact.

Although research is limited, there is evidence to suggest that variables related to the supervision of ABA-based treatment also significantly contribute to treatment outcome. For example, a meta-analysis conducted by Reichow and Wolery (2009) examined the relationship between supervisor training models and treatment outcomes. Their findings suggested that studies that implemented supervisor-training protocols based on the University of California, Los Angeles (UCLA) model produced greater gains in IQ than studies that employed other training procedures.

For the most part, evaluation of the impact of supervision on treatment outcomes has been limited to treatment programs that are parent managed, meaning parents are responsible for managing the implementation of their child’s treatment program while receiving some degree of clinical oversight from a professional. Several studies have been conducted to evaluate parent-managed treatment, combined with varying levels of professional supervision, as a cost-effective alternative to clinic-based treatment programs. For example, Bibby et al. (2002) found parent-managed ABA programs (described in detail by Mudford et al., 2001) to produce relatively poor treatment outcomes as compared to the clinic-based treatment outcomes reported by Lovaas (1987). A number of factors were suggested by the authors to have contributed to the discrepant outcomes, including older age and lower IQ at start of treatment, fewer treatment hours, infrequent supervision conducted about once every 3 months, and less competent supervisors (approximately 80 % of whom were not trained to Lovaas treatment model standards). Therefore, though the outcomes were clearly poorer than those documented by Lovaas (1987), the role of supervision in the work by Bibby et al. (2002) is difficult to evaluate, as it is just one of numerous factors that may have impacted treatment outcomes.

Other studies evaluating the effectiveness of parent-managed ABA treatment programs when combined with more frequent supervision than reported by Bibby et al. (2002) have revealed better outcomes. Both Sallows and Graupner (2005) and Hayward et al. (2009) compared parent-managed treatment to clinic-based treatment and found participants in both groups to make meaningful gains with no significant differences detected between groups. Sallows and Graupner (2005) observed similar treatment outcomes between groups despite the fact that less frequent supervision was given to the parent-managed treatment group. However, in a further examination of the parent-managed treatment group described by Hayward et al. (2009), Eikeseth et al. (2009) identified a strong relationship between greater supervision intensity and improved treatment outcomes. Supervision intensity, which ranged from 2.9 to 7.8 h/month, was significantly correlated with improvements in IQ at follow-up. On average, IQ increased 0.21 points for each hour of supervision with no detectible point of diminishing returns. Given these studies, the relationship between supervision intensity and treatment outcomes is unclear. While there is some evidence to suggest that supervision intensity correlates with treatment outcomes in parent-managed treatment programs (e.g., Eikeseth et al., 2009), such research has not yet been conducted in clinic-based treatment settings.

In spite of limited research on the role of supervision in ABA programs, efforts have been made to promote uniformity in treatment provision. The Behavior Analyst Certification Board (BACB), established in 1998, is among the leading organizations helping to set standards in the field. The BACB summarized best practices for supervision of ABA-based autism treatment in its practice guidelines for funding agencies (BACB, 2014). While the individual demands of each case must be taken into account, the BACB specifies supervision conducted at a ratio of 2 h a week per every 10 h of treatment as the recommended standard, with a minimum of 2 h of supervision provided a week. This reflects an increase in the recommended supervision hours relative to the previously published BACB guidelines, which gave a range of 1–2 supervision hours for every 10 h of treatment (BACB, 2012). The BACB also describes average caseload sizes for supervisors overseeing comprehensive ABA treatment programs to range between 6 and 16 cases, depending on the treatment intensity and demands of each case, competency and accessibility of the supervisor, and the supervisor’s level of support. Average caseloads for supervisors overseeing focused treatment programs are specified by the BACB as ranging between 10 and 24 cases. These recommendations have been suggested as best practices in the field; nonetheless, existing research does not establish whether these recommendations produce superior treatment outcomes.

The BACB has recommended standards for supervisor qualifications, as well. The BACB offers a Board Certified Behavior Analyst (BCBA) certification for clinicians in the field. To become a BCBA, applicants must, as of January 1, 2016, hold a master’s degree in behavior analysis, education, or psychology (previously accepted master’s degrees, which may better represent the current BCBA population, include behavior analysis or related field or other natural science, education, human services, engineering, or medicine); satisfy specific coursework requirements in behavior analysis; have a specific number of work experience hours directly supervised by a BCBA; and pass an exam. The BACB also offers a doctoral BCBA certification (BCBA-D) for those who hold a qualifying doctoral degree and satisfy all other BCBA certification requirements. Additionally, a bachelor’s level certification, Board Certified Assistant Behavior Analyst (BCaBA), is offered; however, the BACB stipulates that any supervision provided by a BCaBA must be overseen by a BCBA or BCBA-D. Despite the rigorous requirements to obtain a BCBA, it should be noted that ABA is a broad field not limited only to the treatment of ASD, and obtaining a BCBA does not necessarily indicate competency in the treatment of ASD (Eikeseth, 2010; Love et al., 2009). Therefore, training and supervised work experiences in ABA treatment specifically for ASD are typically recommended in addition to certification (Eikeseth, 2010) and fall within the BACB’s requirement that certificants practice within the scope of their experience.

To identify the percentage of those with graduate degrees who also hold a BACB certification, Love et al. (2009) surveyed a large group of ASD treatment providers. From the survey of 211 supervisors, 72 % of respondents reported having a graduate degree, and 42 % reported having a BCBA or BCBA-D. These findings may reflect an effort to supplement the insufficient number of supervisors who possess BACB certifications to meet the high demand for ABA services. Additionally, it should be noted that ABA treatment services for ASD have been provided for over 30 years, predating certification efforts. As such, many well-trained and experienced clinicians are not certified, including individuals who pioneered the application of ABA to the treatment of ASD. Additionally, BCBA certification is only one of many credentials recognized by current and emerging state insurance mandates that often specify the education, training, certification, and/or licensure required to supervise ABA programs. While requirements vary from state to state, other recognized professionals include licensed psychologists, marriage and family therapists, speech and language pathologists, occupational therapists, and audiologists practicing within the scope of their licensure and competency. Although numerous state laws define who may supervise ABA programs for individuals with ASD, research evaluating whether such qualifications actually lead to superior treatment outcomes has not yet been conducted.

Given the lack of empirical evidence to guide the development of best practice guidelines for supervision of ABA-based ASD treatment, the purpose of the present study was to examine the relationship between factors related to supervision and ABA treatment outcomes. Specifically, the present study tested the hypothesis that supervision hours, supervisor credentials, years of experience, and caseload would be significant predictors of the number of mastered learning objectives within a large dataset collected from a community-based clinical setting.

Methods

Participants

Clinical records were gathered from a pool of 836 children between the ages of 18 months and 12 years who were receiving ABA-based services from a community-based autism treatment provider during a 12-month period (January 1, 2014 through December 31, 2014). Records were subject to the following inclusion criteria: a diagnosis of ASD (American Psychiatric Association, 2013), autistic disorder (American Psychiatric Association, 2000), pervasive developmental disorder-not otherwise specified (PDD-NOS; American Psychiatric Association, 2000), or Asperger’s disorder (American Psychiatric Association, 2000) by an independent licensed clinician (e.g., psychologist and pediatrician); at least 20 h of ABA-based treatment per month; and at least one full month of continuous services. These criteria produced a sample size of 638 clinical records. The age, diagnosis, and gender profiles of the individuals whose clinical records were used in the study were as follows: 528 males (age range 2.08–11.92 years, mean age 7.42 years, 317 autistic disorder, 166 ASD, 41 PDD-NOS, 4 Asperger’s disorder) and 110 females (age range 3.17–11.83 years, mean age 7.53 years, 73 autistic disorder, 30 ASD, 6 PDD-NOS, 1 Asperger’s disorder). The mean age of the individuals whose records made up this sample was 7.44 years (SD = 2.30). The average number of treatment hours received per month was 71.01 (SD = 35.26), ranging from 20.02 to 197.30 h/month. An average of 10.98 (SD = 6.50) supervision hours were received per month, ranging from 1.40 to 67.40. Furthermore, an average ratio of 1.77 (SD = 1.14) supervision hours were provided for every 10 h of treatment, ranging from 0.25 to 9.73. The average number of mastered learning objectives per month was 31.42 (SD = 34.47), ranging from 1 to 245.75 per month. Individuals whose records were included in this sample resided and received services in the states of Arizona, California, Colorado, Illinois, Louisiana, New York, Texas, and Virginia.

Data Collection

Treatment data were collected retrospectively from a large archival database. Throughout treatment delivery, the Skills™ system was used to identify developmental deficits, design individualized treatment programs, and track ongoing progress. The Skills™ Assessment is an instrument that comprehensively evaluates skills across all areas of child development (Dixon et al., 2011). A study by Persicke et al. (2014) evaluated the validity of the Skills™ Assessment by contrasting parent response to the Skills™ items with direct observation. Pearson product-moment correlation coefficients ranged from moderate (r = 0.65) to high (r = 0.95). Treatment data were combined with the behavioral health agency’s operational data, including treatment hours, supervision hours, supervisor credentials, years of experience, and caseload.

Treatment

Treatment programs were individualized according to each participant’s specific strengths and deficits. Treatment programs addressed all developmental areas in which the participant displayed deficits, including language, academics, social skills, play skills, motor skills, adaptive skills, executive functions, and cognition. Services were provided in the home, school, community, clinic, or a combination of settings, depending on funding agency requirements and other variables. All treatment programs in this study followed the CARD model of treatment delivery (Granpeesheh et al., 2014) and therefore shared the following commonalities: (a) trained behavioral therapists delivered one-to-one treatment; (b) both discrete trial training and natural environment training strategies were implemented; (c) a verbal behavior approach was used for language intervention; (d) both errorless and least-to-most prompting strategies were implemented; (e) empirically validated behavioral principles and procedures were used as needed, including reinforcement, extinction, stimulus control, generalization training, chaining, and shaping; (f) a function-based approach was implemented for the assessment and treatment of challenging behaviors; (g) parents received training regularly and were included in all treatment decisions; and (h) direct supervision was provided on a regular basis (e.g., biweekly). The number of treatment hours per participant was collected from billing records and included all direct treatment services provided to the participant. Activities that were not client-specific, such as attending training, or were not direct treatment services, such as traveling to participant’s home, were excluded.

Mastery of learning objectives was used as the dependent variable for all analyses within this study. The definition of mastery of a learning objective was set on an individual basis by the treatment supervisor but was required to be within the bounds of the following criteria: >70 % accuracy of responding to the learning objective for a minimum of two treatment sessions across two different days. Typically, a more stringent mastery criterion of 80 % accuracy is required, but supervisors have the discretion to deviate if they feel it is clinically appropriate to do so.

Supervision

All supervisors in the present study received a minimum of 6 months of training in ABA-based treatment for ASD and earned a certification in supervision from the Institute for Behavioral Training. A multifaceted training approach was used, which included a combination of eLearning (www.ibehavioraltraining.com), classroom-style training, web classes, and mentorship. Supervisors received mentorship on a weekly basis, which involved direct observation, feedback, and follow-up training to improve clinical skills. Exams were administered at various stages of the training program, and trainees were required to demonstrate fluency in training material before advancing to the next stage. At the end of the training program, supervisors were required to demonstrate clinical competency by passing a written practicum and oral exam.

Supervisors in the present study were responsible for overseeing participants’ treatment programs. The number of supervision hours per participant was collected from billing records. Supervision hours were required to be client-specific and were composed of both direct and indirect services, including: (a) making clinical recommendations on treatment intensity and duration, (b) conducting assessments, (c) developing individualized treatment plans, (d) holding regularly scheduled clinic meetings with families and therapists, (e) observing treatment sessions, (f) reviewing data and adjusting treatment plans accordingly, (g) reporting on treatment progress, (h) consulting with teachers and other service providers, (i) conducting therapist and parent training to implement client-specific protocols, and (j) preserving treatment integrity. Activities excluded from the analysis were as follows: (a) conducting client intakes, (b) conducting therapist performance evaluations, (c) providing staff trainings that were not client-specific, (d) developing discharge plans, and (e) travel to client homes. In general, supervision was provided at a minimum ratio of 1 h of supervision per every 10 h of treatment in accordance with the best practices set forth at the time these services were delivered (BACB, 2012). For a more detailed description of the supervisor training and responsibilities involved in the present study, see Granpeesheh et al. (2014).

Supervisor caseload was determined by counting the number of clients assigned to each supervisor during a 1-month interval (October of 2014) within the larger period of time that records were reviewed (January 1, 2014 through December 31, 2014). Given that each supervisor does not work the same number of hours each week but can range from part time to full time and that their work hours directly impact the number of treatment programs that each can supervise, caseloads were divided by the number of hours that the supervisor worked per week during the same 1-month interval. For example, a part-time supervisor with a caseload of 15, who works 25 h/week, would have a weighted caseload of 0.6. Similarly, a full-time supervisor with a caseload of 25, who works 43 h/week, would have a weighted caseload of 0.63. This adjustment made the caseloads comparable despite the number of hours the supervisor worked per week.

As a part of maintaining their personnel record, supervisors had previously reported the date that they first began to supervise ABA-based treatment for ASD (including times spent as a supervisor at other treatment agencies). A supervisor’s years of experience was calculated as the difference between their supervision start date and October 1, 2014. Supervisor credentials were also obtained through a review of personnel files.

There were 130 supervisors represented in the data set. A total of 37 supervisors were excluded because they held an alternative credential (e.g., licensed psychologists and licensed clinical social worker) or provided incomplete information, leaving 93 supervisors to analyze. Of the 93 supervisors, 67 had a BCBA credential and 26 did not. Of the 26 supervisors without a BCBA credential, 4 reported a bachelor’s degree as their highest level of education and 22 reported a master’s degree as their highest level of education. The 93 supervisors had an average of 8.87 (SD = 4.71) years of experience in the field, ranging from 0.76 to 25.35 and a mean caseload of 11.18 (SD = 4.06), ranging from 2 to 23, as shown in Figs. 1 and 2. The supervisor variables are summarized in Table 1.

Fig. 1.

Distribution of supervisor years of experience

Fig. 2.

Distribution of supervisor caseloads

Table 1.

Summary of supervisor variables

Variable	Mean (SD)	Range
Years of experience	8.87 (4.71)	0.76 to 25.35
Caseload	11.18 (4.06)	2 to 23
Credential	BCBA = 67	No credential = 26

Data Analysis

To explore the role of supervision in the mastery of learning objectives, several linear regression analyses were carried out. Linear regression is a statistical technique for modeling the mathematical relationship between independent variables and dependent variables. In the simple case, this relationship consists on only one independent variable, x, and one dependent variable, y. Linear regression is also named because the underlying assumption of the model is that given a value for x, the predicted value of the dependent variable, ŷ can be explained with a simple line:

$$\widehat{y}=mx+b$$

In the equation above, the slope of the line, m, and the intercept of the line, b, represent the regression parameters to be learned given the sample data. While more sophisticated approaches exist, the most basic technique for determining the value of the regression parameters is the method of least squares. This corresponds to minimizing the sum of squared differences between the observed value of y and its predicted value, ŷ. Mathematically this corresponds to minimizing error, E, where E is defined as:

$$E={\displaystyle \sum {\left({\widehat{y}}_{\mathrm{i}}-y_{\mathrm{i}}\right)}^2\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{all}\mathrm{observations}\left(\mathrm{data}\mathrm{points}\right),i}$$

The simple case of a single independent variable can be generalized to several independent variables, in which case the resulting model is referred to as a multiple linear regression model. For a more thorough mathematical treatment of regression, readers may refer to Ross (2010).

Because the number of treatment hours and mastered learning objectives naturally span orders of magnitude, a logarithmic transform was applied before fitting the linear regression model. During the process of data analysis, it is often the case that the values of both the independent and dependent variables span over several orders of magnitude. For example, one participant may have only mastered five objectives in a given period of time, while another participant may have mastered 100. When it can be verified that this large variance is a legitimate facet of the data, and not driven by outliers, standard mathematical transforms can be applied to the data to reduce the skew caused by this variance, as well as improve the visual and mathematical interpretability of models applied to the data. A common data transform for this purpose is the logarithmic transform, which simply applies the logarithm function to variable values. The logarithm function is order preserving. This is important to note because order-preserving transforms guarantee that the numerical relationship of variables is maintained, which makes it an appropriate choice for this task.

Results

A linear regression analysis on treatment hours was found to be significant and demonstrated a strong linear relationship (see Fig. 3). The resulting R-squared value of 0.32, based on the regression model, suggests that over 32 % of the variance in mastery of learning objectives is accounted for by the number of treatment hours. That is to say that 32 % of the variance in the number of mastered learning objectives can be accounted for by treatment hours alone.

Fig. 3.

Treatment hours vs. mastered learning objectives (log transformed)

To augment this analysis to include supervision hours, the same linear regression was repeated on log-transformed data, this time, capturing the relationship between solely supervision hours and mastered learning objectives. Figure 4 provides the scatter plot of the data, along with the best-fit line. In this case, the R-squared value of the model drops to 0.26, accounting for substantially less variance in learning objectives than treatment hours. This simple exploration of supervision hours, however, fails to account for the fact that best practice recommendations suggest a direct ratio of supervision hours to treatment hours (e.g., 1–2 supervision hours for every 10 h of treatment), and thus supervision and treatment intensities are highly correlated.

Fig. 4.

Supervision hours vs. mastered learning objectives (log transformed)

To identify the full extent of the impact of supervision on mastered learning objectives, a multiple linear regression was performed (again on log-transformed data), using both treatment and supervision hours as the independent variables, with mastered learning objectives as the dependent variable. Table 2 shows the regression parameters for this model, in addition to the parameters for the single variable model. The R-squared for the multiple regression considering both supervision and treatment increased to 0.34, accounting for less than 2 % more variance than treatment hours alone. This is perhaps best demonstrated visually in Fig. 5, which provides a three-dimensional scatterplot of the corresponding regression model. Here, the slope of the treatment-mastered learning objective line is substantially higher than the slope of the supervision-mastered learning objective line.

Table 2.

Linear regression coefficients for supervision and treatment hours

	Supervision	Treatment	Supervision + treatment
Intercept	0.39	−0.54	−0.52
Supervision	0.95	–	0.38
Treatment	–	1.00	0.74
R 2	0.26	0.32	0.34
F test	p < 0.000	p < 0.000	p < 0.000

Fig. 5.

Joint effect of treatment and supervision hours on mastered learning objectives (log transformed)

To further understand the role of supervision in the efficacy of ABA-based treatment, the following three attributes of the ABA supervisors represented by the data set were studied: whether they held a BCBA certification, their number of years of experience, and their caseload. The analysis was conducted using standard regression models. The number of years of experience was found to be statistically significant with a p value of 0.05. Additionally, whether the supervisor held a BCBA certification proved to be statistically significant, resulting in an F value of 9.77 for α = 0.05. Table 3 provides regression coefficients for the three supervision attributes using mastered learning objectives as the dependent variable, with the p value for credential and experience providing the only statistical significance.

Table 3.

Regression coefficients for supervisor attributes

	Credential	Experience	Cases
Intercept	0.315	0.353	0.512
BCBA	0.232	–	–
Experience	–	0.015	–
Weighted cases	–	–	−0.053
Adj. R 2	0.087	0.031	−0.010
F test	p < 0.002	p < 0.050	p < 0.764

Discussion

The present study evaluated the relationship between mastery of learning objectives and elements of ABA supervision. When evaluated alone, a significant relationship was identified between the number of mastered learning objectives and the number of supervision hours. However, supervision hours were generally provided in a direct ratio to treatment hours (e.g., 1–2 h of supervision for every 10 h of treatment). Thus, participants who received greater supervision hours in the present study typically received higher treatment hours as well, which has previously been demonstrated to have a strong impact on treatment outcomes. To mitigate the impact of this confound, a multiple regression analysis was performed. When treatment hours and supervision hours were analyzed together, the addition of supervision hours improved the model’s ability to account for the observed variance by less than 2 %. To be clear, this does not imply that supervision hours have a low impact on mastery of learning objectives. Instead, the implication is that the variance within the bounds of typical supervision intensity (e.g., 2 h of supervision for every 10 h of treatment; BACB, 2014) results in a relatively small improvement in mastered learning objectives.

The relationship between mastery of learning objectives and supervisor credentials was examined in the present study. A significant correlation was found, revealing that supervisors with BCBA certifications produce 73.7 % greater mastery of learning objectives per hour as compared to supervisors without a BCBA. While the number of supervisors with BCBA certifications has grown since Love et al. (2009) reported that less than half of the surveyed supervisors to have BCBA certifications, a scarcity of BCBAs remains. Although requirements vary state to state, in many states, other professionals acting within the scope of their licensure are included among those who may supervise ABA programs. In the present study, samples of supervisors in these groups were either absent or not large enough to evaluate as standalone groups. Given the limited—albeit growing—number of BCBAs and the recognition by some states and funding sources of other licensed professionals, future research should examine the effectiveness of supervisors with BCBA certifications as compared to other licensed professionals practicing in the field.

A supervisor’s years of experience overseeing ASD cases were shown to have a significant effect on the mastery of learning objectives. Specifically, the analysis indicated that for every year of experience that a supervisor had, the number of mastered learning objectives increased by 4 %. This may be trivial when considering the impact of a single year but would indicate that cases that are supervised by practitioners with 10 years of supervisory experience are mastering 40 % more per hour. These data indicate that experienced practitioners should consider ways to share their knowledge and skillset with less-experienced clinicians through mentorship and consultation. The relatively weak statistical significance of the result indicates though that there are still numerous other factors that impact the number of mastered learning objectives. Simply having worked as a supervisor for a long period of time does not guarantee that performance will be better. It may be the case that clinicians improve over time due to experiencing a variety of different cases as well as continuing to train and hone their skills. Unfortunately, determining why there was a correlation between years of experience and increased number of mastered learning objectives was beyond the scope of the current data.

Supervisor caseloads were not found to have a significant relationship to the number of mastered learning objectives. This finding was unexpected, but it is consistent with the relatively weak impact that an increase in supervision hours has on mastered learning objectives. It is likely the case that, once a sufficient level of supervision has been provided, increasing supervision of a case does not improve the number of mastered learning objectives. Similarly, a supervisor with a smaller caseload would likely be able to provide more supervision to each case; as reflected in the analysis of supervision hours, however, the increased supervision hours did not result in a meaningful improvement.

In the current study, treatment and supervision hours were not randomly assigned. Hours were based on clinical recommendations and subject to authorization by diverse funding agencies. Thus, the treatment and supervision hours that an individual received may have been less than what was considered medically necessary by the treating clinician. Furthermore, the participants were not randomly assigned to supervisors. Case assignment was based on a variety of factors, including availability, clinical knowledge, and funding agency credential requirements. It stands to reason that more challenging cases may have been assigned to supervisors with greater experience. Furthermore, supervisors working on challenging cases may have had lighter caseloads than supervisors with less demanding cases. Future research should take into account these case-specific factors. ASD is a multifaceted disorder with each individual displaying unique symptom presentation and treatment response. The present study did not account for individual differences that are known to affect treatment response, including age, symptom severity, and skill level. In addition, a noteworthy limitation is that while interobserver agreement (IOA) is collected as a part of day-to-day clinical practice, these data were not stored in such a way as to be accessible for these analysis. Future researchers would do well to build IOA into their data tracking systems.

The present study examined supervision hours overall. Future research on supervision intensity should investigate the facets of supervision that have the greatest impact on mastered learning objectives to help guide clinical standards. For example, supervision involves a large variety of tasks, including treatment planning, parent training, direct observation, and therapist mentoring, among others. It is possible that particular tasks may improve outcomes more than others. For instance, preliminary evidence shows that support from supervisors positively impacts therapist self-ratings of performance and efficacy (Gibson et al., 2009). Moreover, it is possible that greater supervision intensity may benefit some treatment domains more than others. Similar findings have been revealed with respect to treatment intensity (Virués-Ortega, 2010).

The present study measured treatment outcome in terms of mastered skills. While standardized scales are more commonly used to measure outcome within ASD treatment literature, the measurement of acquired targeted skills may better show individualized progress that is comparable across groups (Matson & Goldin, 2014). However, this measure of outcomes is not without limitations; that is, degree of difficulty varies for each targeted skill, and targeted skills do not necessarily address core deficits of ASD (Fava & Strauss, 2014). As such, future research employing target mastery as a primary outcome measure could consider including only those targets directly associated with diagnostic criteria of ASD, including social communication, social interaction, and restricted, repetitive behaviors.

The findings reported in the present study have large implications. Best practice recommendations for supervision have been made by the BACB to provide needed guidance to funding agencies and to facilitate treatment integrity and effectiveness. Until now, little research has been conducted to substantiate those recommendations. While the results of the current study should be replicated in other samples and explored further, they indicate that the 1–2 h per every 10 h of treatment described in the 2012 version of the BACB guidelines may be more appropriate than the revised recommendations in 2014. Further, given the relationship between treatment response and the supervisor credential, it seems evident that BACB standards for behavior analysis have produced a meaningful certification. These standards, along with the aforementioned additional training in ASD treatment, may be the factors that enabled supervisors in the present study to take on greater caseloads and why, with hours of supervision per case that reflected the reduced 2012 BACB guidelines, supervisors were able to maintain strong clinical outcomes. Given that the current study found no relationship between mastered learning objectives and supervisor caseload, the optimal caseload should be reconsidered. Potentially, supervisor caseloads may be carefully and incrementally increased over time to expand treatment capacity while ensuring treatment quality and integrity.

Another implication of the current study is in regard to how funding resources are allocated. In real-world settings, treatment resources are always limited. Typically, consumers and providers alike often make hard decisions to trade one treatment component in favor of another in an effort to yield the greatest improvement for each individual with ASD. Given the relationship between treatment hours and mastered learning objectives in contrast to the relationship between supervision hours and mastered learning objectives, it seems likely that reallocating funding resources from supervision hours to treatment hours would yield better outcomes overall (see Fig. 5). That is to say, a 10 % increase in supervision hours would yield only a 3.6 % increase in mastered learning objectives. Rather, if those same hours were allocated to treatment, mastered learning objectives would improve by 7.3 %. This effect is further multiplied by the observation that reimbursement rates are often significantly higher for supervision than for treatment, meaning that funding for 1 h of supervision could potentially fund 2–3 h of treatment, which are hours that research consistently demonstrates to produce better outcomes for each child. While supervision is required to ensure progress and treatment integrity, exactly how much supervision is required is an empirical question.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

For this type of retrospective analysis, formal consent was not required.