Evaluating the Quality of Behavior Analytic Practitioner Training Programs

Abstract

Dixon and colleagues (2015) asserted that faculty research productivity is a key indicator of the quality of university programs that train future practitioners of behavior analysis. Based on their analysis of publications in select journals, the authors concluded that many faculty in such programs have published little to no research. Some alternative measures of both faculty research productivity and the quality of practitioner training programs are suggested here.

Evaluating the Quality of Behavior Analytic Practitioner Training Programs

Dixon et al. (2015) raised important questions about graduate training in applied behavior analysis. Their rankings of graduate programs in terms of faculty research productivity were couched in several uncontroversial assumptions: Standards for evaluating training programs are desirable; indices of program quality can be helpful; our discipline needs skilled researchers; and practitioners-in-training benefit from exposure to research. Assumptions that warrant examination include those underlying the authors’ finding of a discrepancy in the research productivity of faculty in two sets of programs and the implication that programs in which faculty research productivity appears low are inferior. Those assumptions and alternative measures of faculty research productivity and program quality are considered here.

Alternative Measures of Research Productivity

An evaluation of training programs must begin by selecting programs for inclusion. Dixon and colleagues sought to compare faculty in programs accredited by the Association for Behavior Analysis International (ABAI) with faculty in what they characterized as “BACB [Behavior Analyst Certification Board] training programs” (pp. 7 and 15). Unfortunately, that is like comparing apples with oranges, because the BACB does not accredit, approve, or endorse programs. It only pre-approves courses and practica that fulfill its coursework and experiential training eligibility requirements. Thus the two sets of programs compared by Dixon et al. were not well-matched, which likely affected the findings. For instance, the ABAI set probably included a larger proportion of doctoral programs, which may have more factors in place to foster research than many master’s-degree programs. Future evaluations might include only training programs that are accredited by ABAI and have BACB-approved course sequences and practica, and separate doctoral from master’s degree programs. It might be best to conduct such evaluations when there is a larger pool of programs that meet the foregoing criteria than there is today.

Faculty research productivity could be measured in several ways. Dixon and colleagues examined publications in two journals published by the Society for the Experimental Analysis of Behavior and four published by ABAI. One of the latter—The Behavior Analyst—does not publish original research; it publishes literature reviews, reinterpretations of published data, discussion articles, and articles on philosophy. Since Dixon et al. counted articles in those latter categories, they actually measured publishing productivity, not research productivity. Their inclusion criteria may have favored programs that prepare students for careers in academia and research. A more accurate picture of the research productivity of practice-oriented faculty could be obtained by counting only research publications and searching additional journals in behavior analysis, education, human development, developmental disabilities, psychology, rehabilitation, and organizational behavior management.

Alternative Measures of Program Quality

Ideally, measures of program quality should include data from experimental analyses of relations among faculty, practitioner, and client repertoires. It would be interesting to see how many of those relations have been examined in experimental—or even correlational—studies and what those studies have shown. Dixon and colleagues cited a correlation between faculty publications and employer satisfaction with graduates of psychology programs (p. 9). Employer ratings of graduates of behavior analytic programs could be valuable, but direct, objective measures of graduates’ repertoires might better reflect program quality. One available measure is BACB certification exam pass rates. Future evaluations might examine relations among those data and various measures of faculty performance.

The question of whether practitioners must be trained to conduct research has been debated extensively (Critchfield 2015; Kelley et al. 2015; Moore and Shook 2001). There appears to be little research on that question and many opinions. Dixon et al. echoed an opinion expressed by some behavior analysts (e.g., Baer 1992; Reid 1992), but others have argued that practitioners-to-be should be trained to apply behavioral technologies and evaluate their effects on clients, not to conduct research (e.g., Johnston 1996; Malott 1992; Shook et al. 2002). Still others have suggested that practitioners need to be critical consumers of research (e.g., Green 2010).

Instead of relying on a few opinions, many professions use job analysis studies to identify essential competencies and the content of credentialing examinations, exam eligibility requirements, training curricula, and credentialed professionals’ scope of work. There are well-established legal and psychometric standards for conducting such studies. They typically entail having panels of experts develop a list of knowledge and skills required of a competent practitioner. That task list becomes the contents of a survey on which a large pool of professionals rate the importance of each task. The resulting data are used to determine which tasks remain on the list. The final task list is used to develop exam items and often to set formal and experiential training requirements. Several job analyses of the practice of behavior analysis have been conducted by the BACB and the Florida Department of Children and Families. They involved thousands of behavior analysts—basic and applied researchers and academics as well as practitioners (Behavior Analyst Certification Board 2011; Shook et al. 1995, 2004).

The task list resulting from the most recent BACB job analysis includes behavior analytic measurement and research designs, so those topics must be covered in BACB-approved courses (Behavior Analyst Certification Board 2015). Although research has not yet determined if proficiency in publishing research is necessary or sufficient for training others in research methods, performing a set of tasks and training others to perform them clearly involve distinct repertoires. Therefore, if faculty research productivity is used to index training program quality, it should be accompanied by some evidence of proficiency in implementing effective procedures for training graduate students in research skills.

The aforementioned job analysis studies have identified many additional competencies for behavior analytic practitioners. Abundant research suggests that to develop those competencies, faculty must be skilled in describing, demonstrating, observing, and providing feedback on trainee performance of a very large array of procedures for assessing behavior-environment interactions, producing generalized behavior change, and arranging systems to support effective service delivery, among others. It follows that objective measures of those repertoires should be weighted as much as or more than faculty research productivity in evaluating the quality of behavior analytic practitioner training programs.