On the Complexity of Correlating a Graduate Program’s Experiences with the Success of its Graduates: a Response to Dixon et al. (2015)

Abstract

Dixon et al. Behavior Analysis in Practice 8:7–15, 2015 evaluated the scholarly productivity of instructors in graduate-level, behavior-analytic training environments as a potential quality metric related to practitioner training. In our reply, we discuss the authors’ premise and methodology, suggest alternative conceptualizations, and recommend a more comprehensive and germane approach to the task.

Dixon et al. (2015) evaluated the scholarly productivity of instructors in course sequences approved by the Behavior Analyst Certification Board^® (BACB^®) and graduate programs accredited by the Association for Behavior Analysis International (ABAI). This effort was contextualized as an attempt to provide data for the ranking of behavior-analytic graduate programs. However, the authors’ choice of metric and focus on practitioner training strongly suggests a premise that graduate program experiences led by productive scholars would result in program graduates who were more successful as practitioners. In general, the authors found a lack of scholarly productivity among instructors of behavior-analytic graduate programs and they drew tentative conclusions about the implications of this finding on practitioner training. Although we value the overarching spirit of the authors’ initiative—identifying the training variables that produce the most successful practitioners—we have some concerns about their premise and methods.

The first point we wish to address is a fundamental misunderstanding Dixon et al. (2015) exhibited in their core categories. The authors included both BACB-approved course sequences (ACSs) and ABAI-accredited programs in their analysis. However, these classifications represent functionally distinct systems with different purposes and they include numerous disparate training environments. BACB approval of a course sequence is merely an acknowledgement that an institution teaches the courses required for a BACB credential (Shook and Johnston 2011), not that the sequence is “… deemed to adequately inform about core behavior-analytic practices” or that it has an “endorsement” from the BACB (Dixon et al., p. 8). Approved course sequences may exist within a defined graduate program in behavior analysis, or they may be offered as a stand-alone educational experience. Alternatively, ABAI accreditation is reserved for defined graduate programs and involves a comprehensive review of multiple aspects of a program’s environment. These are not comparable systems and were not designed to be. The ACS system was developed to reduce the response effort associated with the application process for certification (e.g., an applicant from an ACS does not need to submit individual course syllabi to the BACB for review) and, thus, should never be characterized as a mark of quality or an endorsement from the BACB.

Dixon et al. (2015) found that faculty in ABAI-accredited graduate programs (which, incidentally, all include BACB-approved course sequences) publish more than those in programs that are not accredited. Because ABAI accreditation is reserved for full graduate programs, the authors’ comparison fundamentally consisted of accredited graduate programs versus nonaccredited graduate programs + stand-alone course sequences. The aforementioned finding is not surprising; most would probably predict less scholarly activity in a stand-alone course sequence compared to a defined graduate program. This is not to say that stand-alone course sequences should not be evaluated or that they do not occupy a recognized training role in the profession, but they should be separately evaluated. We encourage future researchers who plan to correlate training experiences with subsequent student outcomes to attend to these important training environment differences.

As previously mentioned, the premise of the Dixon et al. (2015) study appears to be that faculty who are active in a scholarly activity and who participate in graduate student training will produce more scientifically oriented practitioners. This premise is rooted in the scientist-practitioner model of graduate student training (Frank 1984). The commonly valued outcome of the scientist-practitioner training model is a practitioner who contributes to the research literature. However, Hayes et al. (1999) redefined and expanded the scientist-practitioner model as being comprised of three practitioner activities: producing original research, conducting “in-house” experimental evaluations, and locating and consuming research that is relevant to practice. Because the probability of practitioners producing original research is so greatly influenced by current environmental contingencies (see Kelley et al. 2015), it seems that efforts to identify the “research culture” of graduate programs, and their implied differential outcomes, might better be focused on the other two scientist-practitioner training outcomes of being able to behave experimentally and consume the research literature. It is unclear whether the research cultures identified by Dixon and colleagues produce these outcomes, as they wondered themselves. Indeed, it is conceivable that these valuable outcomes could be achieved by faculty who do not frequently publish and that a more direct predictor of these outcomes might be whether students were required to engage in these repertoires during graduate school (independent of how prolific their faculty were).

We acknowledge the difficulty associated with any attempt to identify the training variables predictive of subsequent high-quality practitioner repertoires. To be fair, Dixon et al. (2015) recognized that “… research productivity is only one metric, among many, that consumers may wish to consider when comparing programs” (p. 9). Even so, the authors presented only one variable in an implied correlation. Correlations include two variables: the predictor variable (e.g., faculty scholarly productivity) and the criterion variable (e.g., successful practitioner repertoires). A criterion variable was not included in the Dixon et al. investigation. Of course, correlational research designed to answer questions about the critical aspects of graduate training is truly complex and requires substantial response effort. However, even a cursory analysis of variables that might comprise such a study yielded numerous possibilities. For example, relevant predictor variables might include faculty scholarly productivity (as in Dixon et al.), student research requirements, specific courses, practicum and supervision experiences, and direct training on consuming the scholarly literature. Relevant criterion variables might include BACB examination pass rates, postgraduate publications and presentations, employer and consumer surveys of practitioner performance, and engagement in the discipline (e.g., journal subscriptions, conference attendance, professional association membership). Reconsidering the question of whether faculty scholarly productivity is related to scientist-practitioner-type student outcomes, there are predictor (e.g., practicum and supervision experiences) and criterion (e.g., a survey of practitioner performance) variables above that might be more relevant to the question. We encourage future researchers interested in this question, or the broader question of what constitutes quality behavior-analytic training, to consider a fuller range of predictor and criterion variables (see also Moore and Shook 2001).

Critchfield (2015) chose to publish the Dixon et al. (2015) article based on its “…potential to stimulate a thoughtful discussion…” (p. 5). Indeed, we expect that the numerous replies to the Dixon et al. article will constitute an interesting discussion. However, based on our aforementioned points, we believe the Dixon et al. data should be considered preliminary and that a more extensive effort is required to begin to answer questions about quality behavior-analytic training. That said, the unambiguous result from the Dixon et al. study is that there is a shortage of contributors to the discipline’s scholarship, at least among its faculty. This is problematic given the number of unanswered research questions that remain. Perhaps a separate conversation about this discrepancy is warranted in the scholarly literature.