The Best and Worst Things Staff Report About Behavioral Training Workshops: a Large-Scale Evaluation

Abstract

A variable affecting the success of staff training programs conducted by behavior analysts is trainee acceptance of the training. This study constituted a large-scale evaluation of staff acceptance of behavioral training workshops. Over a 10-year period, 646 human service staff who participated in 132 workshops were questioned regarding the best and worst thing about the workshops. The most common staff comments concerning the best thing pertained to the training content, followed closely by trainer style and then trainee activities. There were far fewer comments regarding the worst thing, with most involving aspects of the physical environment in which training occurred. Implications of the results for practitioners are offered in terms of conducting workshop training in accordance with trainees’ reported preferences. Emphasis is placed on ensuring training content is specific in nature as well as new and relevant for the trainees’ work situation, providing frequent demonstrations and examples, and structuring repeated opportunities for active trainee responding.

A common job responsibility of many behavior analysts involves training groups of human service staff. Conducting group training or training workshops is often needed to train staff within one or more agencies in the knowledge and skills to carry out behavioral procedures with agency clients (Parsons, Rollyson, & Reid, 2013). Training workshops are also conducted to disseminate more general information about the professional field of behavior analysis (Critchfield, 2014). For behavior analysts working in a consulting capacity, providing training workshops is reported to be among their most frequent work activities (Bailey & Burch, 2010, Chapter 11).

Successful workshop training results in participating trainees acquiring the knowledge and skills targeted by the behavior analyst trainers. However, there is a variable that affects the ultimate success of training workshops beyond trainers effectively promoting trainee knowledge and skill acquisition: the degree to which the training is acceptable to trainees (Strohmeier, Mule, & Luiselli, 2014). The importance of acceptability among recipients of behavioral procedures was emphasized relatively early in the history of behavior analysis (Kazdin, 1977; Wolf, 1978) and has continued to be highlighted (Hanley, 2010; Parsons, 1998). Participant acceptance, or social validity of training procedures and/or outcomes (Wolf, 1978), is considered to be linked to the general effectiveness of behavioral interventions with staff (Strohmeier et al., 2014) and particularly in regard to their long-term viability (Schwartz & Baer, 1991). In short, if training workshops are not well accepted by participating trainees, there is an increased likelihood that the training will not continue within an agency regardless of the continued need for the training and its effectiveness (Reid & Parsons, 2006, Chapter 4).

In light of the recognized importance of participant acceptance of training workshops, a number of investigations on methods of training human service staff have included evaluations of trainee acceptance. Typically the evaluations have focused on acceptance of the overall training program or workshop (e.g., Dyer & Karp, 2013; Pritchard et al., 2017; Schepis, Ownbey, Parsons, & Reid, 2000). In contrast, there has been much less research attention directed to trainee acceptance of various components of training workshops. Results of an investigation that did assess trainee acceptance of training components were relatively inconsistent across participants and components (Ward-Horner & Sturmey, 2012), suggesting that additional research is warranted to better identify the acceptability of different training components. Identifying various aspects of training workshops that are considered desirable and undesirable by participating trainees could help behavior analyst trainers develop their training interventions to maximize trainee acceptance and potentially, desired outcomes of the training (Luiselli, Sperry, & Draper, 2015).

The purpose of this study was to evaluate and summarize the best and worst things staff report about training workshops in a way that has relevant implications for behavior analyst practitioners. The intent was to provide information that may assist practitioners in designing and conducting workshops to coincide with what staff report they like best about training workshops and potentially enhance trainee acceptance. The information may likewise help trainers avoid certain things that staff commonly report they like least about workshops. The latter outcome could prevent the situation that occurs with a number of training programs in the human services in which staff reportedly dislike and even dread participating in the training (Test, Flowers, Hewitt, & Solow, 2004).

The evaluation of trainee reports of the best and worst things about behavioral workshop training was designed to be of a large-scale nature. A large-scale evaluation was conducted for two primary reasons. First, many evaluations of trainee acceptance of their training involved relatively small numbers of participants such as three or four trainees (Giannakakos, Vladescu, Kisamore, & Reeve, 2016; Higgins, Luczynski, Carroll, Fisher, & Mudford, 2017; Taber, Lambright, & Luiselli, 2017; Ward-Horner & Sturmey, 2012). Although clearly important for evaluating the training that was conducted, the small numbers limit the external validity of conclusions about acceptability. Second, trainee acceptance evaluations have typically focused on one workshop or program as referred to earlier, which can also limit the external validity concerning conclusions about acceptability. The current study assessed training acceptance among a large number of trainees across a variety of behavioral training workshops.

Method

Settings and Participants

The training settings consisted of classrooms and meeting rooms in human service agencies in which training workshops were conducted. The settings were located across the four geographic regions of a Southeastern US state. The training workshops represented one component of a long-term, multifaceted initiative in the state to improve behavior analysis and support services for adults with intellectual disabilities (Rotholz & Ford, 2003; Wagner, 2004). A total of 132 training workshops were included in the study that were conducted from 2007 to 2017. Participants in the training were primarily staff supervisors in adult service settings for people with intellectual and developmental disabilities including group homes, supported work sites, and center-based day programs such as adult education or activity centers and sheltered workshops. A small number of participants, representing less than 5% of all trainees, consisted of administrators, clinicians, and direct support personnel. Less than 1% of trainees were behavior analysts. There were 646 trainee participants in total with an average of 16 participants per training workshop (range of 6 to 24). The majority of trainees participated in more than one workshop. Each training workshop encompassed between 3 and 6 h of actual training time, with the majority (83%) encompassing between 5 and 6 h.

Training Content

The content for each training workshop was behavioral in nature. Two thirds of the workshops involved application of the Positive Behavior Support Training Curriculum (PBSTC; Reid, Parsons, & Rotholz, 2015). The PBSTC focuses on training staff in basic knowledge and skills for practicing behavior analysis in accordance with the values and goals of positive behavior support (see Reid et al., 2003, for additional information and evaluation of the PBSTC). Implementation of the PBSTC requires three and one-half days of workshop training, with each day involving 5 to 6 h of training. Each of these four training workshops covered different sets of topics (e.g., defining behavior, reinforcement, providing client choices, teaching functional skills and replacement behavior). Implementation also requires one day of on-the-job training in which instructors assess and train application of selected skills in the routine work site of each trainee.

The remaining third of the training workshops addressed three different topics. The topics were writing behavioral objectives and task analyses for teaching programs, more advanced teaching strategies than what is covered in the PBSTC, and effectively involving adults with developmental disabilities in meaningful day activities. Each training workshop on these topics encompassed between 5 and 6 h of classroom training time.

Workshop Composition

The training for all topics involved a performance- and competency-based format. Information on a specific topic was provided vocally by an instructor followed in turn by instructor demonstration where relevant, trainee activity (typically role plays but also some paper and pencil activities), and instructor feedback to trainees. Performance criteria were established for selected skill areas such as with trainee role-play demonstrations and quiz answers, and trainees were required to demonstrate mastery to complete the training. Prior to each workshop, trainees were provided with note-taking guides that summarized key points of the training content. Within each workshop a series of Microsoft PowerPoint slides was displayed that summarized key content points. Additionally, trainee questions were periodically prompted by an instructor during each workshop with subsequent responses to the questions. This training methodology is built into the PBSTC, including instructions to trainers, forms to use, criteria for trainee success, etc. (see Reid et al., 2003, 2015, for elaboration). For the other three workshop topics, instructors developed relevant demonstrations and trainee activities along with related forms such that the training methodology was the same as with the PBSTC.

Due to the frequent use of instructor demonstrations that required two people for a role play and provision of individualized feedback to trainees, two instructors conducted each workshop (various pairs of the authors). Three instructors had doctoral degrees (psychology or education) and one had a master’s degree in special education. All instructors had at least 20 years of practitioner experience in residential and/or day-support settings for adults with developmental disabilities prior to functioning as an instructor. Two instructors were board certified behavior analysts, at least one of whom was present for over 95% of the workshops.

At the conclusion of respective workshops, each trainee was provided with an acceptability form for evaluating the training. Trainees were instructed not to write their names on the form. Trainees were then directed to place the completed form in a pile on a designated table upon exiting the training room. Averaged across all workshops, 95% of the trainees turned in a completed form.

The acceptability form contained four questions to be answered on a 7-point Likert scale. The questions pertained to how much the training coincided with what the trainee was expecting, how well the instructors presented the material, the degree to which the information presented was practical, and how useful the training was overall. The form also contained one yes/no question regarding whether the trainee would recommend the training to a colleague, and two open-ended questions. In accordance with the purpose of this study, the focus was on the latter two questions (summaries of responses to the other questions are available from the authors). The first question asked the trainee “What was the best thing about the training?”. The second question asked “What was the worst thing about the training?”. After each workshop, an instructor recorded verbatim all trainee responses to the two questions onto a summary form for that respective workshop.

Coding of Trainee Comments

Trainee responses to each of the two questions were grouped into six categories and subsequent subcategories. Categories of responses were developed based on discussions regarding important aspects of training and public speaking from a behavioral perspective (Austin, 2000; Bailey & Burch, 2010, Chapter 11; Friman, 2014) as well as the authors’ training experiences. Category definitions were as follows: (1) content: any written comment that mentioned the information or subject matter presented during the training, (2) trainer style: any comment that mentioned the manner in which an instructor presented the material, including the knowledge of an instructor about the subject matter, (3) trainee activities: behavior in which trainees engaged in a structured manner as directed by an instructor, (4) physical environment: any comment about the setting excluding the audio/visual category, (5) audio/visual: any comment about the audio/visual tools used by an instructor, and (6) miscellaneous: any written comment that did not meet the definition for any of the other categories. Trainee comments could be coded in one category or multiple categories. The former situation occurred if the comment met the definition of only one category. For example, for a comment such as “the information presented” in response to the question concerning the best thing about the training, one comment would be coded within the content category. The latter situation occurred if the comment contained several phrases or descriptors that crossed categories. To illustrate, for a comment such as “the information presented, the role plays I did”, one comment would be coded within each of the content and the trainee activities categories, respectively.

Subcategories were developed using the following process for both open-ended questions. First, acceptability forms completed by trainees across three workshops were arbitrarily selected and one author coded the comments under respective categories. Second, a word or short phrase that appeared to best describe each comment within each category was listed as a corresponding subcategory. For example, if a comment in response to “What was the best thing about the training?” was “information presented was helpful”, then a subcategory for the content category was listed as helpful. Similarly, if a comment in response to “What was the worst thing about the training?” was “the chairs were too hard”, then a subcategory for the physical environment category was listed as seating. Third, additional acceptability forms from other arbitrarily selected workshops were reviewed by the same author and responses were coded according to the categories and subcategories that had been developed. If comments were reviewed that did not correspond to the existing subcategories at that point, new subcategories were developed as described in the second step. This process continued until at least 80% of all comments reviewed on acceptability forms from at least three other training workshops were accounted for by the existing subcategories.

The resulting subcategories for the “Best” and “Worst” questions are provided in Tables 1 and 2, respectively. Most of the subcategories under the “Best” question represent verbatim descriptors of trainee comments and appear self-explanatory. A few descriptors were combined into one subcategory as each descriptor appeared to have essentially the same meaning. For example, the relevant to work subcategory included the descriptors pertains to real life, practical, and applicable within the content category. Some subcategories required additional explanation to facilitate subcategorization of various trainee comments (see Appendix). Regarding subcategories under the “Worst” question, almost all represent verbatim descriptors of trainee comments. Those few that required additional explanation are also described in the Appendix. The entire process resulted in a comment-coding form with all categories and subcategories listed for trainee responses to both open-ended questions. Additionally, to account for a comment that might be reviewed on another acceptability form and may be highly idiosyncratic such that the comment would not correspond to the final subcategories, spaces were available on the coding form under each category for the coder to write in the entire comment. Subsequently, the first three authors coded trainee comments on the acceptability forms for all 132 training workshops within categories and subcategories. Coders were trained by reviewing the category and subcategory definitions and practice coding with feedback from the first author using completed forms that were not included in the interobserver agreement checks.

Table 1.

Categories and subcategories for trainee comments in response to “What was the best thing about the training?”

Categories	Subcategories
Content	useful/helpful	interesting
	valuable	specific information acquired
	important	general information provided
	relevant to work (pertains to real live, practical, applicable)
	increased knowledge (informative, new, improved understanding)
	hand outs
Trainer Style	demonstrations	clarity of presentation
	humor (jokes)	response to trainee questions
	examples provided	trainer knowledge
	feedback to trainees	trainers working together
	interaction with trainees	enjoyable (fun, entertaining)
	friendly (personable)	instructors
	attitude	hands on approach
	nonspecific praise for presentation
Trainee	role plays	paper/pencil exercises
Activities	interacting with peers	involvement in training
	interactions (nonspecific)	quizzes
	scenarios
Audio/Visual	PowerPoint slides
Physical	comfortable	relaxed seating
Environment
Miscellaneous	kept me awake	refreshments
	everything	motivational
	general praise	prizes
	getting out early

Table 2.

Categories and subcategories for trainee comments in response to “What was the worst thing about the training?”

Categories	Subcategories
Content	not useful	not valuable
	not relevant to work (not practical, applicable)
	insufficient information	not new (repeat, repetitious)
	too long to present	information could be condensed
Trainer Style	demonstrations	unclear presentation
	humor (jokes)	poor/insufficient examples
	poor trainer knowledge	trainers not working together
	interaction with trainees	boring
	instructors	not hands on enough
	instructor response to trainees
Trainee	role plays	paper/pencil exercises
Activities	quizzes	sitting too long
	scenarios
Audio/Visual	PowerPoint slides (not readable/visable)
	need/want videos
Physical	seating	temperature
Environment	accessibility	location
Miscellaneous	refreshments	nothing (none, no worst thing)
	everything or all was good or great

Interobserver Agreement

Interobserver agreement checks were conducted for comments coded for 29% of the workshops by two coders independently coding each comment. If both coders coded the same category or subcategory, respectively, an agreement was counted. If either coder coded a category or subcategory and the other coder did not code that category or subcategory, a disagreement was counted. Interobserver agreement was then calculated by dividing the number of agreements by disagreements plus agreements, multiplied by 100%, for each category and subcategory. For trainee comments to the “Best” question, interobserver agreement averaged 93% for the categories (range of 88% to 96%). For the subcategories, interobserver agreement averaged 82% (range of 80% to 85%). For trainee comments to the “Worst” question, interobserver agreement averaged 90% for the categories (range of 63% to 100%) and 87% for the subcategories (range of 50% to 100%). The lower ranges occurred with one category for the “Worst” question (trainee activities) in which there were very few trainee comments such that a small number of category or subcategory disagreements deflated the average. There was never more than one disagreement per subcategory for any interobserver check for that category.

Results

Trainee Comments Regarding the Best Thing About the Training

There were 2,319 total trainee comments concerning the best thing about the training. These comments are summarized according to the categories and subsequently, the subcategories.

Trainee Comments by Category

As indicated on Fig. 1, the category with the most trainee comments concerning the best thing about the training was the content category (36% of all comments; n = 840), followed closely by the trainer style category (32%; n = 750). The category with the third most comments was trainee activities (19%; n = 444), and the miscellaneous category had the fourth most comments (12%; n = 281). Both the physical environment and audio/visual categories had less than 1% of all comments (n = 1 and 3, respectively).

Fig. 1.

Percentage of all trainee comments regarding the best thing about the training workshops within each response category

Trainee Comments by Subcategory

Due to the large number of trainee comments, as well as the relatively large number of subcategories and the open-ended write-in comments, the following steps were taken to reduce the total data set to allow a focus on the most common and representative trainee comments in response to the question concerning the best thing about the training. First, only those subcategories that included at least 5% of all trainee comments within a respective category are presented. Second, individual write-in comments were eliminated from the presentation. Write-in comments for each category were highly individualized (e.g., “I thought I knew more than I did”, “meeting people with different jobs”), and no write-in comment was repeated sufficiently to represent as much as 2% of all written comments. The remaining subcategory comments, which accounted for an average of 89% of all subcategory comments per category (range of 70% to 97%) are summarized below.

Each subcategory that accounted for at least 5% of all trainee comments for the content category are shown on Fig. 2 (top left panel). The subcategory with the most trainee comments was specific information that was presented (42%), which accounted for more than twice as many comments than any other subcategory within the content category. The subcategory increased knowledge accounted for the second most trainee comments (17%), followed closely by the general information presented (13%), relevant to work (11%) and useful/helpful (11%) subcategories.

Fig. 2.

Percentage of trainee comments within each subcategory for each response category regarding the best thing about the training workshops

Subcategories of trainee comments within the trainer style category were more evenly represented than within the content category (top right panel on Fig. 2). Trainer demonstrations accounted for 14% of trainee comments, followed closely by examples provided (13%), trainer knowledge (12%), enjoyable (11%), instructors (10%), and nonspecific praise for presentation (10%).

Within the trainee activities subcategories, over 90% of the comments were accounted for by four of the subcategories (bottom left panel on Fig. 2). The subcategory with the most comments was role plays (45%) and interacting with peers accounted for the second most comments (25%). These two subcategories accounted for 70% of all comments regarding trainee activities. The other two subcategories that accounted for at least 5% of all comments, respectively, were involvement in training (12%) and interactions, nonspecific (12%).

Over 75% of comments within the miscellaneous subcategories (bottom right panel on Fig. 2) consisted of comments within the everything subcategory (47%) and the general praise subcategory (30%). The only other subcategories that accounted for at least 5% of the comments within the miscellaneous category were refreshments (10%) and prizes (10%).

The subcategories for the audio/visual and physical environment categories are not presented on Fig. 2 due to their very low number of comments. All subcategories combined for each of these categories, respectively, accounted for less than 1% of all comments in response to the “Best” question.

Trainee Comments Regarding the Worst Thing About the Training

Trainee Comments by Category

There were far fewer comments in response to the question concerning the worst thing about the training (672, representing 22% of all comments) relative to the number of comments concerning the best thing (2,319, representing 78% of all comments). The distribution of comments across categories concerning the worst thing is presented in Fig. 3, with the exception of the miscellaneous category (note the reduced range on the vertical axis due to the smaller number of category comments relative to category comments in response to the “Best” question on preceding figures). The latter category had far more comments than any other category regarding the worst thing (460 comments that constituted 69% of all 672 comments). However, the comments provided little information regarding what the trainees reported to be the worst thing because 95% of those comments were in the everything or all was good or great or the nothing (none, no worst thing) subcategories within the miscellaneous category. The remaining 5% were in the refreshments subcategory. That is, even though the question asked trainees what was the worst thing about the training, 95% of the responses within the miscellaneous category represented favorable comments about the training overall or indicated that there was nothing unfavorable.

Fig. 3.

Percentage of all trainee comments regarding the worst thing about the training workshops within each response category

For the categories regarding comments about the worst thing other than the miscellaneous category, the physical environment category accounted for 20% of the comments (n = 132) and the content category accounted for 8% (n = 54). Each of the other categories involving trainer style, trainee activities, and audio/visual accounted for no more than 2% of all comments (maximum of 15 comments per category). No further consideration is given to the latter three categories and their subcategories due to the very small number of comments.

Trainee Comments by Subcategory

The vast majority of comments within the physical environment category occurred within the accessibility subcategory (53% of all comments within the category) and temperature subcategory (33%). The location and seating subcategories accounted for 7% and 8% of the comments in the physical environment category, respectively. For the content category, the comments were relatively widely distributed across the following subcategories: not relevant to work (29%), too long to present (24%), not new (22%), insufficient information (18%), and information could be condensed (8%).

Discussion and Implications for Practitioners

In light of the large data set consisting of 2,991 comments across 132 training workshops, some explanation is warranted concerning how the implications to be presented were derived from the results. The major issue in this regard is that some subcategorizations and corresponding trainee comments do not allow for specific implications beyond speculation. An example is the subcategory of enjoyable within the trainer style category. An implication here is that to conduct a training workshop in accordance with what a number of trainees indicate they like best about a workshop, trainers should strive to present material in a way that is enjoyable for trainees. However, precisely how to present in a manner that is enjoyable for trainees is not clear. The practitioner implications to be presented are therefore based on those categories and subcategories for which the meaning of the comments seems sufficiently specific to be of use to practitioners.

Practitioner Implications Concerning the Best Thing About a Training Workshop

Implications for practitioners based on trainee responses concerning the best thing about workshop training pertain to how to design and conduct training in accordance with what many trainees like best about workshops. The assumption is that if a workshop includes those trainee preferences, there will be enhanced trainee acceptance of the training. Because this study was not experimental in nature and did not demonstrate functional control over trainee acceptance, the assumption aspect warrants emphasis. The study does not allow an experimentally based conclusion that trainee acceptance will be increased by including the best things trainees have reported about a workshop within training endeavors (see later comments on areas for future research for demonstrating functional control of trainee acceptance). However, given the importance of what recipients of interventions report about liking or not liking the interventions as established in social validity research (Luiselli et al., 2015; Strohmeier et al., 2014; Wolf, 1978), it would seem prudent for practitioners who design the interventions – in this case, training workshops – to attend to reported trainee preferences.

Content is Critical

Arguably the most important implication for practitioners based on trainee feedback is that the workshop content is critical. Noting the importance of the content of a training workshop in one sense seems to be stating the obvious; there would appear to be essentially no reason to present information that is not beneficial to trainees. In this regard, there is a wealth of research supporting the importance of information on behavior analytic principles and practices for human service staff. Therefore, if practitioners conduct workshops with behavioral content for staff in the human services, it seems that the target content has a good likelihood of being beneficial for trainees (in conjunction with other findings about content as described below). Sometimes, however, behavior analysts are requested to provide training within an agency on topics that have no relation to behavior analysis or the trainees’ routine work situation and may not have beneficial content for trainees. This type of training is sometimes conducted, for example, due to various regulatory standards that require certain types of training regardless of the trainees’ regular work duties (Test et al., 2004). Training may likewise be requested of behavior analysts to address a staff performance problem even though the problem is due to issues with staff application of skills rather than lack of skills. In the latter case, the training would likely be irrelevant to the staff trainees as well as resolution of the performance problem (Courtney, Hartley, LaMarca, Rosswurn, & Reid, 2018). It is recommended that behavior analysts carefully review the content of what they are expected to train and when possible, respectfully attempt to refrain from training on respective topics if they have serious concerns over nonbeneficial content. Alternatively, behavior analysts may be able to enhance required but potentially nonbeneficial training content with additional information that is more likely to be relevant to the trainees’ ongoing work.

In addition to the beneficial aspect of content from an overall perspective, results suggest content is best received if it is specific and particularly in regard to the trainees’ existing work situation. The specific information subcategory had more comments (n = 345) regarding the best thing about the training than any other subcategory across all categories. Examples of specific information reported included “how to do a task analysis” and “how negative reinforcement can affect the problem behavior of some of my individuals”. This type of information is more specific than describing what a task analysis is but not specifying how to perform one, and discussing the principle of negative reinforcement but not detailing how it can affect certain behavior among clients of the trainees, respectively. Additionally, results suggest that providing content that is new to trainees such that it increases their knowledge would be in accordance with what trainees like best about content, as would striving to make the content clearly relevant to the trainees’ work situation. The latter aspect would also seem related to ensuring that the content is useful or helpful for the trainees during their routine work. Providing content that is new, relevant, and helpful to trainees may require trainers to question trainees about their work situation and backgrounds at the beginning of a workshop if the trainers are not already familiar with the trainees. Our experience has indicated such questioning can provide some information about trainees’ likely knowledge and skills that can be helpful to trainers in terms of suggesting what they should and should not emphasize with their training topics.

Trainer Style is Almost as Critical as Content

Results suggest that the style in which instructors present the training content is almost as important as the content itself in regard to what trainees report as the best thing about training workshops. The two categories of content and trainer style alone accounted for two thirds of all comments concerning the best thing about the training. However, specifying how instructors should present the content based on the results is somewhat difficult due to the lack of detail with many trainee comments. This was reflected in the subcategory descriptors as noted previously and exemplified with the subcategory of enjoyable. Similar issues exist with the trainer style comments within the instructors and nonspecific praise subcategories; see Appendix for elaboration. Nonetheless, two implications do seem apparent based on the results. First, instructor demonstrations of key points or skills appear to be preferred by many trainees (e.g., in the workshops provided, demonstrating how to use a least-to-most assistive prompting strategy, how to provide a two-item choice to an individual who does not speak). These results correspond to other reports of generally high trainee acceptance of trainer demonstrations (Ward-Horner & Sturmey, 2012). Second, presentation of examples regarding how key points relate specifically to the work situation of trainees likewise appears important for trainee acceptance.

In considering the preferences reported for demonstrations and examples, it is important to note that all demonstrations in the workshops were instructor role plays. No visual media other than PowerPoint slides were used, such as videos. Our experience suggests that the same preferences expressed regarding the instructor role plays would extend to instructor presentation of videos, although research evaluating the relative advantages of live role plays versus videos specifically for trainee acceptance would be helpful. Research has supported the effectiveness of videos in staff training (e.g., Moore & Fisher, 2007; Weldy, Rapp, & Capocasa, 2014) but we are not aware of large-scale evaluations of trainee acceptance of video training.

Another implication related to the importance of demonstrations and examples provided, as well as trainer style in general, is the importance of instructor preparation and practice prior to providing a training workshop. This implication seems to be at least indirectly related to the results regarding trainer style, and comports with our experience and that of others (Bailey & Burch, 2010, Chapter 11; Friman, 2014). More specifically, it is recommended that behavior analysts carefully prepare how a workshop will be conducted, including what role-play demonstrations will be provided as well as what specific examples will be presented to elaborate on key points. Behavior analysts are likewise encouraged to practice their presentation prior to conducting workshops. Conducting training workshops involves a broad set of public presentation skills on the part of instructors, and proficiency and fluency of such skills can be improved with practice (Friman, 2014). Additional considerations regarding the style of instructor presentation in regard to trainee reports of the best thing about training workshops stem from the following summary of trainee comments within the trainee activities category.

Promote Active Responding Among Trainees

The most apparent implication regarding trainee comments within the trainee activities category is the significance of having trainees engage in role playing. The role plays subcategory had the second largest number of comments (n = 199) regarding the best thing about the training relative to all subcategories across all categories. In the workshops in this study, trainee role plays followed instructor demonstrations and specific instructions regarding what trainees should focus on within the role plays. The trainee role plays typically were followed by supportive and corrective feedback from an instructor.

A second implication regarding trainee activities is to incorporate opportunities for trainees to interact with each other during the training. These opportunities were provided in the workshops by having trainees periodically work in small groups of three or four to formulate consensus answers to questions about written scenarios, and then summarize their answers for the entire class. Opportunities for peer interactions also occurred through the trainee role plays.

Both role plays and structured group interactions facilitate active responding on the part of trainees. Two other subcategories within the trainee activities category (involvement in training and nonspecific interactions), each of which accounted for 12% of comments in this category, further support the benefits of promoting active responding in regard to trainee preference. The importance of active participant responding has been established within behavioral approaches to university teaching in regard to the effectiveness of the teaching (Austin, 2000) as well as within staff training programs (Dyer & Karp, 2013). Results here suggest that active trainee responding in behavioral workshops is also well received by many trainees.

Promoting active trainee responding through role plays and structured peer interactions can also relate to trainer style in regard to trainee preference. Specifically, when considering a training approach that involves instructor demonstrations (trainer style subcategory) along with specific trainee responding activities, trainees are provided with frequent changes in the format of a training workshop. Our experience (see also Friman, 2014) is that frequent format changes help promote trainee attentiveness during a workshop. Results here suggest that changes in format such as by interspersing demonstrations and trainee activities within instructor vocal presentation or lecturing also is well received by many trainees. In the workshops in this study, these types of format changes occurred at least every 15 or 20 min. It is recommended that behavior analysts plan and conduct training workshops with consideration of providing demonstrations and active trainee responding opportunities on that approximate schedule to comport with reported trainee preferences.

Practitioner Implications Concerning the Worst Thing About a Training Workshop

Results provide less implications regarding what trainees prefer the least about workshop training. There was a considerably smaller number of comments regarding the worst thing about the training relative to those about the best thing. Additionally, approximately two thirds of all comments to the “Worst” question were in the miscellaneous category indicating everything was favorable or nothing was unfavorable. Nonetheless, it would still appear advantageous to design and conduct workshops where possible to avoid things that multiple trainees report they do not prefer (Schwartz & Baer, 1991).

The most apparent implication of results concerning the worst thing pertains to the physical environment in which training is conducted, although practitioners may not have much control over this aspect. If practitioners can have some control over the workshop environment, results imply several considerations. First, if possible, training locations should be selected that require the least amount of travel for participants to access. Second, efforts appear warranted to make the environment as comfortable for trainees as reasonably possible. This consideration also relates to practitioner preparation as referred to earlier. It is recommended practitioners examine the training environment prior to when the workshop will be initiated (see Bailey & Burch, 2010, Chapter 11, for elaboration). The examination should focus on the likely comfort of the seating arrangement, and making adjustments to the seating if relevant such as replacing hard chairs with softer chairs. It is likewise recommended that practitioners evaluate the temperature control mechanism in the workshop training room and be able to adjust the temperature if possible.

A somewhat related consideration pertains to providing refreshments for trainee participants. The refreshments subcategory accounted for 5% of the comments within the miscellaneous category regarding the worst thing about the workshop. However, the refreshments subcategory accounted for 10% of the comments within the miscellaneous category for the “Best” question. Consequently, it appears somewhat more trainees prefer refreshments than those who report disliking the refreshments provided.

Finally, comments within the content category regarding the worst thing generally support the implications previously discussed with the content comments concerning the best thing. This is particularly the case in regard to ensuring the content is relevant to the trainees’ work situation and represents new information. The comments also imply that practitioners should strive to present the content in as succinct a manner as possible.

Qualifications and Directions for Future Research

In considering the results and corresponding implications, several qualifications warrant mention. In particular, the focus of this study was on written reports of the trainee participants. Although investigations on trainee acceptance typically focus on written and other verbal reports (Parsons, 1998; Parsons et al., 2013), such reports are not necessarily considered the most behavioral measure of acceptability (Schwartz & Baer, 1991). Also, verbal ratings of acceptability by staff recipients of behavioral interventions have typically been positive whereas more behavioral measures have been less consistently positive (Parsons, 1998). Future research seems needed to focus on more behavioral or directly observable measures of trainee acceptance. For example, investigations could expose trainees to workshops with varying instructional formats and then provide the trainees with choices of workshops in which to subsequently participate that differ according to the various formats (Schwartz & Baer, 1991).

A related area for future research is demonstrating functional control of various training formats over trainee acceptability responses. As noted earlier, this study was descriptive in nature in terms of evaluating trainee comments about the workshops in which they participated. Investigations would be useful to compare acceptability measures among trainees with repeated workshops in which certain features are systematically incorporated or eliminated.

Another qualification pertains to the format of the workshops evaluated. All workshops were based on a performance- and competency-based model as described earlier. It is not clear if the results would generalize to workshops that do not involve such a format. In this regard though, use of evidence-based practices within staff training typically requires a performance- and competency-based format such as with behavioral skills training (Courtney et al., 2018).

Concerns over generalization of the results also relate to three other features of the workshops that were conducted. First, although there were large numbers of trainees and workshops relative to existing research on trainee acceptance of training, the workshops were conducted by only four instructors. Second, overall the workshops appeared to be consistently well received by participating trainees, which coincides with other reviews and discussions of trainee acceptance of behavioral training programs (Parsons, 1998; Reid, Parsons, & Green, 2012, Chapter 4). To illustrate, in response to the question on the acceptability form regarding if the trainee would recommend the training to his/her colleagues, 99% of all trainees indicated they would recommend the workshop. Third, the trainee participants were limited primarily to supervisors in the human services, and most supervisors participated in more than one workshop. The limited number of instructors, general acceptance of the workshops, and restriction of the participant population to mainly supervisors should be taken into account when considering the potential generalizability of the results to other training situations and trainees.

An additional qualification with the results and implications for practitioners stems from the open-ended format of the two questions asked of the trainees. Although the questions corresponded to the purpose of the study in terms of assessing what trainees view as the best and worst things about behavioral training workshops, their open-ended nature may have contributed to the lack of clarity of a number of the comments. Future research could evaluate other and perhaps more pointed questions to obtain more specific comments about trainee acceptance.

There is also a practical qualification or at least a consideration with the implications and corresponding recommendations. Several recommendations may require certain time or resource investments of behavior analyst trainers such as accessing training settings to check for likely comfort for trainees prior to training, securing refreshments, and practicing demonstrations for subsequent presentation. The degree to which behavior analysts can realistically adhere to the recommendations that may increase time and cost will likely have to be determined on an individual basis in light of their other existing responsibilities.

A final qualification pertains to the role of trainee acceptance regarding the overall success of staff training endeavors. Trainee acceptance must be considered in conjunction with the effectiveness of staff training interventions; it will be of little if any value if interventions are well received by recipients but the interventions do not affect the recipients’ performance in a beneficial manner. This study focused on staff acceptance rather than effectiveness (although assessments were built into each workshop in terms of requiring trainees to demonstrate criterion performance in selected skill areas as described earlier). The point of concern here is that while trainee acceptance is important and warrants attention with staff training programs, such attention should occur in conjunction with, and not in lieu of, a focus on training effectiveness.

Summary of Conclusions for Practitioners

In summary, results of this study suggest several recommendations for behavior analyst trainers to consider when designing and conducting behavioral training workshops to correspond with reported preferences of trainees. Primarily, it is recommended that trainers: (1) ensure the workshop content is new and relevant for trainees, and specifically relates directly to their work situation, (2) provide demonstrations and specific examples that are well prepared prior to the training, and (3) provide repeated opportunities for active trainee responding (e.g., through role plays or other practice activities) as well as opportunities for trainees to interact with each other and the trainer. It is additionally recommended that whenever possible, trainers avoid scheduling training in settings that require significant travel of trainees to access and are likely to be physically uncomfortable for trainees.

Appendix

The following information was added to selected subcategories to facilitate subcategorization of trainee comments. Each subcategory with additional information is listed followed immediately by the category within which the subcategory was recorded (in parentheses).

Additional Information for Subcategories of Comments in Response to the Question “What was the Best Thing About the Training?”

Specific information acquired (content): a comment that specified a particular piece of knowledge or skill (e.g., “how to do a task analysis”, “how negative reinforcement affects problem behavior”).

General information (content): a comment about what was presented with no specificity (e.g., “the information the trainers presented”).

Instructors (trainer style): a comment only listed “instructors” or “trainers” with no other descriptor.

Nonspecific praise for presentation (trainer style): an approval statement about the presentation with no other specificity (e.g., “good presentation”, “well presented”).

Nonspecific interactions (trainee activities): a comment referred just to “interactions” with no other descriptor.

Scenarios (trainee activities): scenarios referred to detailed descriptions of situations an instructor had experienced that were incorporated into trainee role plays or paper/pencil activities (e.g., trainees were directed to identify likely reinforcers for problem behavior on written descriptions of past events).

General praise (miscellaneous): an approval comment that was not specific to any other category (e.g., “it was good”).

Prizes (miscellaneous): a comment referred to an item provided to trainees at the end of some of the workshops (e.g., a button with a slogan relevant to the content of the training, a pen with a slogan).

Additional Information for Subcategories of Comments in Response to the Question “What was the Worst Thing About the Training?”

Accessibility (physical environment): a disapproval comment about getting to the training site (e.g., “the drive to get here was too long”, “directions to the center were not clear”).

Location (physical environment): a disapproval comment about the training site that did not include information about accessibility (e.g., “the room was too noisy”).

Everything or all was good or great (miscellaneous): Even though the comment was in response to what was “Worst” about the training, the comment indicated general approval for the overall training.

Compliance with Ethical Standards

This study did not include any experimental procedures with human participants. It drew instead on anonymous archival records obtained during the routine course of public agency practices. Therefore, Institutional Review Board oversight and conventions of informed consent do not apply.

Conflicts of Interest

The authors declare they have no conflicts of interest.