EFFECTS OF VIDEO MODELING ON TREATMENT INTEGRITY OF BEHAVIORAL INTERVENTIONS

Abstract

We examined the effects of individualized video modeling on the accurate implementation of behavioral interventions using a multiple baseline design across 3 teachers. During video modeling, treatment integrity improved above baseline levels; however, teacher performance remained variable. The addition of verbal performance feedback increased treatment integrity to 100% for all participants, and performance was maintained 1 week later. Teachers found video modeling to be more socially acceptable with performance feedback than alone, but rated both positively.

Treatment integrity refers to consistent and accurate implementation of an intervention in the way it was planned (Gresham, 1989). Direct measurement of treatment integrity is critical because higher integrity usually produces better intervention outcomes (DiGennaro, Martens, & Kleinmann, 2007; DiGennaro, Martens, & McIntyre, 2005; Wilder, Atwell, & Wine, 2006). In addition, when interventions are ineffective, measures of treatment integrity allow determination of whether the intervention should be modified or might work if it were implemented with higher fidelity (Arkoosh et al., 2007).

Only a handful of studies have examined strategies for improving treatment integrity of behavioral interventions that are explicitly designed to reduce challenging behavior (e.g., DiGennaro et al., 2005, 2007). In these studies, despite receiving initial training reflective of good training practices (e.g., live modeling, coaching, in vivo performance feedback), teachers demonstrated poor treatment integrity when asked to implement behavior support plans independently. Thus, an effective and resource-efficient training technology is needed to prevent treatment-integrity failure during treatment of problem behavior. Video modeling has proven to be beneficial in promoting high integrity in implementing instructional techniques and conducting functional analyses (Catania, Almeida, Liu-Constant, & DiGennaro-Reed, 2009; Moore & Fisher, 2007) and might be beneficial for treatment of challenging behavior as well. For example, Catania et al. taught 3 new teachers to conduct discrete-trial training following video modeling. High integrity was maintained during follow-up and generalized across teaching tasks.

The current study aimed to extend the literature on training procedures to produce and maintain high levels of integrity during treatment of problem behavior. We examined the impact of individualized video modeling and direct performance feedback on procedural implementation by 3 newly employed teachers.

METHOD

Participants and Setting

We recruited 3 teachers during their initial orientation as new employees in a setting that provided educational and residential services to students with autism, brain injury, and other developmental disorders. The teachers were told that the study was to investigate ways to help teachers implement behavioral interventions designed to reduce students' problem behaviors.

Kelly was a 24-year-old Caucasian woman with less than 1 month of experience working with individuals with autism. Although she had just enrolled in a graduate-level program that provided coursework toward certification as a behavior analyst, her only training in behavior analysis was that which was provided during the agency orientation. Lauren was a 28-year-old Caucasian woman with an MA degree in education. She had 4 years of experience working with individuals with autism in both public and private school settings. Her formal training in behavior analysis consisted of an introductory course. Shannon was a 35-year-old Liberian woman with a BA degree who spoke English fluently as a second language. She had no prior experience working with individuals with autism or implementing behavioral interventions but had taken one course in college that covered the principles of behavior analysis.

All training, intervention implementation, and performance feedback occurred in the teachers' work settings. Kelly worked in a community-based residence operated by the agency. Lauren and Shannon worked in separate classrooms at the school. The teachers viewed the video models in a private location within their setting (e.g., a small office or staff room).

Design and Measurement

A concurrent multiple baseline design across participants was used to evaluate the effects of individualized video modeling (IVM) and individualized video modeling plus performance feedback (IVM+PF) on the teachers' treatment integrity. Treatment integrity was assessed through 15-min direct observations of the participant implementing a student's behavioral intervention in a classroom or residence. Depending on student behavior, participants may have had multiple or no opportunities to implement different portions of the behavioral intervention. For example, consequence strategies for appropriate and inappropriate behavior might be delivered repeatedly and not at all, respectively, if the student did not exhibit problem behavior during the observation. Observers used a unique task analysis for each participant that allowed observers to score each intervention step as (1) implemented as written every time, (2) not implemented as written sometimes or never, or (3) no opportunity to observe, teacher had no opportunity to exhibit. Treatment integrity was calculated by dividing the number of intervention steps implemented as written (every time) by the total number of intervention steps implemented, and this ratio was converted to a percentage.

At the conclusion of the study, participants completed a 15-item treatment acceptability questionnaire adapted from the Intervention Rating Profile-15 (Martens, Witt, Elliot, & Darveaux, 1985) to assess the acceptability of the training strategies used to improve treatment integrity. Items were rated on a Likert-type scale ranging from 1 (strongly disagree) to 6 (strongly agree), with higher scores representing intervention acceptability. All 3 participants received the form 1 day after the final participant's follow-up probe and returned it anonymously through interoffice mail.

Procedure

Baseline

The interventions in place for each student were part of a clinical treatment plan developed based on a descriptive functional assessment. The primary author provided and verbally reviewed with each participant a written protocol detailing the steps of the intervention for her student. After a brief question-and-answer session, a five-item written knowledge posttest was administered. Responses were reviewed immediately with teachers, and errors were corrected. Initial training lasted approximately 60 min. One day after completion of the posttest, the teacher began treatment implementation while the student was engaged in a 15-min independent work task (e.g., activity schedule, household chores, typing). Feedback was not provided during baseline sessions. Participants were told, “do your best,” when they asked questions of observers during actual plan implementation.

Individualized video modeling (IVM)

Prior to each observation period, each participant viewed an individualized instructional video that depicted a model (e.g., an experienced teacher) demonstrating accurate implementation of all of the intervention steps with a “student” (e.g., an employee who agreed to role play). Kelly viewed a 6-min 56-s video depicting functional communication training, differential reinforcement, a picture activity schedule, and escape extinction in 20 discrete steps. Lauren viewed a 6-min 21-s video of a picture activity schedule, differential reinforcement, nonexclusionary time-out, least-to-most prompting, and token training with 14 discrete steps. Shannon viewed a 4-min 14-s video of functional communication training, transition warnings, differential reinforcement, and escape extinction in 17 steps. A voice-over component and on-screen text detailed relevant aspects of the procedures. Within 45 min of viewing the video, teachers were asked to implement the intervention with their students.

Individualized video modeling plus performance feedback (PF)

In addition to IVM, the experimenter provided verbal feedback about prior session performance before playing the video. During video viewing, the experimenter paused the tape at relevant points and asked the participant to pay attention to the next segment because of errors exhibited in the previous teaching session. If no errors occurred, the researcher stated such and played the video with no pauses.

Follow-up probe

One week following termination of IVM+PF, an implementation observation was conducted using procedures identical to baseline. Teachers did not view the video or receive feedback during this one-session probe.

Interobserver Agreement and Procedural Fidelity

A second observer simultaneously but independently collected data on teacher behavior during 31% (Kelly), 47% (Lauren), and 42% (Shannon) of sessions. An agreement was scored when both observers scored a teacher's implementation of an individual treatment step identically (i.e., as correct or incorrect). Agreement was calculated as the number of agreements divided by agreements plus disagreements and converted to a percentage. Mean percentage agreement was 96% (range, 95% to 100%) for Kelly, 95% (range, 90% to 100%) for Lauren, and 98% (range, 96% to 100%) for Shannon. An independent observer also collected data on presentation of the video model in its entirety for 64% of sessions during IVM (procedural fidelity was 100%). During 100% of sessions of IVM+PF, the observer also collected data on the accuracy with which the experimenter delivered feedback. An agreement was scored when the experimenter accurately delivered feedback to the participant for each treatment step consistent with performance during the previous observation session. Agreement was calculated as the number of agreements divided by agreements plus disagreements and converted to a percentage. Mean procedural fidelity of performance feedback was 99%.

RESULTS AND DISCUSSION

Figure 1 depicts treatment integrity as the percentage of intervention steps implemented correctly during baseline, IVM with and without feedback, and follow-up. During baseline, mean percentage correct was 41% (range, 23% to 67%). Percentage correct increased immediately during the IVM condition for all participants (M  =  84%; range, 20% to 100%), but was variable for 2 participants, and a decreasing trend was evident for Kelly. Adding performance feedback produced 100% correct implementation that was maintained at 1-week follow-up for all participants. Intervention acceptability ratings fell between 4 and 5 (M  =  4.06 for IVM and M  =  4.93 for IVM+PF). Although both were acceptable interventions, IVM+PF was more acceptable and more effective than IVM.

Figure 1.

Percentage of steps implemented correctly across conditions for each teacher. IVM+PF  =  individualized video modeling plus performance feedback.

These findings support previous research illustrating the benefits of video modeling for procedural implementation (Catania et al., 2009). Initial training procedures produced mediocre to poor baseline performance, whereas video modeling increased accuracy of implementation. Similar to previous studies demonstrating the benefits of performance feedback (e.g., Codding, Feinberg, Dunn, & Pace, 2005; DiGennaro et al., 2005, 2007), adding performance feedback to video modeling produced even better implementation even though only one session included corrective feedback because of subsequent perfect treatment implementation by all teachers.

Several limitations exist and are worthy of note. First, participant reactivity may have affected performance; thus, automated covert observation is an area for future research. Second, not answering questions about plan implementation during baseline may have created a relatively impoverished training environment compared to most clinical environments. Third, participants did not always view the video immediately prior to the observation session, although the delay never exceeded 45 min. It is unclear whether the duration of delay could affect performance, a question that remains unanswered about video modeling in general.