Implementing the Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP): Teaching Assessment Techniques

Abstract

The Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP) is an assessment tool used with individuals diagnosed with autism spectrum disorder and other language delays (Sundberg 2008). The milestones assessment section of the VB-MAPP is used to determine an individual’s current skill level. The results of the milestones assessment can be used to identify instructional goals and objectives. The current study examined the effects of behavioral skills training (BST) on the administration of the milestones assessment by two educational professionals. The BST intervention resulted in immediate increases in performance for both participants.

The Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP) is a five-component program designed to measure verbal behavior, guide individualized instruction needed to address deficits in verbal behavior, and evaluate progress over the course of a treatment program (Sundberg 2008). The assessment is used to evaluate performance on Skinner’s (1957) verbal operants across a number of tasks. The milestones assessment is divided into three developmental levels (0–18, 18–30, and 30–48 months), based on the attainment of developmental milestones by typically developing children. The current study focused on levels 1 and 2 of the milestones assessment of the VB-MAPP only. Level 1 of the assessment includes the evaluation of early mand, tact, listener, social, visual-perceptual and match-to-sample, independent play, motor imitation, and echoic skills, as well as spontaneous vocal behavior. Level 2 of the assessment includes continued evaluation of expanded level 1 skills (with the exception of spontaneous vocal behavior) as well as an evaluation of listener responding by function, feature, and class, intraverbals, classroom/group routines, and linguistic skills (see Sundberg 2008). Level 3 expands on the skills targeted in level 2 and assesses pre-academic behaviors in reading, math, and writing. The VB-MAPP is a tool that can be used in a variety of settings with any number of clinical populations (Sundberg 2008). Many educational settings use the instrument to establish language goals and objectives for individuals with autism spectrum disorder and other developmental disabilities.

As with all tools and protocols, the results of the VB-MAPP will only be meaningful if the assessment is conducted by professionals who are skilled in its administration. Unlike many other language assessments, the VB-MAPP requires that administrators are familiar with Skinner’s (1957) analysis of verbal behavior and basic behavior analysis (Sundberg 2008). That is, administrators must be aware of the functional relationships between verbal behavior and the speaker’s environment in order to correctly design the assessment environment and assess the verbal operants. Without an understanding of Skinner’s (1957) analysis of verbal behavior, an individual may mistake one operant for another based on formal similarity. Table 1 provides examples of verbal operants in which the response topography, saying “train,” is the same, but the type of operant is different as evidenced by the antecedents and consequences. Understanding the environment-behavior relations is also important for distinguishing between operants in which the responses are functionally similar, but topographically different. Table 2 shows an example of an intraverbal probe and a listener response by feature, function, and class probe that have similar antecedents and consequences, but are different topographically and are assumed to be functionally independent responses. These examples underscore the need to ensure that those implementing the VB-MAPP are adequately trained in the behavior-analytic skills necessary to conduct the assessment.

Table 1.

An example of verbal operants that are formally the same, but have different antecedent controlling variables and are maintained by different consequences

Verbal operant	Antecedent	Response	Consequence
Echoic	Another person says “Train”	“Train”	Nonspecific reinforcement/praise
Mand	Providing a track with no traina	“Train”	Access to a train
Tact	A picture scene with a train	“Train”	Nonspecific reinforcement/praise
Intraverbal	“What goes choo choo?”	“Train”	Nonspecific reinforcement/praise

^aThere must be a motivating operation in place for the train set (i.e.., the child must “want” to play with the train set)

Table 2.

An example of different operants that are topographically different, but have similar antecedent controlling variables and are maintained by similar consequences

Verbal operant	Antecedent	Response	Consequence
Listener responding by feature, function, class	“Where do you sleep?” with a picture scene (or other visual) present	Points to a bed	Nonspecific reinforcement/praise
Intraverbal	“Where do you sleep?” with no relevant visuals	“Bed”	Nonspecific reinforcement/praise

The VB-MAPP manual and guide (Sundberg 2008) provides some instruction on how to implement the milestones assessment; however, it is unclear if the instructions are sufficient for conducting the assessment successfully, or with fidelity. Previous research has shown that when a manual is specifically designed to teach a skill, it may be an effective method for establishing the repertoire. For example, Miltenberger and Fuqua (1985) evaluated the effectiveness of both an instructional manual and behavioral skills training on the acquisition of behavioral interviewing skills and found that both instructional methods were equally successful in teaching the target skills. According to these results, it is possible that the written instructions provided in the VB-MAPP guide and protocol (Sundberg 2008) are sufficient to teach professionals to administer the assessment. Alternatively, if written instructions are not sufficient in teaching professionals to administer the assessment, behavioral skills training (BST; Miltenberger 2008) may be an effective instructional method. BST consists of four components as follows: instructions, modeling, rehearsal, and feedback, and has repeatedly been demonstrated as an effective method for teaching a variety of clinical skills to paraprofessional staff and students (e.g., Rosales et al. 2009; Sarokoff and Sturmey 2004).

Iwata et al. (2000) demonstrated the efficacy of a modified BST procedure when teaching undergraduates to implement functional analyses. The BST procedure consisted of group instruction with written materials, video models, rehearsal with a confederate client, and performance-specific feedback. Following exposure to the instructional package, all participants demonstrated mastery performance of the attention, play, and demand functional analysis conditions. Similar results have been demonstrated when using BST to teach special education teachers the skills required for discrete-trial teaching (Sarokoff and Sturmey 2004). Behavioral skills training included instruction with written descriptions, modeling, rehearsal, and feedback. Prior to the intervention, the participants’ mean performance was below 50 % accuracy. Following BST, mean performance for all participants improved to above 95 %. Rosales et al. (2009) used BST to teach two undergraduate and one graduate student to implement the Picture Exchange Communication System (PECS). The BST package consisted of verbal and written instruction, a quiz, role rehearsal, video simulations, modeling, and corrective feedback. Following instruction, all three participants successfully implemented phases 1–3 of PECS with confederate learners. Furthermore, results were shown to generalize to at least one learner with developmental disabilities for all participants and maintain after 1 month for one participant. Together, the results of these studies suggest that BST is an effective instructional method for the acquisition of skills relevant to complex behavior technologies when written instruction alone is not sufficient.

Accurate administration of the VB-MAPP is important so that appropriate goals and objectives can be targeted for language instruction. In addition to consuming the written instruction that is provided in the VB-MAPP guide, individuals interested in administering the assessment can attend VB-MAPP workshops. However, to date, the efficacy of these strategies for teaching others to administer the assessment has not been explored. As the assessment grows in popularity, it is the responsibility of the professional community to ensure that those who will be administering the assessment are taught to run it with fidelity. The first step in ensuring that professionals are taught to conduct the assessment correctly is to evaluate the effects of current behavioral instructional procedures, such as BST, on participant performance. The current study examined the effects of behavioral skills training on implementation of levels 1 and 2 of the VB-MAPP milestones assessment.

Method

Participants, Setting, and Materials

Two school psychologists, Lucy and Ethel, served as participants. Lucy and Ethel had 22 and 17 years, respectively, of experience assessing children with disabilities using a variety of standardized assessments. Examples of the assessments that Ethel regularly conducted include the Wechsler Intelligence Scale for Children®—Fourth Edition (Wechsler 2003), Stanford-Binet Intelligence Scales—Fifth Edition (Roid 2003), Woodcock-Johnson® III (McGrew 2001), Gray Oral Reading Test—Fourth Edition (Wiederholt and Bryant 2001), Kaufman Test of Educational Achievement—Second Edition (Kaufman and Kaufman 2004), Gilliam Autism Rating Scales—Second Edition (Gilliam 2001), and the Adaptive Behavior Assessment System®—Second Edition (Harrison and Oakland 2003). Lucy regularly conducted the aforementioned assessments and also had experience administering the Autism Diagnostic Observation Schedule—Generic (ADOS; Lord et al. 2000). Neither participant had administered a VB-MAPP prior to participating in the study. Both participants had a basic understanding of behavior analysis and were completing the coursework necessary to sit for the Behavior Analysis Certification Board^© examination. A basic overview of the verbal operants was included in the coursework that the participants completed prior to their involvement in the study. This overview consisted of definitions and examples of each of the verbal operants in text (e.g., Cooper et al. 2007) as well as in-class lecture and discussion.

The setting for each session was determined by the preference of the caregiver of the child being assessed. Locations included a university office, the child’s home, the child’s school, a youth room in a church, and a conference room at the office where the participants worked. Commercially available materials included the VB-MAPP assessment guide and protocol (Sundberg 2008), the reinforcer assessment for individuals with serve disabilities (RAISD; Fisher et al. 1996), and various toys and assessment materials used during the assessment. Additional materials developed by the experimenter for the purposes of this study included a VB-MAPP checklist, pre-assessment interview, administration handbook, and instructional PowerPoint® presentation (available upon request from the first author). The pre-assessment interview was designed to be conducted with caregivers to determine at which level of the VB-MAPP it is most appropriate to begin a child’s assessment, and to determine the best type of assessment environment for that child. Questions designed to identify the initial assessment level were questions about the child’s use of verbal operants. Questions designed to determine the best type of assessment environment were questions about the conditions under which the child would most likely comply with instructions and interact with the assessment administrator. Examples of interview questions are shown in Table 3. The administration handbook consisted of a brief description of each verbal operant assessed in the VB-MAPP and lists of potential ways to probe those skills using materials commonly found in classrooms (e.g., books and inset puzzles). The instructional PowerPoint® presentation was developed based on the information in the administration handbook and was used as an aid during instruction. The VB-MAPP checklist was used to measure the participant’s responses during levels 1 and 2 assessments (see Appendix).

Table 3.

Sample questions from the pre-assessment interview

Category	Question
Environmenta	Will your child sit at a table to work? If so, how long will he or she generally work?
Environment	Are there any items, noises, or activities that your child dislikes that should be avoided during the assessment?
Levelb	Does your child label actions, like clapping, sleeping, eating, or jumping?
Level	Does your child use words to tell you what he wants? If so, what are some examples?
Level	Can your child sort pictures according to categories like putting all animals together and all clothes together?

^aQuestions labeled “environment” are from the portion of the interview used to determine the best type of assessment environment

^bQuestions labeled “level” are from the portion of the interview used to determine which level of the milestone assessment to begin with

Design, Dependent Measures, and Interobserver Agreement

A multiple-probe design across participants was used to evaluate the effects of training in implementation of levels 1 and 2 of the milestones assessment. Instruction was implemented with Lucy once the results showed stable responding for both participants during pretest probes. Once Lucy’s posttest results were visually judged to be stable, and Ethel’s pretest results were also judged to be stable and low, instruction was implemented with Ethel.

The dependent measure was the percentage of points earned on the implementation checklist for levels 1 and 2 of the VB-MAPP milestones assessment (see Appendix). The checklist for level 1 consisted of a total of 27 responses. A total of 29 responses were measured during level 2 assessments. The responses on each checklist were selected by a team of clinicians who observed VB-MAPP assessments and identified administrator behaviors that appeared to result in successful assessments. Successful assessments were those in which the child being assessed engaged in minimal challenging behavior and the assessor completed all of the probes necessary to evaluate the child’s skills.

For each response on the checklist, participants could earn 0, 1, or 2 points with a higher point value indicating better performance. Specific criteria for each point value varied for each response and were explicitly stated on the checklist. For all items rated using the categories “never,” “for a portion of the session,” and “throughout the session,” a “never” was scored if the participant was not observed to engage in the response at any point during the assessment. “For a portion of the session” was scored if the participant engaged in the response at any point during the first half of the session or the second half of the session. “Throughout the session” was scored if the participant engaged in the response during both halves of the session. If a response was not applicable, that item on the checklist was crossed out and eliminated from the calculation of the percentage of points earned. For example, in order for a participant to perform the response specified in item 12 on the checklist, the child who the participant was assessing would have had to avoid a task that was presented naturalistically. If the child never avoided any tasks presented in a naturalistic format during the assessment, the item could not be scored; therefore, the observer crossed that item out and did not count it toward the participant’s percentage of points earned.

Interobserver agreement (IOA) data were collected for 80 % of all pretest probes and 50 % of all posttest probes. Agreement was evaluated on a point by point basis and was calculated by dividing the number of agreements by the number of agreements plus disagreements and multiplying by 100. Mean pretest and posttest IOA for Lucy and Ethel was 82.5 (range 78–89.7 %) and 85.9 % (range 78–93.1 %), respectively.

Procedures

Pretest Probes

Before BST was implemented, participants were asked to read the VB-MAPP protocol and guide book in order to prepare to implement levels 1 and 2 of the milestones assessment. Participants also had access to toys and assessment materials purchased by the district for which they worked; no other materials were provided to the participants. Access to these materials was provided in an effort to simulate the sort of minimal training conditions to which professionals implementing new assessments are often exposed. Seven to 10 days after receiving the VB-MAPP protocol and guide, both participants began pretest probes. A level 1, test probe consisted of the participant implementing an entire level 1 VB-MAPP milestones assessment and a level 2 test probe consisted of the participant implementing an entire level 2 VB-MAPP milestones assessment with a child.

No formal child demographics (e.g., age, diagnosis, and placement) were collected as the focus was on the behavior and skills of the test administrators; however, anecdotally, we can report that the children who were assessed varied in skill level, diagnosis, and verbal ability. Participants assessed different children throughout the study and never assessed the same child more than twice. The only instructions provided to the participants during test probes were to complete the assessment and to notify the experimenter when they were finished. The experimenter observed the participant and recorded her responses on the VB-MAPP checklist. No feedback or error correction was provided.

Behavioral Skills Teaching

BST consisted of five components as follows: instruction, modeling, rehearsal and feedback, and remedial teaching as needed.

Instruction

Instructions were delivered in a small group format with 1–3 additional clinicians who were not serving as participants in the present study. During instruction, the first author gave a presentation using PowerPoint® slides (available from the first author upon request). The slides were shown on a large screen and each participant was given a printed copy of the slides and the administration handbook. The presentation began with general information about the VB-MAPP and its uses. Next, descriptions and examples were provided of each operant assessed in the milestones assessment. For example, a slide was presented on the mand that provided a definition of manding and examples of evoking mands by either contriving establishing operations or capitalizing on existing establishing operations. Similar slides were presented for each operant. The final portion of the PowerPoint® included general assessment techniques. This section included items such as interspersing probes for different operants throughout the assessment, providing a rich environment by attending to the child, providing access to preferred items, and taking breaks for play-based assessment or preferred activities; altering task presentation from table top to naturalistic presentation; and using materials for multiple sub-skill assessments. Each behavior on the VB-MAPP checklist was covered in the instructions portion of the intervention package. Participants were able to ask questions throughout the instructional session and were encouraged to do so. Instructions lasted between 90 and 120 min depending on the number of questions asked by participants and number of additional people attending the session.

Modeling

Video models of the experimenter and other experienced clinicians implementing the VB-MAPP were shown immediately after the instruction component of training was completed. The video models consisted of different clips from levels 1 and 2 milestones assessments modeling all of the behaviors on the VB-MAPP checklist used to assess the participants’ performance (see Appendix). For example, one video clip illustrated following the child’s lead, creating a rich environment, interspersing tasks, and maximizing the use of materials (e.g., using books to probe multiple operants). Text was embedded in the video clips to orient the viewer to the target behavior(s) being modeled. Participants were also shown an organized assessment kit, a completed pre-assessment interview, and a completed RAISD during the modeling portion of BST.

Rehearsal and Feedback

Immediately following the instruction and modeling components of the intervention, the rehearsal and feedback component began. Participants were given an opportunity to rehearse 3–5 target behaviors at a time in a series of short role plays with a confederate learner. Prior to each role play, the experimenter told the participant which target behaviors, from the VB-MAPP checklist, she was required to perform. Participants were also told that they should select their own materials from an array of available stimuli. For example, in one role, play participants were asked to demonstrate probes for mands, tacts, and listener responding as well as to determine if books capture the child’s attention (i.e., steps 8, 11, 23, 25, 30, and 31 on the checklist). After each role-play, the experimenter provided performance-specific feedback. Feedback included telling the participant which responses were correct and would earn full points on the checklist and which responses were not correct and would not earn full points on the checklist. The experimenter told the participant how to alter the incorrect response and had her practice it correctly with the confederate. Role plays were the same for both participants.

Posttest

Posttest probes were identical to pretest probes. The mastery criterion was set at 90 % across two consecutive posttest probes for both levels 1 and 2.

Remedial Teaching

If participants earned less than 90 % of the possible points on posttest probes, remedial teaching was implemented. Remedial teaching included providing additional performance specific feedback. The experimenter reviewed each step on the checklist that was not scored full points and provided a description of how they could have completed the step. During feedback, participants were also praised for 3–5 steps on the checklist that earned full points.

Results and Discussion

Results for Lucy and Ethel are presented in Fig. 1. Pretest probes for both participants indicated that they did not implement either level of the milestones assessment at the criterion of 90 % of points earned prior to BST. Lucy’s pretest performance was 64 and 51.9 % of points earned for levels 1 and 2, respectively. Ethel’s mean points earned on pretest probes for level 1 of the assessment was 58.4 % (range 40.4–70 %), and for level 2 her mean performance was 55.3 % (range 42.8–72 %). Following BST, Lucy’s performance increased for both levels 1 and 2. Her level 1 performance post-training was a mean of 94.5 (range 90–97.9 %) and 88.25 % (range 78.6–98.1 %) for levels 1 and 2, respectively. Ethel’s performance also improved following training to a mean of 91.6 (range 78.9–98 %) and 94.1 % (range 88–98.1 %).

Fig. 1.

The percentage of points earned for Lucy (top panel) and Ethel (bottom panel) for levels 1 and 2 of the VB-MAPP milestones assessment during pretest and posttest probes. Remedial training sessions were conducted following all posttest probes that the participants scored less than 90 % correct on

The results of the study indicate that BST may be an effective method for teaching school psychologists to implement levels 1 and 2 of the VB-MAPP milestones assessment. As shown in Fig. 1, performance for both participants increased immediately after the small group BST. That is, while additional feedback was needed for participants to reach the mastery criterion of 90 % or better, the initial training session did result in improved performance. These results are important when considering the possibility of implementing similar training methods in other districts and settings where time and resources for staff training are minimal. Future research should further examine the effects of BST with groups of increased size. It would also be wise to evaluate whether the assessment techniques learned for levels 1 and 2 (e.g., organizing materials, establishing one’s self as a reinforcer) generalize to the level 3 milestones assessment. The maintenance of such skills over time should also be evaluated.

As shown in Fig. 1, Ethel’s pretest score deteriorated over time. This decreasing trend may have been a function of time between the pretest probes and her exposure to the VB-MAPP guidebook. In a conversation with the experimenter, she indicated that she carefully read the guidebook prior to her initial pretest probes for each level; however, she did not re-read the material prior to subsequent pretest probes. It is unclear if Ethel’s performance would have improved upon further review of the guidebook. Additional research should focus on the effects of written material review prior to every assessment administration.

The initial pretest performance for both participants was relatively high with each earning at least 50 % of the possible points prior to instruction. Further analysis of the participants’ performance shows that both Ethel and Lucy were earning full points for the responses on the checklist that may be identified as “general assessment techniques.” For example, during baseline, both participants completed the RAISD, established rapport with the child they were assessing, followed the child’s lead during play based probes, provided access to preferred items/probed mands throughout the assessment, and provided breaks for play (or play-based assessment). These assessment skills may be similar to those used with the other tests that the participants regularly administered (e.g., ADOS). It was not until after BST that the participants consistently earned full points on checklist steps that are more specific to the VB-MAPP, such as probing each of the operants throughout the assessment, contriving motivating operations for mands, and managing assessment materials. These results suggest that the instructional material in the VB-MAPP guide and protocol (Sundberg 2008) should be supplemented in order for participants to reach the mastery criterion.

Future research should assess the efficacy of workshops and other popular instructional techniques that are currently being used to teach professionals to implement the VB-MAPP. In the current study, Lucy earned 90 % of the points available for level 1 of the milestones assessment after the initial BST session (e.g., instructions, video model, role-playing the assessment with a confederate). Her level 2 scores improved after the initial BST session, but she required three remedial teaching sessions before meeting the 90 % criterion. Ethel only required one remedial teaching session for levels 1 and 2 each before earning 90 % of the points possible for each level after the initial BST session. These results may suggest that if workshops include these components of BST, they could be effective in establishing some of the skills necessary to implement the levels 1 and 2 of the VB-MAPP milestones assessment.

There are several limitations that should be noted. First, it is unclear if all of the behaviors listed on the checklist are necessary for successful implementation of the VB-MAPP. Similarly, because BST and the remedial instruction were implemented as a treatment package, it is unclear if all of the components are necessary for the desired behavior change. Future research should examine the effects of BST with and without remedial instruction on administration of the VB-MAPP to determine which instructional components should be included in VB-MAPP workshops and training. The initial BST package (instructions, modeling, rehearsal with a confederate, and feedback) was completed in less than 3 h per participant. However, the remedial instruction was much more time consuming, as this component required ongoing observation and communication between the individual conducting the training and the individual learning to implement the assessment. It will also be important to evaluate whether the results of the VB-MAPP assessments can be used to develop instructional programs for the children being assessed to validate that the assessment is being administered in an effective way that generates useful learner results.

Participants in this study were preparing to become Board Certified Behavior Analysts. It is unclear how their coursework and field experience required to be eligible for certification may have affected their performance, as both participants were familiar with the terminology used in the VB-MAPP (e.g., mand, tact, intraverbal) and basic behavior change procedures (e.g., reinforcement, extinction). While the VB-MAPP guide suggests that individuals who conduct the assessment have a “basic understanding of behavior analysis, Skinner’s (1957) analysis of verbal behavior, and linguistic structure” (Sundberg 2008, p. 16), additional research should be conducted with participants who may be likely to use the assessment, but who have less experience behavior analysis and the verbal operants (e.g., social workers, special education teachers, early intervention providers). Results of such studies may provide a further understanding of the prerequisite skills necessary to implement such an assessment.

Finally, future research should employ a more rigorous experimental design. The current study used a multiple-probe design across two participants design. This design could be strengthened by adding additional participants, increasing the number of pretest probes for the first participant (Lucy in the current study), and adding follow-up probes. Despite the limitations to the experimental design used in the present study, the results are promising and convincing. Specifically, the immediate and robust effects shown for both participants during posttest probes provide evidence that the intervention was effective in establishing the skills necessary to complete levels 1 and 2 of the VB-MAPP milestones assessment.

Appendix