Using a Checklist to Increase Objective Session Note Writing: Preliminary Results

Abstract

We evaluated the extent to which a checklist increased objective note writing following simulated teaching sessions for 17 special education staff members. In general, participants improved in their description of the reinforcer earned by the child and of prompts delivered by the teacher during a session. Nevertheless, participants’ correct reporting of problem behavior decreased following the training.

With the rise in prevalence of individuals diagnosed with autism spectrum disorder (ASD; Baio et al., 2018), there has been an increased demand for trained professionals to deliver behavior-analytic services. To meet this need, the Behavior Analyst Certification Board introduced the credential of a frontline paraprofessional referred to as a Registered Behavior Technician (RBT; Carr & Nosik, 2017). Although the corresponding task list indicates RBTs should “generate objective session notes by describing what occurred during sessions,” this task item lacks operational definitions (Behavior Analyst Certification Board, 2016, p. 3). Toward this end, Carr, Nosik, and DeLeon (2017) encouraged additional research regarding the RBT task list, which may include creating training tools.

One cost-effective strategy that may increase skill performance is to provide a checklist. Other fields (e.g., behavior-based safety) have documented improved performance outcomes by using checklists (e.g., Cooper, 2009). Checklists require minimal resources and may be optimal when developing training tools for the RBT task list. The current study examined the extent to which a checklist increased session note-writing skills for school personnel across five dimensions. Given the implications of how school change agents may communicate to stakeholders with session notes, research on efficient methods for training objective note writing is needed.

Method

Participants and Materials

We conducted this study as part of a school district’s RBT training. Two special education teachers and 15 paraprofessionals who work with students diagnosed with ASD participated. Teachers had master’s degrees, and the paraprofessionals, at a minimum, had an associate’s degree and experience working with children in educational environments. Two paraprofessionals were recently hired. The remaining participants had at least 1 year of experience working in special education in the district.

The first author obtained informed consent from participants prior to the start of the study. Materials included (a) note cards, (b) the session note checklist (Appendix), and (c) a writing utensil.

Research Assistant Training

Prior to the study, the first author trained two research assistants (RAs) to conduct role-play scenarios. The first author assigned one RA to serve as the child and the other to serve as the teacher. During the training, the first author modeled the procedure the child RA and teacher RA would be simulating for the participants. She instructed the RAs to conduct a brief preference assessment (one trial of presenting two stimuli equidistant from each other within eye gaze) prior to conducting the teaching session. The first author also trained the RAs to conduct an auditory-visual discrimination task containing an array of three two-dimensional targets for which the teacher RA used a three-step prompting procedure (vocal directive, gesture, and physical guidance) to evoke a correct response by the child RA. In addition, she trained the child RA to engage in problem behavior (e.g., self-injurious behavior) for 5 of 10 trials during the simulated sessions.

Session Note Checklist

The lead author created a checklist, shown in the Appendix, as a way to operationalize components that school personnel may need when writing a note to parents for a child in special education. She created the checklist based on feedback from a special education coordinator, teachers, and two special education lawyers.1 During these conversations, each party agreed that session notes should not contain opinions. We coded Steps 1–4 in the Appendix for the purposes of this study; Steps 5–9 were included for purposes beyond those outlined in this study. For example, Item 7 could be used to document either illness or side effects of psychotropic medication.

Design

We used a pretest and posttest design to evaluate the effects of a session note checklist on participants’ writing of session notes after they viewed a 2-min simulated teaching session between a teacher RA and a child RA. Each participant watched three simulated sessions before the first author provided the checklist. The first author averaged scores from these three sessions for each of the dimensions (reinforcer, subjectivity, prompt, problem behavior, and task) and repeated this process after she provided the checklist (Appendix). We compared participants’ pre- and posttraining performance using an exact sign test to determine if the checklist improved objective note writing.

Procedures

Prechecklist

The first author informed the group of participants they would be writing notes following simulated scenarios between two RAs, one teacher and one child. The lead author introduced the RAs to the participants and then passed out note cards and writing utensils. Thereafter, the lead author provided the following instructions:

Right now, you are going to watch RA 1 and RA 2 pretending to be a child and adult interacting in a special education setting. Please observe the interaction carefully. After it is complete, you will be given 2 minutes to write a note describing their interaction. Pretend that this note will be given to the child’s parents.

The lead author then instructed the RAs to engage in the role-play scenario, as described previously. Once the scenario was completed, the lead author provided 2 min for participants to write their notes. After 1 min elapsed, the lead author informed the participants they had 1 min remaining. When the time elapsed, she gathered the participants’ note cards and redistributed new note cards for the subsequent role-play simulation. This process was conducted three times. The lead author did not provide feedback or answer questions. The total time required for this process was approximately 30 min.

Checklist delivery

The lead author distributed the session note checklist to the participants. She oriented the participants to the checklist, reviewed each of the components (Steps 1-4) that should be included on these session notes, and then reviewed the components (Steps 5-9) that may be included on notes sent home when working in a school setting. The lead author instructed the participants to review the checklist independently for approximately 10 min.

Postchecklist

The lead author conducted postchecklist sessions in the exact same manner as prechecklist sessions. All participants viewed three simulated interactions between the RAs with the same number of teaching trials and instances of problem behavior seen pre-checklist delivery. The lead author did not answer questions or provide feedback.

Session Note Coding

For each of the participant’s session notes, trained RAs coded each of the five dimensions as being present (1) or not present (0). For the subjectivity dimension, coders recorded the frequency of instances in which the participant’s response met our definition.

Reinforcer

The purpose of this dimension was to ensure the note indicated the teacher RA delivered preferred items contingent on appropriate behavior displayed by the child RA. To be scored a 1 on this dimension, the participant’s note would have indicated the specific stimulus used as a putative reinforcer during the role-play scenarios. If a note indicated the child RA earned or was working for an edible, tangible, or praise, coders scored this as a 1. If the note indicated the child RA “selected treats” or “treats were available” but did not specify what the child was earning or working for, coders scored this as a 0. If the note did not indicate there was a stimulus for which the child was working, coders scored this as a 0.

Subjectivity

The purpose of this dimension was to ensure the note did not contain opinions from the participant. If a participant’s note was written such that problem behavior was an adverb (e.g., “aggressively hit the teacher”), it was scored as an occurrence of subjectivity. In addition, we coded descriptions of internal states such as “in a bad mood” and “not a happy camper” as subjective. If a note included adjectives to describe the overall session without specific referents to behavior, we scored this as subjective. For example, if the note described the scenario as a “bad teaching session,” the coder scored this as an instance of subjectivity. In addition, coders scored instances in which the notes described personality traits or characteristics of the student RA as subjective (e.g., writing “she needs more patience”).

Prompting

The purpose of this dimension was to ensure the note described the level of guidance the teacher RA provided to produce a correct response by the child RA. To be scored a 1 on this dimension, the note needed to indicate the type(s) of prompting used by the teacher to evoke a correct response from the child. If the note indicated the child either engaged in an independent correct response or needed “a gesture” or “guidance” to produce a correct response, coders scored this as a 1. In addition, notes that included words such as “hand-over-hand guidance” and “physical” prompting were scored as a 1. Notes with broad statements, such as “needed prompting,” “required prompts,” and “needed assistance,” without reference to correct responding were scored as a 0.

Problem behavior

The purpose of this dimension was to ensure the note stated the topographies of disruptive behavior displayed by the child RA. To be scored a 1 on this dimension, the note needed to describe the form of behavior displayed by the child RA during the role-play simulation. If the note contained the words “self-injurious behavior,” “hand bite,” or “head hit,” this was scored as a 1. Notes that included phrases like “acting out” or “engaged in behavior” received scores of 0. If the note did not mention problem behavior, this was scored as a 0.

Task

The purpose of this dimension was to ensure the note indicated the acquisition targets in the teaching session. To be scored a 1 on this dimension, the note would have indicated the specific instructional directives conducted during the role-play scenarios. If the note indicated the child RA had participated in a task involving “identifying” or “touching shapes, colors, animals, or numbers,” this was scored as a 1. If the note indicated the child RA was “working” or generically referred to “object identification,” “flash cards,” “listener responding,” or “receptive identification,” the response was coded as a 0.

Intercoder Agreement

We assessed intercoder agreement on an event-by-event basis by having trained RAs independently recode 100% of the participants’ notes. Intercoder agreement scores (both observers’ total percentage agreement summed and averaged for each participant’s note) averaged 95% (range 89%–100%).

Results and Discussion

Table 1 describes each participant’s scores as improved (gray, improved in postchecklist), ceiling (gray, highest score in both pre- and postchecklist), no change (white, score the same in pre- and postchecklist), and worsened (black, worsened in postchecklist). Across participants, we also calculated the overall averages for those who improved, did not change, or worsened for each dimension. We removed “ceiling” participants from the calculations of averages. These participants scored perfect scores across three notes during both pre- and postchecklist.

Table 1.

Effects of Session Notes Across Dimensions and Participants

Note. “Celing” indicates participants with perfect scores during pre- and post-checklist delivery. We did not include these participants in the calculation of mean percentages of participants who improved, did not change, and worsened. Significant at the p < .05 level

We also conducted an exact sign test to compare the differences in session note writing before and after checklist delivery. Results of the sign test (bottom of Table 1) indicate the checklist produced significant improvements in participants’ note writing for (a) indicating the reinforcers used by the teacher, (b) reducing subjectivity, and (c) indicating the types of prompts used by the teacher. Results also indicate no significant change in either reporting of problem behavior or reporting of the task completed.

For the reinforcer and subjective dimensions, the checklist led to consistent improvements with minimal worsening across participants. That is, the checklist did not have a deleterious effect for those with high scores. By contrast, for the problem behavior dimension, six participants who had perfect scores during prechecklist displayed worse scores following the checklist. It is unclear why the session note checklist used in this study was more effective in increasing participants’ reporting in certain dimensions and not others. One explanation for decreased subjectivity reporting may be the italic examples (e.g., “in a great mood”) on the checklist. However, this does not explain why correct reporting of the reinforcer and prompt increased post-checklist delivery, which were dimensions in which the checklist did not have italicized text.

Interestingly, the session note checklist does not include a component describing problem behavior, yet most participants correctly described problem behavior pre-checklist delivery but failed to do so post-checklist delivery. It is possible the checklist exerted very tight stimulus control such that participants may have only written about the dimensions enumerated in the checklist. Alternatively, it is possible that tracking multiple dimensions of the teacher-child instructional interaction precluded participants from detecting the child’s problem behavior. Future studies will need to address this question. Also, most participants did not improve in the task dimension, which may have been due to our coding criteria of indicating the specificity of the directives. Future research should evaluate the necessary components when providing visual prompts to staff to increase objective session note reporting. For example, providing a “fill-in” template may increase objectivity of session notes.

Some limitations in this study warrant discussion. The one-group, pre- and posttest design is not strictly experimental in nature due to the lack of a control group. Although internal validity threat such as history and maturation are unlikely (this study took place over 2 days), improvement in participants’ performance from prechecklist to postchecklist could be a practice effect rather than a treatment effect. Future research should include a control group when conducting further evaluations of this checklist.

Due to our prioritization of other instructional skills within the RBT training, we did not use evidence-based training to increase the participants’ reporting. Future researchers should continue to investigate how to train groups of change agents to implement evidence-based procedures. For example, brief group feedback (e.g., Luna, Petri, Palmier, & Rapp, 2018) could have been a relatively easy way to enhance performance with minimal personnel and time resources.

Another limitation is that we did not determine the extent to which this instructional method would transfer to settings when participants are actually conducting lengthy training sessions (e.g., 30 min) with learners. Likewise, we did not assess whether participants’ exposure to the session note checklist led to more objective notes when reporting to stakeholders in real-world settings (e.g., writing a note to parents for a special education student or for review by an insurance representative). The effectiveness of the session note checklist in increasing objective writing may be influenced by multiple variables (e.g., interaction duration, setting, intensity and frequency of problem behavior). Future research could modify the current session note checklist and validate notes created with or without the checklist with stakeholders to identify relevant components.

Implications for Practice

1. Practitioners should operationally define RBT task list items prior to training.

2. Practitioners should instruct staff to include relevant information in session notes.

3. Practitioners should develop checklists to guide staff members’ written reporting.

4. Practitioners should consider the implications of staff using subjectivity when writing notes to stakeholders.

Appendix

Session Note Checklist
Components	Completed?
1. Indicate the task was completed during the session.
2. Indicate the reward earned/earning during the session.
3. Indicate the level of prompting needed for the child to complete the task.
4. Avoid using subjective terms (e.g., in a great mood, bad mood, not feeling it).
5. Pair positive statements with outlining the behavior (e.g., They did amazing sorting!).
6. Indicate next steps: Will you continue teaching the same skill? New skills?	Steps 6-9 may be needed when writing a note for parents when working in a school setting.
7. Indicate if there are other concerns (e.g., the child said he was sleepy, did not eat today, possible sickness).
8. If severe behavior occurs, touch base with the teacher, supervisor, or administrator (teachers only) prior to writing the note.
9. Ensure the note is sent home as soon as possible.

Compliance with Ethical Standards

This manuscript is not under review nor has it been published elsewhere. This submission has been approved by all authors and by the responsible authorities where the work was carried out.

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

The participants provided informed consent before the authors initiated the study.