Systematic instruction for individuals with acquired brain injury: Results of a randomized controlled trial

Abstract

The goal of this study was to experimentally evaluate systematic instruction compared with trial-and-error learning (conventional instruction) applied to assistive technology for cognition (ATC), in a double blind, pretest-posttest, randomized controlled trial. Twenty-nine persons with moderate-severe cognitive impairments due to acquired brain injury (15 in systematic instruction group; 14 in conventional instruction) completed the study. Both groups received 12, 45-minute individual training sessions targeting selected skills on the Palm Tungsten E2 personal digital assistant (PDA). A criterion-based assessment of PDA skills was used to evaluate accuracy, fluency/efficiency, maintenance, and generalization of skills. There were no significant differences between groups at immediate posttest with regard to accuracy and fluency. However, significant differences emerged at 30-day follow-up in favor of systematic instruction. Furthermore, systematic instruction participants performed significantly better at immediate posttest generalizing trained PDA skills when interacting with people other than the instructor. These results demonstrate that systematic instruction applied to ATC results in better skill maintenance and generalization than trial-and-error learning for individuals with moderate-severe cognitive impairments due to acquired brain injury. Implications, study limitations, and directions for future research are discussed.

Introduction

Overview

Cognitive impairment due to acquired brain injury (ABI) is strongly linked to reduced independence and community integration (Ben-Yishay & Daniels-Zide 2000; Coelho, DeRuyter, & Stein, 1996; Kreutzer et al., 2003; Sohlberg & Mateer, 2001). The acquisition and retention of new information and skills that help remediate or compensate for those impairments are therefore critical to rebuilding one’s life and facilitating greater independence (Sohlberg & Mateer, 2001; Sohlberg & Turkstra, 2011). In particular, skilled use of external aides, including assistive technology for cognition (ATC) (e.g., personal digital assistants, smart phones) offers an excellent means of compensating for cognitive impairments and improving independence (de Joode, van Heugten, Verhey, & van Boxtel, 2010; LoPresti, Mihailidis, & Kirsch, 2004; Sohlberg et al., 2007). However, three inter-related challenges can make ATC use unsuccessful: (a) Lack of instruction: Individuals who are good candidates for ATC may not receive any instruction on how to use these systems (Evans, Wilson, Needham, & Brenthall, 2003; Hart, Buchhofer, & Vaccaro, 2003; O’Neil-Pirozzi, Kendrick, Goldstein, & Glenn, 2004; Wehmeyer, 1999); (b) Lack of effective instruction: Instruction received may not be adequate to ensure skill acquisition (Evans et al., 2003; Wilson, Baddeley, Evans, & Shiel, 1994); and (c) Lack of instruction that targets maintenance and generalization: Instruction is not adequate to ensure retention and generalization of ATC skills in the natural environment (Cicerone et al., 2011; LoPresti et al., 2004; Ylvisaker, Hanks & Johnson-Green, 2003).

To illustrate, a 43-year old male with moderate-severe memory impairments due to ABI is prescribed a personal digital assistant (PDA) for keeping track of important daily routines and appointments. During his outpatient rehabilitation therapy sessions, his clinician provides him with a basic orientation to the device and some practice in clinic. However, the patient/client resorts to trial-and-error (i.e., guessing) when attempting to use the device at home and in the community. The lack of skill mastery, maintenance and generalization to other environments eventually leads to device abandonment. Systematic instruction applied to ATC offers a comprehensive approach to addressing these concerns (Ehlhardt et al., 2008; Ehlhardt, Sohlberg, Glang, & Albin, 2005).

Systematic Instruction

Systematic instruction is an evidence-based package used to teach facts, concepts, strategies, and skills (Ehlhardt et al., 2008; Sohlberg & Turkstra, 2011; Stein, Carnine, & Dixon, 1998). A major component of systematic instruction is direct instruction, an extensively researched instructional methodology from the field of special education that emphasizes mastery of new skills and information through careful design and delivery of instruction (e.g., Kameenui & Simmons, 1990; Madigan, Hall, & Glang, 1997; Stein et al., 1998). Design of instruction components includes: (a) detailed assessment of learner’s needs and abilities, including determining the environments in which the instructional targets will be used; (b) content or task analysis—breaking instructional targets down into component parts; and (c) careful example selection and sequencing.

Delivery of instruction components includes: (a) on-going assessment of the learner’s performance to gauge mastery; (b) modeling of the skill by the instructor prior to learner practice; (c) carefully faded support; (d) pre-instruction of components skills; (e) frequent, correct, distributed practice and review; (f) immediate, corrective feedback; (g) individualized instructional pacing to facilitate engagement; and (h) strategy instruction (Englemann & Carnine, 1991; Glang et al., 2008; Horner & Albin, 1988; Stein et al., 1998). Several meta-analyses have underscored the benefits of direct instruction for learners with a variety of learning challenges or disabilities (Swanson & Hoskyn, 1998; 1999; 2001).

Systematic instruction includes an emphasis on the use of “error control” techniques (i.e., errorless learning, method of vanishing cues) intended to prevent or minimize errors during the acquisition stage of learning, (e.g., Sohlberg & Turkstra, 2011; Wilson et al., 1994). These techniques have been extensively researched in the field of neuropsychological rehabilitation and successfully used to teach individuals with moderate-severe memory impairments due to ABI a variety of new skills and information, including the use of external aids such as ATC (e.g., Bourgeois, Lenius, Turkstra, & Camp, 2007; Campbell, Wilson, McCann, Kernahan, & Rogers, 2007; Ehlhardt et al., 2005; Glisky & Schacter, 1989; Hunkin, Squires, Aldrich, & Parkin, 1998; Kalla, Downes, & van den Broeck, 2001; Lloyd, Riley, & Powell, 2009; Squires, Hunkin, & Parkin, 1997; Tailby & Halsam, 2003; Todd & Barrow, 2008; Wilson et al., 1994). The benefits of error control techniques are often reported among individuals with relatively spared procedural/implicit memory (memory without recalling the experience of learning) and severe declarative/explicit memory loss (memory involving conscious recollection) (e.g., Sohlberg, Ehlhardt, & Kennedy, 2005; Wilson et al., 1994). However, these techniques have not been shown to be consistently better than trial-and-error, particularly among individuals with dementia or mild cognitive impairment (MCI) (e.g., Bier et al., 2008; Dunn & Clare, 2007; Evans et al., 2000; Jean et al., 2010). Further, the previously described memory systems associated with improved performance are a subject of on-going debate and research (e.g., Evans et al., 2003; Greenberg & Verfaellie, 2010; Page, Wilson, Shiel, Carter, & Norris, 2006; Tailby & Haslam, 2003).

Whereas error control techniques are particularly influential during the acquisition stage of learning, other systematic instruction techniques help promote maintenance and generalization of instructional targets. Maintenance is facilitated through the distribution of practice trials. Research across many different populations (including individuals with and without cognitive disabilities) and types of instructional targets (e.g., facts, multi-step skills) has shown that retention improves when practice trials are distributed rather than massed (e.g., Challis, 1993; Haslam, Hodder, & Yates, 2011; Melton & Bourgeois, 2005: Swanson, 2001; Todd & Barrow, 2008). Generalization is enhanced through the careful selection and sequencing of teaching examples and by training in the environments in which the skills will be used (Albin & Horner, 1988; Cohen, Ylvisaker, Hamilton, Kemp, & Claiman, 2010; Horner, Albin, & Ralph, 1986; Horner, McDonnell, & Bellamy, 1986; Horner, Williams, & Steveley, 1987; Svoboda & Richards, 2009).

Conventional Instruction

Conventional instruction (i.e., trial-and-error learning, errorful learning) differs from systematic instruction in several ways. (Note: The term conventional instruction will be used synonymously with trial-and-error learning in this manuscript.) Rather than constraining the number of instructional targets, conventional instruction emphasizes learner-initiated exploration of instructional targets; the learner attempts the target(s) (i.e., guesses) before an instructor model is provided (e.g., Evans et al., 2000; Wilson et al., 1994). Errors are considered an opportunity to problem solve and discover solutions and therefore are not always immediately corrected (e.g., Heal, Hanley, & Layer, 2009; Klahr & Nigam, 2004).

Conventional instruction and related techniques (e.g., rapid fading of cues) are thought to promote more active engagement and effortful recall that, although potentially increasing the number of errors, can lead to more durable encoding of instructional targets, particularly for individuals with relatively mild memory impairments (e.g., Riley & Heaton, 2000; Riley, Sotiriou, & Jaspal, 2004). However, the risk of encoding errors during the acquisition stage of learning is problematic for individuals with more severe memory impairments, and therefore systematic instruction emphasizing error control is preferred (e.g., Baddeley & Wilson, 1994; Evans et al., 2000; Wilson et al., 1994).

Systematic Instruction Applied to ATC

Reviews of the literature indicate that instruction in the use of external aids and, in particular, ATC is rarely or inconsistently reported in the research (deJoode et al., 2010; Sohlberg et al., 2007). This trend is mirrored in clinical practice. Comprehensive design and delivery of systematic instruction are not routinely used in rehabilitation settings (Lemoncello & Sohlberg, 2005). Specific to ATC, O’Neil-Pirozzi, Kendrick, Goldstein, and Glenn (2004) found that the majority of the clinicians they surveyed do not routinely incorporate the assessment and instruction of ATC into their clinical practice, citing as possible constraints both clinicians’ lack of proficiency with devices and clinician’s lack of confidence in instructional methods. Furthermore, in their review of the literature, Evans Wilson, Needham, and Brenthall (2003) cite the need for teaching the “effective use” of these aids as an important part of rehabilitation interventions. Experimental research evaluating instructional practices applied to ATC is therefore a critical component in addressing these concerns.

In a previous study, we experimentally evaluated systematic instruction using the previously described design and delivery components to teach multi-step skills using one form of ATC—adapted email (Ehlhardt et al., 2005). In a single case, multiple baseline design (Kazdin, 2011), five adults with severely impaired new learning and executive functions deficits due to TBI were taught an adapted email program. Four of five participants reached the criterion for mastery (100% correct on all steps for three consecutive sessions) within 7–15 training sessions; the fifth participant dropped out of the study. Three participants retained the procedure after a 30-day break, and all participants generalized their skills to a slightly altered email interface. One limitation of that study was that fluency or efficiency (i.e., time to complete tasks) was not included as a dependent variable. In addition to performance accuracy, fluency is critical to ensuring skill mastery (e.g., Glang, Singer, Cooley, & Tish, 1992; Glisky, 1995; Glisky & Schacter, 1987). These results and recommendations provide the foundation for the current study comparing systematic and conventional instruction in a larger group study that includes measures of accuracy and fluency applied to ATC.

Research Questions

The present study compared systematic instruction with conventional instruction targeting selected functions on a personal digital assistant (Palm Tungsten E2 PDA) among 29 individuals with moderate-severe cognitive impairments due to ABI. A between-subjects, double blind randomized controlled trial was used to experimentally evaluate outcomes. The independent variable was the type of instruction (systematic vs. conventional); the dependent variables were performance accuracy and fluency applied to the use of selected PDA programs. Our specific research questions were:

1. Posttest accuracy. Does systematic instruction result in more accurate task performance at immediate posttest than conventional instruction as measured by the number of correctly completed tasks across selected PDA programs?

2. Maintenance accuracy. Does systematic instruction result in more accurate task performance at 30-days follow-up?

3. Fluency. Does systematic instruction result in more efficient or fluent task performance than conventional instruction as measured by the time needed to complete correct tasks at posttest and 30-days follow-up across selected PDA programs?

4. Generalization. Does systematic instruction result in better generalization than conventional instruction at posttest and 30-days follow-up as measured by the number of correct tasks completed across environments and untrained PDA programs?

5. Social validity. Do participants prefer systematic instruction to conventional instruction as measured by a survey administered immediately following instruction? (Non-experimental)

We hypothesized that systematic instruction would produce significantly better outcomes at posttest and 30-day follow-up on all aspects of performance addressed by questions 1–4 (accuracy, maintenance, fluency, and generalization). Furthermore, we anticipated that the participants in the systematic instruction group would report higher satisfaction rates than the conventional instruction group (question 5).

Methods

Participants

Participants were recruited through local and statewide community-based rehabilitation and day treatment programs, residential facilities, community work transition programs, and associations serving individuals with ABI. Eligibility was determined by the Principal Investigator, using structured interviews with the participants, their family and/or caregivers, and selected screening measures. Specific inclusion/exclusion criteria included: (a) age 18 years and older; (b) documented ABI; at least one-year post injury and medically stable; (c) impaired ability to perform daily activities and participate in social/vocational pursuits due to cognitive impairments as determined by medical records and caregiver report; (d) a rating of at least 1 on the Disability Rating Scale (Rappaport, Hall, Hopkins, Belleza, & Cope, 1982); (e) impaired new learning as measured by a z score of at least −1 in two of the following three categories—long delay free recall, cued recall, intrusions—on the California Verbal Learning Test II-Short Form (Delis, Kramer, Kaplan, & Ober, 2000); (f) no significant problems with sustained visual tracking as measured by performance on Trails A (Reitan & Wolfson, 1995); (g) no other significant neurological impairments; (h) no evidence of severe aphasic disturbances; (i) no problems with vigilance, motor abilities, visual acuity/perception, or hearing that would interfere with the ability to use the PDA; (j) reading comprehension at the simple paragraph level; (k) no recent changes in medication; (l) no history of alcoholism or drug abuse, nor significant psychiatric disturbance within 6 months of the study; (m) no prior experience with PDAs; or past attempts to learn to use a PDA were reported but the device was abandoned, per caregiver and participant report; and (n) the use of simple cell phones for sending–receiving calls was acceptable but not a study requirement; however, candidates who used these for more complex functions such as texting were not included in the study.

A total of 32 out of 45 individuals screened for the study met the above criteria and were consented according to Institutional Review Board standards; 29 of these participants completed the study. Two participants from the conventional instruction group dropped out of the study, one for medical reasons and the other due to anxiety/discomfort using the PDA after four sessions. A third participant—from the systematic instruction group—dropped out of the study due to relocation to another state. Of the 29 participants who completed the study, three were unavailable to complete follow up (30-day) testing due to medical conditions unrelated to their injury. The participants lived in three northwest communities, including two metropolitan areas and one mixed urban-rural community. Twelve participants were seen in an outpatient clinical setting; 13 were seen in their respective supported residential communities, and four participants were seen in a day treatment program. All testing and training sessions were conducted in a quiet room in these settings. The systematic instruction participants also received training outside of the room and in the immediate environment of their particular setting.

Commonly used assessment tools with well-established psychometric properties were administered for the purposes of broadly describing and comparing the study groups across the domains of disability severity, memory and new learning, attention/executive functioning, and estimated pre-injury IQ.

Disability severity

The Disability Rating Scale (DRS) is a widely used measure of disability severity combining information across several domains of functioning including eye opening, orientation, cognitive ability for self-care, dependence on others, and psychosocial adaptability. The DRS was originally designed for tracking progress from the acute to post-acute stage of functioning (Rappaport et al., 1982). Total scores range from 0–29 with a rating of 1 indicating minimal disability, 4–6 moderate disability, 7–11 severe disability, and so on. The DRS was selected as the primary measure for determining severity for this study for two reasons: (1) Very few participants had available Glasgow Coma Scale scores or information concerning duration of coma and post-traumatic amnesia from which to determine severity of injury (Lezak, 1995); (2) Given that the majority of the participants were several years post-injury, a more current measure of global functioning was deemed appropriate.

Memory and new learning

California Verbal Learning Test II-Short Form was used both as a screening tool for the study and a descriptive measure of verbal memory and new learning using word list recall (Delis et al., 2000). The Logical Memory subtest of the Wechsler Memory Scale III (WMS-III; Wechsler, 1997) measures verbal memory and new learning using story recall. The Visual Reproduction subtest of the WMS-III measures visual memory and new learning using figure drawing.

Attention and executive functioning

Trail Making Tests A and B were used to evaluate focused attention and attentional switching, a component of executive functions (Reitan & Wolfson, 1995). The Controlled Word Association Test (COWAT) was used as a measure of verbal fluency (Spreen & Benton, 1977), and the Clock Drawing Test from the Cognitive Linguistic Quick Test was used as a global measure of executive functioning (Helm-Estabrooks, 2001).

Estimated pre-injury IQ

Wechsler Test of Adult Reading (WTAR) (Wechsler, 2001) was used to estimate the level of intellectual functioning before the onset of injury.

Overall, there were no significant differences between the groups on any of the above measures. Mean total group performance across the measures revealed moderate disability on the DRS with moderate-severe impairments across the domains of memory/new learning, attention/executive functioning and pre-injury IQ in the normal range.

Demographics

Groups were compared across several demographic variables, including gender, chronological age, education, years since injury, income, and living environment. Independent samples t-tests and chi-square tests of significance revealed no significant differences between the groups (see Tables 1 and 2 for participant demographics and neuropsychological test information).

Table 1.

Demographics

Characteristics	Systematic Instruction Group (n = 15)	Conventional Instruction Group (n = 14)	Total Group (n = 29)	P*
Gender (%)#				1.00
Male	60.00	57.14	58.62
Female	40.00	42.86	41.38
Age~				0.80
Mean	42.93	41.64	42.31
Standard deviation	14.02	13.26	13.43
Range	20–68	22–62	20–68
Race/Ethnicity (%)#				1.00
White	86.67	92.86	89.66
Non-white	13.37	7.14	10.34
Type of ABI (%)#				1.00
TBI	80.00	78.57	79.31
Other	20.00	21.43	20.69
Years since ABI~				0.94
Mean	13.73	13.43	13.59
Standard deviation	11.62	10.87	11.06
Range	1–35	3–48	1–48
Living arrangement (%)^				0.96
Long-term care	20.00	21.43	20.69
Supported living community	46.67	50.00	48.28
Private in-home	33.37	28.57	31.03
Marital status (%)^				0.71
Single	46.70	50.00	48.30
Married	6.70	14.30	10.30
Divorced	40.00	35.70	37.90
Deceased Spouse	6.70	0.00	3.40
Education level (%)^				0.40
< High school/partial high school	24.67	7.14	17.24
High school &/or diploma	26.67	28.57	27.59
Some college	40.00	57.40	48.28
≥ Bachelor’s degree	6.67	7.14	6.90
Employment status (%)^				0.23
Unemployed	93.37	71.49	82.76
Volunteer	0.00	14.29	6.90
Part-time	6.67	14.29	10.34
Income level (%)#				0.22
< $15,000	100.00	85.71	93.10
$15,000–$30,000	0.00	14.29	6.90

Note:

^*P ≤ .05 for significance,

^~Independent samples t test,

^#Fisher’s Exact test,

^{^}Pearson’s Chi-square test.

Table 2.

Neuropsychological tests

Test	Systematic Instruction Group (n = 15)	Conventional Instruction Group (n = 14)	Total Group (n = 29)	p
	Mean ±SD	Mean ±SD	Mean ±SD
DRS Total~				.81
Total	82.00	80.00	162.00
Mean	5.47±3.22	5.71±2.26	5.59±2.75
Range	2.0–13.5	1.5–9.5	1.5–13.5
CVLT II–SF
LDFR	−2.63±0.93	−2.39±0.81	−2.52±0.87	.47
LDCR	−2.37±0.92	−2.54±0.93	−2.45±0.91	.63
Total Intrusion Score	2.87±2.17	2.04±1.94	2.47±2.07	.29
WMS–III Logical Memory
Logical Memory I Recall	4.87±2.26	3.93±2.40	4.41±2.34	.66
Logical Memory II Recall	3.60±2.26	4.79±2.58	4.17±2.45	.18
Logical Memory % Retention	38.00±23.94	56.86±26.34	47.10±26.47	.04
WMS–III Visual Reproduction*
Visual Reproduction I	5.07±3.22	5.43±3.37	5.25±3.24	.78
Visual Reproduction II	5.07±2.50	6.57±3.61	5.82±3.14	.21
Visual Reproduction% Retention	29.44±28.81	43.78±35.55	36.61±32.58	.20
Trail Making*
Trails Part A	32.73±11.20	29.33±13.10*	31.22±11.96*	.48
Trails Part B	31.50±8.16	31.36±12.96*	31.44±10.30*	.98
COWAT	24.80±10.60	22.00±10.79	23.45±10.60	.49
Clock Drawing				.55
WNL	28.6%	26.7%	27.6%
Mild	28.6%	46.7%	37.9%
Moderate	28.6%	20.0%	24.1%
Severe	14.3%	6.7%	10.3%
WTAR	93.27±16.76	91.43±17.41	92.38±16.81	.78

Note: Boldface denotes statistical significance.

^~P value calculated using independent samples t-test.

^{^}P value calculated using chi-square test.

^*Missing values.

CVLT II–SF (z scores: M=0; SD = ± 1); WMS III (Scaled Scores: M=10; SD=3); Trails A & B (T Scores: M=50; SD=10); COWAT (raw scores); Clock Drawing (% of participants that scored within each specific range of severity); WTAR (Standard Scores: M=100; SD=15).

Materials

The Palm Tungsten E2 personal digital assistant (PDA) was selected for the study based on its commercial availability and relative ease of learning and use. (Note: Since the beginning of the study, Palm has stopped manufacturing the Tungsten E2; however, the focus of this study was on the type of instruction, not the technology itself; thus the findings are relevant to the instruction of all multistep technologies.)

Design

A between-groups, pretest-posttest, randomized control-group design (Gall, Gall, & Borg, 2007) was used to compare performance across the two types of instruction. Prior to the first meeting with the first participant, a staff member not directly involved with the study generated a tracking sheet composed of six blocks of six slots each—three per treatment condition (36 total slots). The six treatment slots within each block were randomized using a random sequence generator. Participant names were entered into the tracking sheet once they had passed the screening for the study. Allocation to study condition was concealed until each participant had completed pretesting. Participants were entered into the study in the order in which they completed screening and were blind to study condition.

Procedures

Overview

Between pre and post testing, all participants received 12 individualized, 45 minute training sessions, 2–3 times weekly for 4–6 weeks. To control for amount of exposure, the participants did not take the PDAs home with them during this period, nor were the PDAs available during the 30-day period between post testing and follow-up. Social validity (Research Question #5) was assessed through a post-training survey.

Independent Variable

The independent variable was the type of instruction—systematic vs. conventional instruction. The systematic instruction condition emphasized mastery, incorporating all of the previously described design and delivery elements tailored to the instruction of ATC. These elements included: (a) a limited number of training targets (selected checking and entering calendar skills only); (b) training targets broken down into component parts (steps for checking information trained to mastery individually then in sequence before entering skills introduced); (c) multiple, personally relevant training examples per target (participants’ activities and appointments incorporated into training); (d) multiple instructor models of component steps and carefully faded support; (e) high rates of correct, distributed practice; (f) training outside of the controlled training environment to facilitate generalization of calendar skills across people and environments; (g) repair strategy instruction (participants trained to press the calendar “quick button” to return to current monthly calendar and date when unsure of what to do); and (h) probes each session to determine retention and carefully tailor instruction to the learner’s needs. Buttons and icons (e.g., “Contacts” button; “Home” icon) on the PDA extraneous to the training targets were covered up with stickers during instruction to minimize confusion and increase salience of the instructional targets.

Conventional instruction emphasized exploratory learning, not mastery, and therefore included: (a) the calendar targets taught in the systematic condition as well as targets chosen by the participants according to their personal preferences (e.g., advanced calendar skills, media program, calculator, world clock, games); (b) a limited number of examples per target; (c) general orientation to new material rather than multiple models of individual component steps; (d) no more than 1–2 practice opportunities per target per session; (e) role play to encourage generalization; and (f) no probes each session. No buttons or icons were covered during conventional instruction. (Note: None of the PDA buttons or icons was covered during the pre, post, and 30-day follow-up testing sessions for either group.)

Intervention procedures

The implementation of each training session followed a specific format that aligned with the key elements of each instructional condition. For consistency of implementation, the PI conducted all training sessions across both groups. Systematic instruction sessions were conducted using a specially designed, scripted curricula for the Palm PDA developed by the PI during pilot testing. Specific steps included:

1. Except for the first training session, each session began with a calendar skills check to determine retention from the previous session and skill mastery (100% correct performance across two consecutive skills checks);

2. Any previously taught skills that were not firm (i.e., accurate and fluent) during the skills check were then isolated for intensive modeling and practice;

3. New steps/skills were introduced only when previous material was at or near mastery;

4. The trainer always modeled the step/skill first followed by participant practice, and component parts of multi-step skills were taught to mastery first before integration into multi-step sequences;

5. Regular review of previously mastered skills was integrated into each session;

6. The trainer gave immediate corrective feedback (i.e., modeled step/skill) in response to all errors;

7. Periodic mini-breaks were taken to allow the participant a chance to relax and chat with the trainer on preferred topics; these breaks also served as a means of distributing practice trials;

8. Toward the end of each session, the trainer asked the participants to predict how s/he would do applying their skills outside of the training (e.g., office) setting and/or with another person;

9. The trainer and participant would then implement the skill in that context; and 10. The trainer concluded each session by asking the participant to reflect on how they did in the alternative setting.

Conventional (trial-and-error) instruction sessions were conducted as follows:

1. The trainer and participant began each session by discussing what they had worked on during the previous session, focusing on calendar skills;

2. The trainer would then review or introduce new calendar skills according to participant preferences; these calendar skills included not only those taught in the systematic instruction condition but other skills chosen by the participant (e.g., alternative views of the monthly calendar; alternative methods of input such as graffiti writing);

3. Approximately 15–20 minutes per session were allocated to calendar skill instruction;

4. The remainder of the session was spent exploring other PDA programs chosen by the participant according to her/his needs and preferences (e.g., “Media” program for displaying photos of jewelry; games such as “Solitaire”);

5. Corrective feedback was given only after the participant had attempted to repair an error on his/her own;

6. New multi-step skills (whether the calendar program or other programs) were modeled in their entirety once or twice before the participant attempted practicing the skill; component parts were not isolated and trained to mastery first; and

7. At the conclusion of each session, the trainer and participant reviewed what had happened during the session and role played how the participant might apply PDA skills for use in daily life.

Fidelity of implementation

A trained observer conducted three fidelity checks per participant. These observations were conducted during separate training sessions distributed throughout the course of each participant’s training (e.g., Weeks 1, 3, and 4 of treatment). Fidelity of implementation for systematic instruction was 87% and for conventional instruction 82%, based on the number of instances an instructional component was evident divided by the total number of opportunities to demonstrate that element. Lapses in instructor performance included forgetting to conduct the role-play and review at the end of the conventional instruction sessions and ending the systematic instruction sessions too early. These lapses were discussed between the trainer and fidelity observer following the end of each observed session. In addition, two co-investigators (experienced instructional consultants) viewed videotaped sessions of each condition, offering feedback to the instructor for making slight adjustments to her instructional techniques.

Dependent Variables—Outcome Measure

The outcome measure was a criterion-based, functional assessment of selected PDA skills administered at pretest, posttest, and at 30-days follow up (see Appendix A—Outcome Measure). The evaluator was a speech-language pathologist with over 30 years of clinical experience working with adults with acquired neurogenic disorders. She was trained to criterion to administer the outcome measure and was given regular feedback from the PI and coders during the course of the study in the event that any deviation from the prescribed protocol was noted. Administration procedures for the pre, post, and follow-up sessions included: (a) orienting the participant to each of the 10 tasks; (b) presenting the target information to be checked on an index card for reference; (c) asking the participants to repeat back what they were being asked to do to ensure task comprehension prior to starting the task; and (d) avoiding any cues or prompts that would guide task performance. However, neutral, supportive prompts were allowed (e.g., While pointing to the index card, states—”You can look at the index card as a reminder.”). The evaluator stopped the task under the following conditions: (a) after 1 minute for checking tasks if the participant engaged in no action or appeared to be randomly selecting/pressing items on the PDA; (b) after 2 minutes for entering tasks if no action or random selection; or (c) if the participant requested to stop the task.

APPENDIX A.

Outcome Measure

Participant #:
Date:
Circle which one:		Pre	Post	30-days
Whole Task #	Starting point	Stylus status	Task Content		Target Verbal response	Whole Task (+/−)
1	PDA off	in slot	What is the next appointment today?
2	Today’s daily schedule	in slot	What is the first appointment tomorrow?
3	Tomorrow’s daily schedule (on hour)	out	Enter this appointment: ( ) Return the stylus to its slot when you’re done.		N/A
4	PDA off (not current month)	in slot	What time is the ( ) on ( )?
5	PDA off (current & not current, month not on the hour)	out	You will be scheduling a ( ) appointment. Which of these dates work best? 1) 2) 3))
6	Daily schedule (not on the hour)	out	Enter the ( ) appointment on ( ).		N/A
PDA TASK GENERALIZATION: TASK PROGRAM
7	PDA off	in slot	What are the items on the task list?
8	Task list	out	Enter this item on the task list: ( )		N/A
SETTING GENERALIZATION: OUTSIDE OF CLINIC ROOM
9	PDA off (on the hour)	in slot	TR office staff: What time is your next appt today?
10	Today’s daily schedule	out	TR office staff: Enter your next appt here at TR.

Outcome measure inter-rater reliability

Inter-rater reliability was calculated on 23% of the total number of testing sessions (84 total). Testing sessions selected for reliability coding were evenly distributed across instructional conditions and pre, post, and follow-up testing sessions. The process for establishing reliability included the following steps: (a) Each coder separately watched each video, entering the steps taken to complete each task, the whole task score (+ or −), and time stamp (i.e., beginning and end of each task as determined by the evaluator) into the scoring template; (b) the PI later compared the scoring templates item by item for whole task scores and time stamps to determine inter-rater reliability; and (c) the PI reviewed these comparisons with the coders during weekly meetings, highlighting discrepancies, and strategies for resolving these while remaining consistent with the previously described scoring system. For any item in question, the videotape of that session was replayed during the meetings to help resolve discrepancies. At no time did the PI reveal the instructional condition of the participants to the coders, nor did she conduct comparisons (item c above) while participants were active in the study, thus guarding against scoring bias. Inter-rater reliability for whole tasks scores was 97% and reliability for time-stamp scores was 88%.

Results

Group Equivalence

Pretest comparisons were determined using an independent samples t-test of the differences in sample means. The conventional group earned a mean of 1.67 correct whole tasks (SD = 2.15), and the systematic instruction group earned a mean of 2.29 (SD = 3.45). The number of correct whole tasks out of the ten possible ranged from 0 to 7 correct for the conventional group and from 1 to 10 for the systematic instruction group. Levene’s test for equality of variances was conducted and showed no significant differences in variances of group mean scores, F(1, 24) = 2.50, p = 0.127. Most important, the groups’ pretest performances were not statistically significantly different (t = −0.54, p = 0.60).

Research Question #1—Accuracy at posttest

Prior to analysis, all distributed assumptions for ANCOVA were examined and found to be tenable. To evaluate group differences in accuracy at posttest, a mixed model analysis of covariance (ANCOVA) was conducted, using treatment condition as the predictive factor and pretest performance as a covariate. Effect sizes for the main effects of pretest and treatment condition on the posttest outcome were calculated using the standard equation for Cohen’s d. At posttest, the interaction effect of treatment by pretest performance was not statistically significant. Thus, main effects were examined. The main effect for pretest performance on posttest was statistically significant, at F(5, 13) = 5.22, p < 0.01 as was expected; effect size d = .88. The main effect of treatment condition on posttest performance was not statistically significant, F(1, 13) = 2.28, p = 0.16; effect size d = .58. The model with treatment and pretest scores as factors explained approximately 77% of the variance in the outcome at posttest, R² = .77.

Research Question #2—Accuracy at 30-day follow up

As with posttest, performance at 30-days follow up was analyzed with a mixed model analysis of covariance (ANCOVA) and effect sizes with Cohen’s d; treatment condition was used as the predictive factor and pretest performance as a covariate. At 30-days follow up, the interaction effect of treatment by pretest performance was not statistically significant. Thus, main effects were examined. There was a significant effect of pretest performance on posttest performance, F(5,10) = 26.20, p < 0.01, as expected; effect size d = 2.09. There was also a statistically significant main effect of treatment condition, F(1, 10) = 12.96, p < 0.01; effect size d = 1.44. The results indicated that the treatment condition affected 30-day follow-up performance regardless of initial performance at pretest. The direction of the effect was such that those in the systematic instruction condition performed significantly more correct tasks at 30-day follow-up assessment than did participants in the conventional condition, regardless of their pretest performance level.

The model, including both factors, explains roughly 96% of the variance in the 30-day follow-up scores, R² = .96. This same model with pretest performance and treatment condition as factors explained more variance at 30-day follow-up assessment than it did at posttest. Thus, differences between groups were more apparent at 30-day follow-up than they were at immediate posttest (see Figure 1 below).

Figure 1.

Significant difference in mean number of whole tasks correct, controlling for pretest level (total number of tasks = 10).

Research Question # 3—Fluency at posttest and 30-day follow up

Independent samples t-tests for the difference between sample means were conducted to compare group fluency on correct tasks. First, the seconds per task for all correct whole tasks were totaled. Next, a fluency score was generated by dividing the total seconds by the number of correct whole tasks for each participant. Group mean fluency rates at posttest and 30-days were then calculated and compared using t-tests. (Pretest performance was not included due to a “floor” effect precluding statistical analysis; some participants performed only one or two of the ten tasks correctly at pretest.) At posttest, the systematic instruction group mean fluency score was 42.03 seconds, and the conventional group mean fluency score was 43.60 seconds, thus they were roughly equivalent, t = 0.25, df = 26, p = 0.81; effect size d = .09. At 30-days, however, the systematic instruction group mean was 41.15 seconds, and the conventional group mean was 57.73 seconds revealing a significant difference, t = 2.08, df = 21, p = 0.05; effect size d = .76. The participants in the systematic instruction group, on average, performed the correct tasks 16 seconds faster than did the participants in the conventional instruction group: MT = 41.15 and MC = 57.73 (see Figure 2 below).

Figure 2.

Conventional and systematic instruction mean fluency rates (seconds per correct whole task) were not significantly different at posttest, but fluency was significantly better for systematic instruction participants at follow-up (t = 2.074, p = 0.051).

Research Question #4—Generalization at posttest and 30-day follow-up

Generalization was evaluated in two ways: (a) Content generalization—participants were evaluated on the untrained “Tasks” program (Items 7 and 8 on the outcome measure); and (b) Environmental generalization—participants were evaluated on calendar tasks attempted with someone other than the evaluator, typically another staff member (Items 9 and 10). As noted above, systematic instruction participants were explicitly trained to use their calendar skills with other people and in other environments whereas the conventional instruction participants were not. A mixed model analysis of covariance (ANCOVA) and effect sizes with Cohen’s d were again used for between group comparisons; treatment condition was used as the predictive factor and pretest performance as a covariate. (Note: Items 7–10 were included with Items 1–6 as part of the analyses for Research Questions 1–3 described above. These four items were then separately analyzed to evaluate the generalization component for Question 4.)

For content generalization (i.e., untrained Tasks program), the interaction effect at posttest was not statistically significant. Thus, main effects were examined. There was a significant effect of pretest performance on posttest performance, F(1, 25) = 8.26, p < 0.01; effect size d = 1.11. There was no statistically significant main effect of treatment condition, F(1, 25) = .23, p = 0.63; effect size d = .18. The interaction effect at 30-days follow up was also not significant, and therefore main effects were examined. There was no significant effect of pretest performance on 30-day follow up, F(1, 22) = 2.74, p = 0.011; effect size d = .68. There was also no significant main effect of treatment condition, F(1, 22) = 0.06, p = 0.82; effect size d = .10.

For environmental generalization (i.e., calendar program used with other people), again there were no interaction effects; hence, main effects were examined. There was a significant effect of pretest performance on posttest performance, F(1, 25) = 7.57, p = 0.01; effect size d = 1.06. There was a statistically significant main effect of treatment condition as well, F(1, 25) = 4.32, p < 0.05; effect size d = .80. There was a significant effect of pretest performance on 30-day follow-up, F(1, 22) = 5.39, p = 0.03; effect size d = .95; however, there was no significant main effect of treatment condition at follow-up, F(1, 22) = 1.47, p = 0.24; effect size d = .50.

Research Question #5—Social validity (non-experimental)

Immediately following completion of their post testing on the PDA outcome measure, the evaluator surveyed the participants to determine their overall impressions of the training program they had received (see Appendix B—Social Validity). The majority of the participants across groups conveyed that they were satisfied with the training (92% and 96% systematic and conventional instruction participants, respectively) and would recommend it to a friend; no clear differences were found between the groups.

APPENDIX B.

Social Validity Questionnaire

Circle the most appropriate response
1. The instructor showed me how to do the steps.
strongly disagree	disagree	undecided	agree	strongly agree
2. I was provided with adequate practice opportunities.
strongly disagree	disagree	undecided	agree	strongly agree
3. The instructor provided help if I made a mistake.
strongly disagree	disagree	undecided	agree	strongly agree
4. I am satisfied with the teaching program.
strongly disagree	disagree	undecided	agree	strongly agree
5. I feel confident I can use the PDA.
strongly disagree	disagree	undecided	agree	strongly agree

Missing data

The most obvious patterns of missing data were those due to participants’ lack of assessment at one or more of the data collection waves. Other than a completely missing data collection wave, there was very little missing data (1.3%) for the remaining participants who were present for the three data collection times. Little’s MCAR test was non-significant (Missing Completely At Random; MCAR χ²= 28.69, df = 106, p = 1.00) indicating that the data missing within completed waves were missing completely at random. Missing values imputation was not necessary for these data (Little & Rubin, 2002).

Internal Reliability

To assess the outcome measure’s internal reliability, Cronbach’s alpha was conducted for the three test sessions (pre, post, and 30 days) individually and for the total combined items (pretest reliability alpha = .89, posttest alpha = .84, and 30-day follow-up alpha = .76). For the combined three data waves, internal reliability was satisfactory (alpha = .92). Overall, the three waves individually and combined demonstrated good internal reliability.

For measure stability over time, correlations between individuals’ performance across the three measurement occasions were examined. As expected, there were statistically significant positive correlations between scores on the data at pre, post, and follow-up for both the treatment groups, ranging from .64 to .94. The positive correlation indicates that higher the pretest scores were associated with higher posttest scores.

Discussion

Summary

This study compared the outcomes of skills training using a systematic instructional package with those achieved using a conventional instruction (i.e., trial-and-error learning) package. A double blind, randomized controlled trial was conducted among 29 adult participants with moderate-severe cognitive impairments following ABI to evaluate the instructional effects for learning the Palm Tungsten E2 personal digital assistant. Outcome data revealed no significant differences in accuracy or fluency of learning between groups immediately following instruction (posttest); however, significant differences emerged at 30-day follow-up. With regard to generalization, neither group demonstrated significant changes on an untrained PDA program (the Tasks program); however, significant differences were found in favor of systematic instruction at immediate posttest only for the generalization of trained calendar skills used across environments and other people.

Implications

These results suggest that systematic instruction applied to ATC results in better maintenance and generalization for trained skills than conventional instruction. Furthermore, individuals showed no preference for either instructional approach: participants in both conditions expressed satisfaction with their training.

These results support previous research demonstrating that individuals with moderate-severe cognitive impairments are capable of learning new skills and information when provided with systematic instruction emphasizing error control techniques (Ehlhardt et al., 2008; Sohlberg & Turkstra, 2011; Wilson et al., 1994). Specific to ATC, the results support the incorporation of systematic instruction applied to ATC into clinical practice, particularly given the robust retention at 30-day follow-up. With reduced inpatient and outpatient rehabilitation treatment sessions post-ABI (Kreutzer et al., 2009), it is imperative that clinicians use instructional approaches that will have a lasting impact on client outcomes. Fewer, systematically taught instructional targets are preferable to several targets taught in a less systematic, exploratory fashion. These findings extend the work of others in the field of special education who have shown that systematic instruction leads to improved maintenance and generalization of newly learned skills and information (e.g., Glang et al., 1992; Swanson & Hoskyn, 1998; 1999; 2001).

It is unclear why both groups improved at posttest while the SI group demonstrated significantly better performance at follow-up. One possible explanation could be that both groups received some form of modeling and practice of the target skills; the systematic instruction group received multiple models of component steps to minimize errors combined with high rates distributed practice (i.e., expanded rehearsal; spaced retrieval), while the conventional instruction group received a limited number of models and undistributed practice opportunities for each new target. Hence, while both conditions resulted in initial improvements, it is possible that minimizing errors and distributing practice opportunities in the systematic instruction condition resulted in more durable learning. Distributed practice is thought to enhance the consolidation of learning leading to improved storage and recall (e.g., Challis, 1993; Landauer & Bjork, 1978) and has been shown to be a critical element leading to maintaining performance across different types of instructional targets and populations (e.g., Haslam et al., 2011; Svoboda & Richards, 2009).

Limitations

The previous discussion concerning the possible role of error reduction and distributed practice points to a clear study limitation; it was not possible to determine exactly which systematic instruction design and delivery components contributed most to the therapeutic effects. In addition to error reduction and distributed practice, it is possible that other systematic instructional components such as careful example selection, training to mastery, and strategy instruction also contributed to better maintenance (e.g., Ehlhardt et al., 2008; Kennedy et al., 2008; Stark, Gordon, & Stark, 2008; Stark, Stark, & Gordon, 2005). The determination of which instructional components, whether singly or in combination, contributed most to study outcomes would strengthen the interpretation of results and more clearly inform clinical practice. In addition, the relatively small number of participants limits the extent to which the results can be generalized to the broader population of interest—adults with moderate-severe cognitive impairments due to ABI.

Another primary concern was that the Principal Investigator was the trainer across both instructional conditions. Although this provided consistency during the implementation of training, it also limits the extent to which results can be generalized to other trainers and introduces the potential for significant treatment bias in favor of systematic instruction. Although fidelity of implementation was checked systematically, such bias may still exist. In addition, there was only one follow-up testing interval; a longer interval post-instruction would provide more information about maintenance of skills taught.

Participant-specific performance patterns also introduce potential qualifiers to study results. For example, the systematic instruction participant who accurately performed all 10 whole tasks at pretest was also one of the participants who had previous (but failed) experience using a PDA. It is possible this experience increased his overall comfort with the technology leading to enhanced learning and more durable retention at follow-up. This points to the larger question addressing the extent to which past experience, in this case attempted use of ATC, influences learning and retention of new skills.

Future Research

The previously described study results and limitations offer several possible directions for continuing this line of research. First, additional analyses from this study (e.g., evaluation of steps taken to complete each whole task and use of repair strategies) will enrich the interpretation of the results and possibly offer additional directions for future research. The study should also be replicated with a higher number of participants, different trainers, different types of ATC, other types of instructional targets (e.g., cognitive strategies), and additional follow-up intervals (e.g., 3 months, 6 months, 1 year). Component analysis will help determine which instructional elements contribute most to improved performance. Finally, the experimental evaluation of systematically trained ATC, its impact on daily life and longitudinal use is also warranted in addition to the exploration of person-centered variables (e.g., past use of technology) on learning and retention.

To illustrate, anecdotal verbal reports of one systematic instruction group study participant indicated that prior to her participation in the study, she was very anxious about learning to use technology. Following her participation, she obtained a different type of PDA as part of her vocational rehabilitation program. She quickly learned how to operate the new PDA with systematic instruction specifically designed for that system (Wild, 2009) and rapidly integrated the device into her daily life. She now rarely misses appointments or loses track of tasks she needs to accomplish each day, both important precursors to seeking gainful employment.

Conclusion

Clinicians who work with clients with ABI do not routinely incorporate the assessment and instruction of ATC into their clinical practice (O’Neil-Pirozzi et al., 2004). Consequently, the risk of low utilization and device abandonment increases commensurate with lost opportunities for increased independence and community participation. The results from this study suggest that systematic instruction applied to ATC results in better retention and generalization of trained skills than conventional instruction, with the potential to significantly improve client outcomes. With the rapid pace of technology development, it is essential that the instructional technology used to train these devices keep pace.