Staff Training in Assisted Living: Evaluating Treatment Fidelity

Abstract

Objectives

To investigate a protocol for identifying and evaluating treatment fidelity in STAR (Staff Training in Assisted-living Residences), a structured yet flexible program to train direct care staff to improve care of residents with dementia.

Design

Multi-site feasibility trial.

Setting

Assisted living facilities (ALFs).

Participants

44 direct care staff and 36 leadership staff.

Intervention

STAR is a comprehensive, dementia-specific training program to teach direct care staff in ALFs to improve care and reduce affective and behavioral problems in residents with dementia. It is conducted on-site over two months via 2 half-day group workshops and 4 individualized sessions.

Measures

Treatment fidelity was assessed following the National Institute of Health Behavior Change Consortium model utilizing observations and self-report of trainers, direct care staff and leadership.

Results

Each key area of treatment fidelity was identified, measured, and yielded significant outcomes. For example, significant increases included: direct care staff identifying ABCs (an essential component of training); understanding basics of dementia care; and applying STAR techniques.

Conclusions

Results support that STAR is ready to be translated and disseminated into practice. Because ALFs will continue to provide care for individuals with dementia, the need for effective, practical, and sustainable staff training programs is clear. STAR offers one such option. Hopefully, this report will encourage others to conduct comprehensive evaluations of the treatment fidelity of their programs and thereby increase the availability of such programs to enhance care.

Objective

Effective approaches to improving dementia care in assisted living facilities (ALFs) are becoming increasingly important as the numbers of older adults being cared for in these facilities grows and as evidence accumulates indicating that the level of cognitive impairment, depression and behavioral symptoms in these individuals significantly impacts their quality of life and the effectiveness of care they receive.⁽¹⁾ ALF staff who provide care for individuals with dementia are from diverse backgrounds, and have varying levels of education and experience in caring for older adults with physical and emotional difficulties. Consequently, any program seeking to improve care must improve staff skill as well as enhance on-site supervision and support for them. To avoid the pitfalls of clinical trials conducted in other areas which have had limited utility in changing practice, we sought to develop and evaluate an innovative program for training staff in ALFs to change their behaviors when working with residents with dementia. We specifically delineated, monitored, and evaluated a treatment fidelity protocol as part of this training to insure effective translation of the program to practice. This aspect of program development and evaluation is critical to insure successful translation of evidence-based treatment programs into the context of practice settings. Thus far, no such programs exist in ALFs.

The number of older adults with dementia who live in ALFs is increasing as is the number of states that have specific regulatory provisions for ALFs serving people with Alzheimer’s disease and other dementias.⁽²⁾ Nationally, there are an estimated 38,373 licensed ALFs with 974,585 units/beds.⁽²⁾ Estimates on the percentage of individuals who now reside in ALFs and have a diagnosis of dementia range from 45 – 67% ; 25 –33% of them suffer depression, anxiety, delusions or hallucinations⁽³⁾ and 34 – 56% exhibit behavioral symptoms such as aggression, irritability and restlessness.⁽⁴⁾ These psychological and behavioral symptoms of dementia have been associated with increased morbidity and mortality, decreased quality of life, earlier transfer to nursing homes and increased costs.^(5–7) In addition, dementia-related symptoms are associated with negative outcomes for paid and unpaid caregivers including increased illness, stress, decreased job commitment and increased emotional exhaustion.^(8–10) Therefore, effective training approaches to improving dementia care may improve the health of both ALF residents and staff.

While the level of care in ALFs is generally lower than in other long term care settings and families may be more involved, unlicensed staff still provide the majority of hands on care.⁽¹¹⁾ Consequently, programs aimed at improving staff skills are essential. STAR, Staff Training in Assisted-living Residences (STAR) was developed specifically to address this need.⁽¹²⁾ Investigated in a series of open-ended feasibility trials and one small-randomized controlled trial, STAR has been successful at improving staff skill and resident outcomes. To date, it is the only program specifically for ALFs with some evidence behind it. We were, therefore, interested in determining whether it could be exported into the community and making this determination using a protocol that provided a structured, systematic method of identifying, monitoring, and evaluating treatment fidelity.

The Treatment Fidelity Workgroup of the National Institute of Health Behavior Change Consortium (BCC)⁽¹³⁾ have recommended that a comprehensive approach to treatment fidelity include five areas: (1) design of study, (2) training providers, (3) delivery of treatment, (4) receipt of treatment, and (5) enactment of treatment skills. Consequently, we sought to identify, monitor, and evaluate STAR in each of these areas via a multi-site feasibility trial. In the case of STAR, ‘treatment’ consisted of training direct care staff to increase their skills and change their behaviors when interacting with and providing care for individuals with dementia.

Methods

Overview

The objectives of this study were to:

1. Investigate whether STAR could be implemented successfully in ALFs beyond the initial development sites and with novice trainers who were not part of the original development group;

2. Establish a standardized method for implementation and assessment of STAR’s essential components (realistic expectations, effective communication, identifying activators and consequences of behaviors, problem-solving, pleasant events and making environmental changes); and

3. Establish a protocol for obtaining data on each of the five treatment fidelity areas deemed essential by the NIH Behavior Change Consortium for successful translation of program to practice (design of study, training of providers, delivery of treatment, receipt of treatment and enactment of treatment skills).

To meet these objectives, a multi-site feasibility trial was conducted. ALFs across three different states (WA, IL, ARK) were enrolled to represent urban, rural and suburban areas of the country. Trainers were recruited from each of these states to learn the STAR program and conduct it at their respective ALFs.

Intervention

STAR is a comprehensive, structured yet flexible training program designed to teach direct care staff in ALFs to improve care of residents with dementia. Staff are taught to use the ABC approach⁽¹⁴; ¹⁵⁾ to reduce affective and behavioral problems in residents with dementia by identifying factors within the environment and within staff-resident interactions that can altered. Training focuses on the staff’s interaction with the resident, their role in changing their behavior in order to change resident behavior and the potential for them to intervene in order to decrease resident distress. Training topics include: 1) Realistic expectations for residents with dementia, 2) Effective communications, 3) Identifying and using the ABCs (Activators, Behaviors, and Consequences) to improve resident care, 4) Problem-solving dementia-related affective and behavioral problems 5) Identifying, establishing, and increasing pleasant events, 6) Understanding and altering the environment’s effect on residents, and 7) Issues related to teamwork and resident family issues.

STAR was conducted in the ALFs over two months via two half-day group workshops and four individualized sessions. A manual details all aspects of training including written text for instructors’, ideas for stimulating class discussion; case vignettes to illustrate training concepts; and copies of all overheads and handouts.

Sample

Staff

Forty-four direct care staff (hereafter referred to as staff) and 36 leadership staff (hereafter referred to as leadership) participated from the 8 ALFs. To insure they were familiar with residents and able to institute training objectives, they had to work at least one full day or evening shift two days per week. Consistent with national data, staff and leadership were predominantly female with staff being more ethnically diverse (Table 1).

Table 1.

Staff and Leadership Characteristics (N = 80).

Characteristics [mean ± SD or N (%)]	Staff (n = 44)	Leadership (n = 36)
Female	41 (93.2)	32 (88.9)
Age, years	37.2 ± 12.6	41.6 ± 11.4
Birthplace
USA	26 (59.1)	30 (83.3)
Philippines	8 (18.2)	2 (5.5)
Mexico	5 (11.4)
Africa	4 (9.1)
Russia	1 (2.3)
India		1 (2.8)
Romania		1 (2.8)
Thailand		1 (2.8)
Ukraine		1 (2.8)
Ethnicity
Caucasian	13 (29.5)	26 (72.2)
African-American, Black	9 (20.4)	6 (16.7)
Hispanic, Latino	8 (18.2)	1 (2.8)
Asian	7 (15.9)	3 (8.3)
African, Black	4 (9.1)
Native Hawaiian, Pacific Islander	1 (2.3)
More than one ethnicity	2 (4.5)
Language spoken at home
English	28 (63.6)	31 (86.1)
Spanish	6 (13.6)
Filipino	6 (13.6)	1 (2.8)
African	3 (6.8)
Malaylan		1 (2.8)
Romanian		1 (2.8)
Tagalog		1 (2.8)
Ukrainian		1 (2.8)
Unknown	1 (2.3)
College degree (AA, BA, or graduate degree)	8 (18.2)	22 (61.1)
Married	21 (47.7)	21 (58.3)
Time employed at current facility, months	41.3 ± 53.7	42.0 ± 56.0

Notes: Percentages may not total 100% due to rounding.

Trainers

Three trainers, one from each state conducted STAR; each had specialized training in dementia care with different professional backgrounds (two MSWs and one PhD in Nursing); two women and one man; age 38 –52; with 2-8 years experience in long-term care.

Assessment

Demographic and clinical characteristics

For residents, staff and leadership, we obtained: age, gender, ethnicity, and education. We also obtained length of time in current residence; type, duration, and age of onset of dementia; and the Mini Mental State Exam (MMSE)⁽¹⁶⁾ on residents; and duration of employment at current ALF and years of ALF work experience of staff and leadership.

Treatment fidelity protocol for STAR

Addressing the five areas of treatment fidelity identified by the BCC required development of a protocol to: evaluate that theory based elements are maintained (design of study), assess that individuals implementing the program are consistently trained (training of providers), ensure the intervention is delivered as intended (delivery of treatment), ensure the participants received and understood key content (receipt of treatment) and assess participants appropriate use of program components in real life situations (enactment of treatment skills). The treatment fidelity components and the measures used to assess each component are listed in Table 2 as follows:

Table 2.

Description of Treatment Fidelity Protocol for STAR.

BCC* Treatment Fidelity Components	STAR** Treatment Fidelity Protocol
Design of study	STAR is based on theory and developed by experts. Standardized and manualized training intervention; two 4-hour staff workshops and 4 individual staff sessions (30–45 minutes each), conducted on-site incorporating real-life resident-care situations.
Training of providers	Training manuals and standardized two day seminar to learn STAR and problem-solve difficult training situations. Role play of training strategies during seminar. Ongoing coaching and problem solving with monthly trainer conference calls.
Delivery of treatment	Attendance records (% completion) Checklist of training content and protocol procedures completed by trainer. Regular training phone conferences to address and correct any deviations or issues around training protocols. Staff and leadership evaluations of usefulness of STAR concepts and effectiveness of training materials.
Receipt of treatment	Checklist of staff knowledge and skills demonstrated during role plays in workshops and during individual sessions, e.g., identification of problems, antecedents and consequences; introduction and maintenance of pleasant events; correct communication skills; etc.
Enactment of treatment skills	Checklist of observed staff behaviors during resident care by trainer. Staff completion of assignments (e.g., ABC Cards). Checklist of observed staff behaviors by leadership staff. Staff report of use of STAR strategies (as above).

^*The National Institute of Health Behavior Change Consortium (BCC).

^**STAR – Staff Training in Assisted-living Residences

1. Design of Study

The goals of monitoring and evaluating this area of treatment fidelity include ensuring treatment interventions are based on relevant theory and clinical experiences and that dose of treatment is consistent within and across conditions. The STAR program is based on social learning theory and developed through an iterative process of consultation with various experts in dementia care and staff training. A manual details all aspects of training: it includes written text for trainers (with complete agendas for each session, ideas for stimulating discussion, case vignettes to illustrate training concepts), staff handouts and materials and a DVD including case vignettes specifically designed for STAR.¹ The number of workshop and individual sessions, and frequency and length of contact with staff were consistent across sites.

2. Training of Providers

Monitoring and evaluating this area of treatment fidelity required assuring that training was similar for all program trainers and to minimize ”drift” in trainer skills. A standardized two-day seminar with opportunity for direct observation of participant skills was attended by all program trainers. Ongoing coaching and problem solving was facilitated by monthly trainer conference calls.

3. Delivery of Treatment

Fidelity to delivery of treatment was accomplished in a number of ways. Regular phone conferences were initiated to address and correct any deviations or issues in conducting treatment procedures. For each workshop and individual session, the length of time and the number of sessions attended was obtained and the percent of training received calculated (trainee attendance). At the end of each session, trainers completed a checklist documenting the specific content covered and the videos vignettes viewed (workshop content checklist). Staff and leadership also completed anonymous surveys evaluating STAR content, materials and trainers.

4. Receipt of Treatment

This area of fidelity refers to the extent to which trainees receive the intended treatment; in the case of STAR, we evaluated the degree to which training was received and the content of training understood by the staff. After each session, trainers completed notes and checklists of staff understanding and skill development related to key STAR strategies; they evaluated the degree to which staff could identify problems using ABCs and whether they could do so independently or with assistance (trainer evaluation of staff understanding).

5. Enactment of Treatment Skills

This component of treatment fidelity monitors and evaluates that staff perform treatment related skills in relevant real life settings as intended. This refers to the degree to which staff performed training-related skills on the job. Trainers and ALF leadership rated staff use of each STAR strategy (realistic expectations, effective communication, pleasant events, ABCs to establish a resident-behavior-change plan and environmental cues) and their overall use of STAR concepts (trainer and leadership evaluation of staff performance). Ratings were reported using standardized forms and were based upon direct observation of staff while interacting with residents and during individual sessions. Staff also rated themselves on the same skills and indicated the degree to which they felt these strategies were (or were not) useful.

Statistical Analysis

Data were analyzed using Stata version 10.1⁽¹⁷⁾ using 1) descriptive statistics and graphical examinations of trends over the course of the trainings and 2) McNemar’s test to investigate differences in proportions of staff between initial and final sessions.

Results

Fidelity to Design of Study

The underlying theory and treatment dose within and across ALF sites was consistently applied. A standardized treatment manual was used during the full period of training which consisted of two 4-hour staff workshops and three 1-hour workshops for leadership and 4 individual staff sessions (30–40 minutes each) at each site.

Fidelity to Training of Providers

The three STAR trainers attended a two-day seminar to learn procedures, review training materials, role play training strategies, practice presenting concepts and provide suggestions as to how program might be improved given the potential regional differences across sites. Following this seminar, each trainer conducted STAR in one pilot ALF and then in 2-3 feasibility trial ALFs. Throughout this period, trainers participated in regular conference calls with the STAR development team to ensure protocols were being followed and problems encountered were addressed.

Fidelity to Delivery of Treatment

Participant Attendance

Attendance data was available for 39 staff (89%) and 33 (92%) leadership. One hundred percent of staff attended at least one of the two workshops, 92% attended both, 100% attended at least 2 of the 4 individual sessions, 77% attended all 4. Among leadership, 100% attended one session of 3 sessions, 88% attended two, 51% attended three.

Workshop content checklist

Content checklists were completed on 4 of the 8 ALFs. For workshop 1: each ALF completed all modules and viewed 5 out of 6 videos; for workshop 2: each ALF completed all modules and viewed 4 out of 4 videos.

Staff Evaluation of Training

Twenty-seven(75%) leadership responded. The majority of training strategies were reported helpful including, handouts (89%), STAR tools (discussion: 85%; practical application: 63%), video vignettes (67%), and plans for sustaining STAR in their facilities (89%).

Thirty-three (75%) staff responses were obtained: 100% reported training components helpful and 100% would recommend STAR training to a colleague.

Fidelity to Receipt of Training

Trainer Evaluation of Staff Understanding

Trainer ratings were predominantly positive: 89% of staff were able to identify a dementia-related behavior; 90% were able to identify ABCs of their interactions with residents. Over the course of training, staff ability to independently identify and develop plans for problematic dementia-related behaviors in residents significantly improved over time (eg., able to identify a problem using behavioral terms: 68% session 1 – 93% session 4; McNemar’s χ²(1) = 4.45, p = .03; identify consequences of behavioral problems: 31% at Session 1 to 77% at Session 4 McNemar’s χ²(1) = 8.00, p = .005). The number of staff rated as fully understanding ABC’s also increased significantly (59% session 1 – 97% session 4: McNemar’s χ²(1) = 11.00, p = .001). (See Figure 1.)

Figure 1.

Fidelity to Enactment of Training Skills

Trainer Evaluation of Staff Performance

Resident specific plans for addressing dementia-related behaviors were developed by 90% of the direct care workers at least once during the four individual sessions. Furthermore, a significant improvement in their ability to independently establish these plans was obtained (32% Session 1 - 68% in Session 4; McNemar’s χ²(1) = 6.23, p = .01)) Across all sessions, 89% included pleasant events, 95% improved communication and 74% implemented a change in environment. The use of these techniques increased over time, although they did not reach statistical significance : for example, pleasant events (68% Session 1 – 77%; in Session 4 McNemar’s χ²(1) = 1.29, p = .26), communication skills (74% Session 1 – 81% in Session 4; McNemar’s χ²(1) = .50, p = .48) and environmental change (39% Session 1 – 52% Session 4: McNemar’s χ²(1) = 1.33, p = .25). (See Figure 2.)

Figure 2.

Leadership Evaluation of Staff Performance

Leadership reported that, over the course of training, Staff demonstrated significantly increased ability to communicate (78% to 100%; McNemar’s χ²(1) = 4.00, p = .05); and increased (albeit not statistically significant) abilities to: resolve conflicts between residents (increased 6%), use pleasant events (increased 5%) and alter the environment to manage resident behaviors (increased 6%). Leadership observed staff using improved communication (100%) and the STAR card to problem solve behaviors (74%) resulting in a positive effect on resident care (85%).

Staff self report of performance and attitude

Thirty-five staff (75%) self evaluations were obtained: 77% reported STAR strategies made a difference in how they worked with residents and 80% reported using a specific strategy in their daily interactions with residents in the past 4 weeks. A majority of staff reported frequent use of communication skills (80%), pleasant events (66%), development of a behavioral plan (57%), use of realistic expectations (understanding dementia as a brain disease) (54%) and use of ABC cards (54%).

Discussion

This study investigated a structured protocol for identifying and evaluating treatment fidelity in STAR: Staff Training in Assisted Living Residences. STAR is, a structured yet flexible, comprehensive, training program intended to teach staff in Assisted Living Facilities (ALF) to improve care and reduce affective and behavioral problems in residents with dementia. State-of-the-art treatment fidelity components identified in the literature as essential for the successful translation of programs to practice⁽¹³⁾ were integrated into a multi-site feasibility trail of STAR. These components included monitoring and evaluation of: 1) the maintenance of theory based design (fidelity to study design), 2) consistency in training of individuals implementing the program (fidelity to provider training), 3) delivery of STAR program as intended (fidelity to treatment delivery), 4) participants understanding of program components (fidelity to receipt), and 5) participants appropriate use of program components in real life situations (fidelity to enactment).

Results support we were able to: 1) successfully extend the implementation of STAR beyond the initial ALF sites and with trainers not involved in the initial design; 2) establish a protocol for implementing and assessing STAR’s essential components; and 3) obtain data on the domains deemed essential to establishing and evaluating treatment fidelity. High attendance rates, consistent adherence to treatment procedures and acceptability of training materials indicates successful treatment delivery. As the “dose” of training (number of training sessions) increased there was a significant “response” to the training (increasing recognition of dementia-related behaviors and increased ability to identify antecedents and consequences of behaviors), demonstrating receipt of treatment. While working with ALF residents, staff were observed to be consistently and appropriately applying STAR strategies aimed at improving the affect and behaviors of residents with dementia, indicating enactment of treatment skills. Support for the treatment design (two workshops and 4 individual sessions over 2 months) is supported by the progressive and significant improvements in staff knowledge and skill application over the course of the intervention. In sum, STAR successfully met treatment fidelity objectives. Expansions of STAR both empirically (via a larger multi-site randomized controlled trial) and clinically (via programmatic implementation into broader community practice) are now needed as well as an investigation into how best to sustain these programs over time.

Conducting a multi-site feasibility trial such as this, designed to test the treatment fidelity of such a complex program in actual sites, is not without its challenges and the limitations of our data illustrate these challenges. First, despite our best efforts and consistent monitoring of trainers and staff, we had missing data – while the bulk of our forms were completed, the workshop content was administered to each site but was only completed by half. We had to make a decision to work within the confines of our clinical partners and not push more than we deemed reasonable. Staff were engaged in active clinical work and had other responsibilities in addition to our project. Consequently, our priorities were not necessarily theirs and we had to adjust.⁽¹⁸⁾

Some strategies to monitor and evaluate fidelity were easier to conceptualize than measure. For example, while we have data on each facet of fidelity, some seemed more central to the mission of improving outcomes and therefore had a disproportionate share (e.g. heavier emphasis on measuring enactment of treatment skills). In addition the high initial scores as well as a lack of sensitivity in our measures may have created a ceiling effect in our data and limited the amount of change we identified and/or created. Despite this, significant change was obtained, the pattern of change was consistent and we were successful in reaching our goals of evaluating treatment fidelity for STAR. It will be interesting to determine, over time, if other investigators moving their programs into practice share this emphasis and measurement concerns and how this does or does not influence our collective understanding of how best to operationalize treatment fidelity and move from traditional clinical trials to translational research.

ALFs will continue to grow as a residential option for individuals with cognitive impairment. Therefore, identifying and developing evidence-based and clinically applicable training strategies to improve care will become more important over time. The positive evaluation of treatment fidelity during this multi-site feasibility trial of STAR provides a beginning to this important work. The question now is how best to disseminate and sustain this program; increase the skills of staff and improve the lives of residents living in ALFs.