Mental Illness Training for Licensed Staff in Long-Term Care

Abstract

Licensed care staff working in long-term care facilities may be poorly prepared to work with residents with mental illness. This research reports on the program evaluation of Caring Skills: Working with Mental Illness, a training program delivered on the Internet. It was tested with a randomized treatment-control design, with an eight-week follow-up. The training provided video-based behavioral skills and knowledge training. Measures included video situations testing and assessment of psycho-social constructs including empathy and stigmatization. ANCOVA analysis at 4-weeks posttest showed significant positive effects with medium-large effect sizes, which were largely maintained at the 8-week follow-up. The training was well-received by the users.

Mental illness in long-term care (LTC) facilities is a serious problem. Estimates among the 3.2 million residents vary (US Department of Health and Human Services [US DHHS], 2009), but a recent review indicated that dementia, depression, and anxiety disorders were the most common disorders, occurring more frequently than among community-dwelling older adults (Seitz, Purandare, & Conn, 2010). While 68% of LTC residents have been reported to have cognitive impairment, 41% show moderate-severe impairment (US DHHS, 2009). Individuals diagnosed with mental illness who are admitted to an LTC facility now outnumber those admitted who are diagnosed with dementia (Fullerton, McGuire, Feng, Mor, & Grabowski, 2009); the relative number of individuals with mental illness have grown in recent years (Fullerton et al., 2009; Hoover et al., 2010). Depression alone may affect 17% (Davison et al., 2007) to 30% (Tabloski, 2006) with less than half of LTC residents with major depressive disorder being diagnosed and treated (Davison et al., 2007). Because 47.9% of LTC residents had taken nine or more medications the day before the facility interview for the National Nursing Home Survey, adverse drug interactions also could influence behavior problems (Jones, Dwyer, Bercovitz, & Strahan, 2009). Dementia has been linked to aggressive behaviors in multiple reports (Fick, Agostini, & Inouye, 2002), but up to 90% of patients with dementia are thought to have psychiatric co-morbidities (Lyketsos & Olin, 2002; Plassman et al., 2007), which can exacerbate the risks of problematic behavior (Leonard, Tinetti, Allore, & Drickamer, 2006). Finally, younger individuals (i.e., 21–65 years of age), who are often undiagnosed and without adequate treatment for mental illness (Sedensky, 2011; US DHHS, 1999, 2002), have increased in LTC facilities from 11.1% in 1999 to 13.6% in 2008 (US DHHS, 2009).

Despite the high incidence of psychiatric disorders in LTC facilities, nursing staff are inadequately trained in both geriatrics and mental illness (American Geriatrics Society & American Association for Geriatric Psychiatry, 2004; Glaister & Blair, 2008; Institute of Medicine, 2008; Molinari et al., 2008; Morris & Mentes, 2006; Perese, 2007; Scherer, Bruce, Montgomery, & Ball, 2008). Research suggests that nursing students and health care professionals may hold stigmatizing beliefs about individuals with mental illness (Arvaniti et al., 2009; Bayar, Poyraz, Aksoy-Poyraz, & Arikan, 2009; Glaister & Blair, 2008; Heliker & Nguyen, 2010; Mason, Hall, Caulfied, & Melling, 2010; Schafer, Wood, & Williams, 2011), perhaps at the same level found in the general population (Schulze, 2007), which is contrary to the mores of a caring profession. Thus, nursing staff may be ill prepared when encountering challenging resident behaviors, or when supervising direct care workers who must react to problematic behaviors caused by mental illness. Initiatives to promote a culture change to improve resident care and worker training are recognized by most LTC facilities (Doty, Koren, & Sturla, 2008). Engendered in this approach is a person-centered care concept, replacing the traditional medical management model with one that espouses resident choices and positive resident-staff relationships (Brooker, 2007; Noland, 2001). Culture change, however, is being adopted relatively slowly by the LTC industry (Doty et al., 2008).

The repercussions for care workers subjected to challenging behaviors are beginning to be addressed in the literature. Caregiver reactions to being victims of aggressive resident behavior (i.e., physical or verbal abuse) include anger, stress, fear, job dissatisfaction, and decreased feelings of safety (Gates, Fitzwater, & Succop, 2003; Gates, Fitzwater, & Deets, 2003; Gates, Fitzwater, & Meyer, 1999). Emergency Department health care workers may have experienced symptoms of posttraumatic stress disorder and suppressed immune system functioning after assaults by patients (Gates, Gillespie, & Succop, 2011). Negative experiences or fears of violence increase the likelihood that staff will fear or avoid residents with mental illness (Gates, Fitzwater, & Succop, 2003; Gillespie, Gates, Miller, & Howard, 2010; Molinari et al., 2008; Park, 2010), which then erodes the quality of resident care. A repercussion for the resident is that mental illness behaviors are too often treated with psychotropic medications or restraint instead of behavioral methods (Brooker, 2007; Donat & McKeegan, 2003; Kolanowski, Fick, Frazer, & Penrod, 2010; Voyer & Martin, 2003).

Education and generic mental health training is beneficial for staff performance, improves job satisfaction, and promotes quality of care for residents with mental illness (Glaister & Blair, 2008; Moyle, Hsu, Lieff, & Vernooij-Dassen, 2010; Yeatts, Cready, Swan, & Shen, 2010). Good communication skills are important for positive staff-resident interactions (Williams & Herman, 2011), and research has shown the potential for brief communication skills training to positively impact interactions with residents (Williams, 2006; Williams, Kemper, & Hummert, 2003).

Previously, we developed an Internet mental illness training program for Nursing Assistants (NAs) that was designed to improve knowledge of mental illness and communication skills. A pilot study demonstrated significant effects with NAs and a follow-up quasi-experimental test of the same training showed positive effects with licensed health professionals (LHPs; e.g., Registered Nurses [RNs], Licensed Practical Nurses [LPNs], Occupational Therapists [OTs], Social Workers [SWs], and Physical Therapists [PTs]) (Irvine, Billow, Bourgeois, & Seeley, 2012).

The purpose of the research reported here was to examine training effects of an expanded Internet mental illness training program with a larger sample of LHPs. The program evaluation goals were to measure changes in: (a) appropriate behavioral and communication techniques for working with residents with mental illness, (b) knowledge of mental illness, and (c) attitudes, self-efficacy, empathy, and stigmatizing beliefs regarding resident with mental illness. Additionally we sought to measure user acceptance of the training.

METHODS

Overview

For this research, we re-designed the prototype mental illness training (Irvine, Billow, Bourgeois et al., 2012), dividing it into three courses: knowledge about mental illness, fostering caregiver-resident relationships, and behavioral skills to work with mental illness behaviors. Additionally, we developed a total of eight new courses, five of which applied the behavioral skills and knowledge to a specific mental disorder (i.e., depression, anxiety, manipulation, confusion, false beliefs), two that dealt with specific behavioral issues (i.e., aggressive resident behaviors; danger to self), and another course designed to promote caregiver self-care after upsetting interactions with residents. The training was designed to develop skills and confidence to deal with symptoms of whatever mental illness was causing the behavior. Minimum total viewing time for all the courses was approximately four hours.

This research was conducted in collaboration with staff from a large nursing home chain, which assisted with preparation of the research proposal, but they had minimal input on content development. Corporate executives selected two disorder-specific courses to be included in the evaluation from a menu of six courses. The training program was evaluated in a real-world effectiveness study (Flay & Sobel, 1983) of LHPs as reported here, and in another study using all NAs in three LTC facilities, to be reported elsewhere (Irvine, Billow, McMahon, Eberhage, Seeley, et al., 2012).

Research Design

Participants were recruited and randomized into treatment and control groups as described below (see Figure 1). After consenting and submitting a baseline assessment, treatment participants were asked to visit the training program twice, approximately one week apart, to view two courses in each visit. Approximately two weeks after completion of Visit 2, participants in both groups completed a posttest assessment. One month later, they completed a follow-up assessment.

FIGURE 1.

Research design for program evaluation of Internet Mental Illness training program

The Internet Program: “Caring Skills: Working With Mental Illness”

The training program consisted of Internet courses presented in a Learning Management System format, which is often used for continuing education courses. The courses were designed to be 10–30 minutes in length and to embrace a person-centered care philosophy. That is, a resident's potentially problematic behavior is viewed as an expression of an unmet need, rather than just a behavior to be managed. Staff reactions, as modeled in the training, include engaging the resident in conversation to attempt to understand the resident's perspective, listening to the resident, and then acknowledging the resident's emotion as part of the behavioral solution. As described elsewhere, the mental illness training approach included video modeling vignettes, right-way and wrong-way exemplars, testimonials, and narration supplemented by short on-screen text designed create empathy for residents with mental illness. Each course included up to ten multiple choice response items embedded at the end of the program to test comprehension, each of which included feedback summarizing the correct information for correct or incorrect responses.

The training was funded to be designed for NAs who are typically less educated than LHPs. Consequently, on-screen text elements were short and written at the second–sixth grade reading level. Text was shown on-screen after the video concluded to be consistent with cognitive theory of multi-media learning (Learning Theories Knowledgebase, 2011; Mayer, 2001), which suggests that viewers receive information via separate audio and video channels, and learn best by processing information from a single channel at a time. Except for log-in, keyboarding was not required (i.e., navigation and responses to questions required only mouse clicks).

Visit 1

Two courses (i.e., About Mental Illness; The AID Care Strategy) were presented in Visit 1. These courses were designed to provide fundamental education and communication skills. In About Mental Illness the video narrator briefly presented descriptions of common mental disorders (i.e., anxiety disorders, schizophrenia and psychotic disorders, mood disorders, cognitive disorders, and personality disorders). For each disorder, potentially observable behaviors were listed on-screen. Interested users could click on a “More Details” button to see a more sophisticated 100–300 word article about that disorder, written at a tenth grade reading level or less. Additional content was designed to dispel common myths about mental illness by presenting statements (e.g., “Residents with mental illness are often violent.” “Depression is normal for residents.”) derived from mental health websites. For each statement, the user clicked on icons indicating whether the statement is a myth or a fact. An animated screen scored the choice as correct or incorrect, with voice-over narration and text providing information to reinforce the facts or dispel myths. The user then responded to the same statement until a correct choice was selected.

The AID Care Strategy course provided communication and observational skills training for interactions with residents who display behaviors possibly caused by mental illness. Video vignettes (e.g., an NA reacting to resident having hallucinations) were supplemented by narration and supportive testimonials. The video models demonstrated fundamental techniques of a training strategy called the AID, which has been used in previous training research (Irvine, Billow, Bourgeois, et al., 2012; Irvine, Billow, Gates, et al., 2011; 2012; Irvine, Bourgeois, Billow, & Seeley, 2007). In brief, “A” stands for Assess the situation from out of arm's reach. “I” stands for Investigate, which includes approaching calmly, interrupting the behavior with a friendly greeting, watching the resident's body language, and listening to gather clues about the cause of the behavior. “D” stands for Do something, which includes acknowledging the resident's emotions and finding a suitable redirection.

Visit 2

Two courses focusing on more advanced skills were offered in Visit 2: Care Strategies for Manipulative Behavior and Care Strategies for Aggressive Behavior. Both courses demonstrate the application of the AID principles (e.g., manipulation: resident attempts different manipulative techniques to obtain snacks restricted by the care plan; aggression: angry resident demands to leave). New content on symptoms and the importance of proactively recognizing symptoms was presented. The principles of the AID approach, as taught in Visit 1, were applied, and modeled with audio-scenarios. Use of active listening and validation techniques were emphasized as part of the “I” of AID, and were specifically explained and modeled in each course. Multiple exemplars of sample responses and redirection ideas were interactively available by clicking the computer mouse.

Procedures

After the research protocol was approved by an Institutional Review Board, participants were recruited via e-mail announcements, e-newsletters, online message boards, Internet advertising, and word of mouth. Potential participants were required to visit our informational website. If interested in participating, they were linked to an online screening instrument that determined eligibility and gathered contact information. Eligibility criteria included: (1) identification of professional license from a list (i.e., RN, LPN, Nurse Practitioner, Physician's Assistant, Psychologist, OT, SW, PT, Speech Therapist, Licensed Nursing Home Administrator, Registered Dietician, and Other, which was examined on a case by case basis). Also listed, but causing exclusion if checked, were Certified Nursing Assistant, Nursing Assistant, and Home Health Aide; (2) type of facility worked: nursing home and assisted living settings were accepted; psychiatric/Alzheimer's care units and hospitals were excluded; an Other category was examined on a case by case basis; (3) hours worked per week: less than 20 hours per week caused exclusion; (4) self-reported level of mental illness-specific training: exclusion resulted from checking 4 or 5 on a 5-point scale (i.e., 0 = none, 4 = A moderate amount; 5 = a lot); and (5) self-reported self confidence to deal with resident behaviors associated with mental illness: 5 on a 5-point rating scale (1 = Not very confident; 5 = Extremely confident) caused exclusion. Participants were automatically tested to ensure they were enrolling from an Internet-video capable computer with broadband connection, and a valid e-mail address was required. Once qualified, they were asked to provide full contact information to permit e-mail communication and monetary compensation by mail.

In our previous Internet research, a few applicants have attempted to screen-in by providing false information to qualify for a study. Consequently, in this study the data provided by each potential participant was checked against our database of 8500 records of previous Internet study applicants, to cross check for fraudulent information (e.g., same name, inconsistent age, gender, or ethnicity). Screened-in participants providing suspicious data were telephoned, and if the inconsistencies were not resolved, the individual was excluded from the study.

Approved participants were randomized into treatment (Tx) or control (Ctrl) conditions. They were e-mailed a link to an online informed consent form, and if they agreed, they were then linked to the baseline assessment. After submitting the baseline assessment, treatment participants were e-mailed login information to the Internet training program for Visit 1. One week after logging on to the Visit 1 courses, each participant was sent a second e-mail with log-in information for Visit 2. No effort was made to require participants to complete the training coursework at either visit. As needed, research staff sent two e-mail reminders, three and five days after participants were sent the program link. If, after seven days, the coursework was still not accessed, research staff also telephoned the participant to ensure he or she had received the link and did not have any questions or difficulties logging on to the training. The additional e-mail, and, if necessary, phone contact, was framed as problem-solving, and was designed to not be coercive.

Four weeks after the baseline assessment and about two weeks after Visit 2 to the training program (see Figure 1), Tx and Ctrl participants were e-mailed log-in information to complete the posttest assessment. The Tx group assessments also included additional items to measure viewer acceptance of the program. After submission of the posttest assessment, the Tx group did not have access to the training program. Four weeks later (i.e., eight weeks after baseline), all participants were e-mailed login information for follow-up assessments, which were identical to the baseline assessment instrument. After follow-up assessments were submitted, all participants were provided access to the program for three months.

All participants were paid up to $175 by check for submitting assessments: $50 for baseline, $50 for posttest, and $75 for the follow-up. Tx group participants were not reimbursed for program visits. Tx participants who completed the training program were mailed a certificate of completion indicating their time spent viewing the courses.

Measures

The assessment was designed to measure situational learning, knowledge, attitude, self-efficacy, empathy, stigmatization, and program acceptance. Changes were measured with video situational testing (VST) and constructs associated with behavior change from social cognitive theory (Bandura, 1969, 1977) and the expanded theory of reasoned action (Fishbein, 2000), including attitudes, self-efficacy, and behavioral intentions. Additional items measured potential changes in empathy and stigmatization because research reports indicate that health care professionals may benefit from training designed to alter stereotypes and negative attitudes towards mental illness (Arvaniti et al., 2009; Glaister & Blair, 2008; Heliker & Nguyen, 2010; Mason et al., 2010; Schafer et al., 2011). We postulated that positive training effects would improve user knowledge, attitudes, intentions, and self-efficacy, with empathy increasing, and stigmatization decreasing.

VST

VST assessed participant reactions to short video vignettes of resident behaviors (e.g., resident is upset and delusional; resident grabs another resident forcefully; resident appears depressed). VST approximates real-life behavior when in vitro observations are not practical. Situational testing previously has shown positive changes after training for NAs (Irvine, Ary, & Bourgeois, 2003; Irvine, Billow, Gates, Fitzwater, Seeley, et al., 2011; 2012; Irvine, Billow, McMahon et al., 2012; Irvine, et al., 2007) and a small population of LHPs (Irvine, Beatty, & Seeley, 2012; Irvine, Billow, Bourgeois, et al., 2012). Video situational testing vignettes were presented to participants.

VST Knowledge

One multiple choice item per VST segment was used to assess a participant's knowledge of the appropriate reaction to the situation. The number of correct items was summed and divided by seven to indicate total percent of knowledge items correct. Adequate test-retest reliability in the control condition was demonstrated from baseline to posttest (r = .47)

VST Self-Efficacy

Two items per VST segment were used to assess a participant's level of confidence in their ability to know what to do and to successfully alter the residents’ behaviors in the depicted situation. Response options were recorded on a 7-point rating scale (1 = not at all confident, 7 = extremely confident). A mean composite score was computed across the 14-items and showed excellent internal consistency (α = .96) and good test-retest reliability in the control condition from baseline to posttest (r = .70).

Attitudes

Nine items were used to assess participant's attitudes towards residents with mental illness, and behaviors possibly related to mental illness. Response options were recorded on a 7-point Likert scale (1 = completely disagree, 7 = completely agree). A mean composite score across the nine items was computed and showed adequate internal consistency (α = .70) and adequate test-retest reliability in the control condition from baseline to posttest (r = .54).

Self-Efficacy

Eleven items were used to assess level of participant confidence in their ability to apply the concepts taught in the program (e.g., “How confident are you in your ability to investigate what is bothering an agitated resident?”). Response options were recorded on a 7-point rating scale (1 = not at all confident, 7 = extremely confident). A mean composite score was computed across the 11 items and showed excellent internal consistency (α = .93) and test-retest reliability in the control condition from baseline to posttest (r = .79).

Intentions

Nine items were used to assess participant's intentions if faced with an aggressive resident behavior (e.g., “How likely is it you will give a positive greeting to a resident who is acting aggressively towards you?”). Response options were recorded on a 7-point rating scale (1 = not at all likely, 7 = extremely likely). A mean composite score across the nine items was computed and showed excellent internal consistency (α = .92) and adequate test-retest reliability in the control condition from baseline to posttest (r = .50).

Empathy

Four items were used to assess participant's empathy toward a resident (Ray & Miller, 1994). Response options were recorded on a 7-point Likert scale (1 = completely disagree, 7 = completely agree). A mean composite score across the four items was computed and showed good internal consistency (α = .80) and good test-retest reliability in the control condition from baseline to posttest (r = .75).

Knowledge

A total of 16 items tested user knowledge. The ten myths and facts presented in the training program were used, with respondents asked to judge each statement as “myth,” “fact,” or “unsure,” which was scored as incorrect. Six additional multiple-choice items tested knowledge of causes of aggression, communication skills, and dealing with manipulative behavior. Correct items were summed and divided by the total number of items to represent the percent correct. Adequate test-retest reliability in the control condition was demonstrated from baseline to posttest (r = .61)

Stigma

The 27-item Attribution Questionnaire AQ-27 (Corrigan, 2004, 2008) was used to assess nine stereotypes about people with mental illness: blame (α = .53), anger (α = .71), pity (α = .68), help (α = .82), dangerousness (α = .82), fear (α = .75), avoidance (α = .74), segregation (α = .70), and coercion (α = .55). It was expected that training effects would cause all subscales except “help” to decrease. The items are statements about Harry, a fictional 30-year-old single man with schizophrenia who sometimes hears voices and becomes upset (e.g., I think Harry poses a risk to his neighbors unless he is hospitalized). Response options were recorded on a 9-point rating scale (1 = not at all, 9 = very much). A higher score indicates higher levels for each stereotype.

User Acceptance

Items to gauge user acceptance of the website were included on the posttest assessment for Tx participants only. The items included the System Usability Scale (SUS; Brooke, 1996; Sauro, 2011), and others adapted from Internet studies (Chambers, Connor, Diver, & McGonigle, 2002; Vandelanotte, De Bourdeaudhuij, Sallis, Spittaels, & Brug, 2005) and from our previous research to measure non-overlapping usability indicators and perceived program impact by the trainee (Chambers et al., 2002; Vandelanotte et al., 2005).

Two items asked users to rate the training program on a 7-point scale (1 = not at all; 7 = extremely) in terms of user satisfaction (“Overall, how satisfied were you with the training website?”) and recommend-ability (“How likely is it that you would recommend the training site to a co-worker?”).

System Usability Scale (SUS)

Originally developed for use after usability testing as a global measure of satisfaction, the SUS is a 10-item survey that assesses usability and learnability. The SUS has been referenced in over 600 publications (Sauro, 2011). Users respond to agree-disagree statements on a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree). Negatively worded items are reversed coded. The SUS may be scored as a percentile rank and compared with 500 other studies in a process comparable to grading on a curve, with a score of 68 considered average (Sauro, 2011).

Program Value

Eight items elicited responses about the perceived program value as a training tool to the user. Five items were about personal values (i.e., applicable to job, good use of time, video vignettes helpful, audio vignettes helpful, compared to an in-service), one item asked about the value of the training for non-direct care staff, and two items asked about the value of the training for direct care staff. Users were asked to agree or disagree with statements by responding on a 6-point Likert scale (1 = strongly disagree; 6 = strongly agree).

Program Impact

Seven items measured how the program would impact users’ perceptions of their employer and their job. The stem was, “If my employer made this training available to all company employees. . .” Items mentioned feeling cared about, positive feelings towards the company, commitment to the company, better care for residents, and personal productivity, safety, and satisfaction (6-point Likert scale; 1 = strongly disagree; 6 = strongly agree).

Data Analysis

Analysis of covariance (ANCOVA) models were used to test for differences between conditions at posttest and follow-up adjusting for pre-training scores. Missing data ranged from 0% to 9% and the missing at random assumption remained tenable (see Attrition Analysis below). Despite the low rates of missing data we employed an intent-to-treat analysis by using maximum likelihood estimates to impute missing data, as it produces more accurate and efficient parameter estimates than listwise deletion or last-observation-carried-forward (Schafer & Graham, 2002).

Effect size computations complement inferential statistics (i.e., p-values) by estimating the strength of the relationship of variables in a statistical population. Reporting effect sizes facilitates an understanding of the substantive, as opposed to the statistical, results of each study and allows for comparison of effects across different studies. Effect sizes are reported as eta-squared (Cohen, 1988; .01 = small effect, .06 = medium effect, and .14 = large effect).

Ancillary analysis for the treatment participants included dose response (i.e., did greater program usage result in greater improvement in study outcomes?) and descriptive summaries of program acceptance and satisfaction, usability, and impact on employee/employer relationships, job performance, and satisfaction. To evaluate effects of dose response, posttest scores were correlated with total time of program use (reported in minutes) while controlling for pretest scores. Effect sizes are reported as partial Pearson product-moment correlation coefficients and interpreted with Cohen's (1988) convention of small (r = .10), medium (r = .30), and large (r = .50) effect sizes.

RESULTS

Baseline Equivalency and Attrition Analyses

Study experimental condition was compared to the demographic characteristics shown in Table 1, and to the baseline assessment of all study outcome measures (Table 2). No signifi-cant differences were found (p < .05), suggesting randomization produced initially equivalent groups. Of the 172 study participants 91% completed all three assessment surveys, 6% completed two surveys, and 3% completed one survey Participants who completed all three surveys were compared to those who completed one or two surveys on study condition, demographic characteristics, and all baseline outcome measures. Attrition was not significantly related to any of the measures, which suggests that dropping out of the study did not bias results.

TABLE 1.

Demographic Characteristics by Study Condition

	Treatment (n = 84)		Control (n = 88)
	N	%	N	%
Female	72	85.7	75	85.2
Hispanic or Latino heritage	8	9.5	8	9.1
Race
American Indian/Native American	2	2.4	0	0.0
Asian	5	6.0	7	8.0
African American	5	6.0	10	11.4
Caucasian	67	79.8	61	69.3
Mixed	3	3.6	10	11.4
Other	2	2.4	0	0.0
Age
21–35	30	35.7	33	37.5
36–45	27	32.1	26	29.5
46–55	23	27.4	25	28.4
56–65	4	4.8	4	4.5
Highest level of education completed
High school diploma or GED	2	2.4	0	0.0
Some college or associates degree	32	38.1	46	52.3
College degree	38	45.2	30	34.1
Graduate school	12	14.3	12	13.6
Annual household income
Less than $20,000	3	3.6	1	1.1
$20,000–$39,999	6	7.1	11	12.5
$40,000–$59,999	23	27.4	21	23.9
$60,000–$79,999	24	28.6	25	28.4
$80,000–$99,999	12	14.3	14	15.9
More than $100,000	16	19.0	16	18.2
Professional licensure
Registered nurse	41	48.6	37	42.0
Licensed practical nurse	25	29.8	30	34.1
SW	6	7.1	1	1.1
Physical therapy	3	3.6	4	4.5
Speech Therapy	1	1.2	3	3.4
Occupational therapy	1	1.2	2	2.3
Other	7	8.3	11	12.5
Percent of workday spent directly caring for residents
0 – 25%	13	15.5	14	15.9
26 – 50%	15	17.9	12	13.6
51 – 75%	23	27.4	16	18.2
76 – 100%	33	39.3	46	52.3

TABLE 2.

Descriptive Statistics for Study Outcome Measures by Study Condition

	Treatment (n = 84)						Control (n = 88)
	Baseline		Posttest		Follow-up		Baseline		Posttest		Follow-up
	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
VST Self-efficacy	3.95	1.13	4.76	1.10	5.24	1.06	3.87	1.00	4.16	1.03	4.44	1.05
VST Knowledge	0.51	0.18	0.63	0.19	0.61	0.22	0.49	0.17	0.49	0.18	0.49	0.21
Attitudes	5.70	0.75	5.99	0.77	6.05	0.67	5.77	0.61	5.71	0.74	5.80	0.77
Self-efficacy	4.32	1.11	5.33	1.01	5.51	0.92	4.20	0.95	4.47	1.05	4.77	1.08
Intentions	4.75	1.31	5.47	1.21	5.46	1.27	4.96	1.15	4.98	1.24	4.96	1.23
Empathy	5.12	1.00	5.54	0.94	5.47	0.97	5.07	1.08	5.21	1.04	5.20	1.13
Knowledge	0.71	0.13	0.80	0.13	0.80	0.15	0.70	0.15	0.76	0.12	0.76	0.13
Stigma
Blame	8.14	4.00	8.20	4.33	9.88	4.92	8.33	4.28	9.35	4.38	9.68	4.56
Anger	5.62	2.79	5.20	2.77	5.85	3.75	5.83	3.20	6.18	3.39	6.09	3.26
Pity	17.57	4.44	16.75	4.59	16.04	4.40	17.30	5.70	16.94	5.53	17.06	5.38
Help	22.08	4.39	23.19	3.60	23.00	3.98	22.10	4.33	22.11	4.03	22.34	3.92
Dangerousness	8.04	3.67	6.02	3.04	6.19	3.19	8.28	4.48	7.93	4.36	7.88	4.30
Fear	4.76	2.50	3.67	1.88	3.82	2.24	4.61	2.70	4.62	2.61	4.57	2.61
Avoidance	14.04	5.48	11.26	5.71	11.30	6.02	14.00	5.98	12.89	6.30	12.31	6.34
Segregation	6.52	3.83	5.62	3.54	6.46	4.76	6.40	3.67	6.18	3.70	5.98	3.48
Coercion	15.21	4.99	14.10	4.88	13.07	5.33	14.18	4.95	14.08	5.19	13.23	5.20

M = mean, SD = standard deviation, VST = video situation test.

Group Differences at Posttest

Results from the ANCOVA models show significant group differences at posttest in VST self-efficacy, VST knowledge, attitudes, self-efficacy, intentions, empathy, and knowledge after adjusting for pretest scores (see Table 3). Comparison of the posttest scores, adjusted for pretest scores, show greater gains by the Tx condition for each of the outcomes with most having significant differences compared to the Ctrl condition. Significant differences correspond to large effect sizes for self-efficacy, medium effect sizes for VST self-efficacy, VST knowledge, attitudes and intentions, and small effect sizes for knowledge, empathy. Stigma subscales showed significant gains for Anger, Help, Dangerousness, Fear, and Avoidance, and a positive trend for Blame (p = .06), with small to medium effect sizes.

TABLE 3.

Posttest ANCOVA Results.

	Adjusted Means1		Test Statistics
	Treatment	Control	F-value	p-value	Eta2
VST self-efficacy	4.74	4.19	19.51	<.001	.10
VST knowledge	0.62	0.49	24.59	<.001	.13
Attitudes	6.01	5.70	9.07	.003	.06
Self-efficacy	5.30	4.50	44.79	<.001	.21
Intentions	5.52	4.92	14.43	<.001	.08
Empathy	5.52	5.23	8.67	.004	.05
Knowledge	0.80	0.76	7.14	.008	.04
Stigma
Blame	9.30	8.26	3.58	.060	.02
Anger	5.25	6.13	4.42	.037	.03
Pity	16.67	17.03	0.37	.544	<.01
Help	23.20	22.11	4.84	.029	.03
Dangerousness	6.10	7.86	16.26	<.001	.09
Fear	3.63	4.66	12.56	.001	.07
Avoidance	11.25	12.90	5.28	.023	.03
Segregation	5.58	6.22	2.56	.111	.02
Coercion	13.77	14.40	1.05	.306	.01

Eta² = measure of effect size (.01 = small effect, .06 medium effect, and .14 large effect.)

¹Adjusted for T1 scores. All F statistics evaluated with 1 and 169 degrees of freedom.

Maintenance of Effects at Follow-up

Results from the ANCOVA analysis show significant group differences at follow-up for VST self-efficacy, VST knowledge, attitudes, self-efficacy, and intentions after adjusting for pretest scores (Table 4). Comparison of the follow-up scores adjusted for pretest scores show greater gains by the Tx condition compared to the Ctrl condition for each of the outcomes with significant differences. Significant differences correspond to (a) medium to large effect sizes for VST self-efficacy and self-efficacy, (b) medium effect sizes for VST knowledge and intentions, and (c) a medium to small effect size for attitudes. Trend level group differences for empathy and knowledge (p = .055, Tx; p = .078, Ctl) were found with the Tx condition showing larger adjusted means compared to the Ctrl group. Stigma subscales of Pity, Dangerousness, and Fear were significant, with small to medium effect sizes. Overall, the results indicate that the training effects demonstrated at posttest were largely maintained through follow-up, about 8 weeks after the pretest and 6 weeks after the training.

TABLE 4.

Follow-up ANCOVA Results

	Adjusted Means1		Test Statistics
	Treatment	Control	F-value	p-value	Eta2
VST self-efficacy	5.22	4.47	33.29	<. 001	.17
VST knowledge	0.61	0.50	12.84	<. 001	.07
Attitudes	6.06	5.79	6.46	.012	.04
Self-efficacy	5.48	4.81	30.92	<.001	.16
Intentions	5.51	4.90	13.60	<.001	.08
Empathy	5.46	5.22	3.74	.055	.02
Knowledge	0.80	0.77	3.14	.078	.02
Stigma
Blame	9.92	9.63	0.20	.653	<.01
Anger	5.89	6.04	0.09	.762	<.01
Pity	15.96	17.14	4.35	.039	.03
Help	23.01	22.35	1.88	.172	.01
Dangerousness	6.25	7.81	10.98	.001	.06
Fear	3.77	4.61	8.28	.005	.05
Avoidance	11.29	12.32	2.19	.141	.01
Segregation	6.42	6.02	0.65	.422	<.01
Coercion	12.74	13.55	1.56	.214	.01

Eta² = measure of effect size (.01 = small effect, .06 medium effect, and .14 large effect).

¹Adjusted for T1 scores. All F statistics evaluated with 1 and 169 degrees of freedom.

Program Usage and Dose Response Analysis

Most treatment participants (n = 74; 88%) viewed all four courses, and all participants except six (7%) saw at least one course. The average amount of time spent using the program was 92.0 minutes (SD = 40.9). AID Care Strategy took the longest to view (Mean = 30.2 minutes, SD = 16.1), followed by Decreasing Risks for Aggressive Behavior (Mean = 24.7 minutes, SD = 12.7), Care Strategies for Manipulative Behaviors (Mean = 21.1 minutes, SD = 11.1), and About Mental Illness (Mean = 18.0 minutes, SD = 8.7).

To assess dose response, posttest scores were correlated with total time of program use while controlling for pretest scores. Statistically significant effect sizes in the small to medium range were found between time of program usage and improvement in VST self-efficacy (partial r = .24, p = .040), VST knowledge (partial r = .25, p = .031), attitudes (partial r = .28, p = .015), intentions (partial r = .18, p = .122), empathy partial r = .17, p = .144), and knowledge (partial r = .28, p = .016). A statistically significant small effect size was found for the stigma Anger subscale (partial r = –.27, p = .020). Taken together, these results suggest treatment participants who invested more time using the program showed modest improvement in study outcomes compared to participants who used the program less.

Program Acceptance

Treatment participants responded to the 10-item SUS, which assessed usability and learnability (Table 5). The total score of 90.6 corresponds to the 99th percentile, indicating that the usability and learnability of the program outperformed 99% of the 500 studies used to norm the data. Next, treatment participants responded to eight items assessing the perceived value of the program as a training tool (Table 6). An overall Program Value score was computed across the eight items and the mean of 5.4 (SD = 1.0) on a 6-point scale indicated that the participants found the program highly valuable as a training tool. Finally, treatment participants responded to seven items assessing the impact the program would have on employee/employer relationships, and job performance and satisfaction (Table 7). An overall Program Impact score was computed and the mean of 4.8 (SD = 1.4) on a 6-point scale indicates that participants were mostly positive about the impact of the training program. The overall mean program satisfaction score on a 7-point scale was 6.13 (SD = 1.18), and the recommendability score on a 7-point scale was 5.99 (SD = 1.37).

TABLE 5.

Descriptive Statistics for the System Usability Scale

	Strongly disagree %	Disagree %	Neither agree or disagree %	Agree %	Strongly agree %	Mean	SD
I think that I would like to use this program frequently.	5.1	1.3	12.8	38.5	42.3	4.12	1.03
I found the program unnecessarily complex.	73.1	10.3	0.0	12.8	3.8	1.51	0.99
I thought the program was easy to use.	0.0	0.0	3.8	10.3	85.9	4.82	0.48
I think that I would need the support of a technical person to be able to use this program.	84.6	11.5	3.8	0.0	0.0	1.19	0.49
I found the various functions in this program were well integrated.	0.0	0.0	5.1	19.2	75.6	4.71	0.56
I thought there was too much inconsistency in this program.	79.5	12.8	5.1	1.3	1.3	1.32	0.75
I would imagine that most people would learn to use this program very quickly.	3.8	0.0	2.6	19.2	74.4	4.60	0.87
I found the program very cumbersome to use.	83.3	5.1	9.0	0.0	2.6	1.33	0.85
I felt very confident using this program.	2.6	0.0	0.0	13.0	81.8	4.71	0.76
I needed to learn a lot before I could use this program.	83.3	7.7	5.1	1.3	2.6	1.32	0.85

SD = standard deviation.

TABLE 6.

Descriptive Statistic for Program Perceived Value as a Training Tool

	Strongly disagree %	Mostly disagree %	Somewhat disagree %	Somewhat agree %	Mostly agree %	Strongly agree %	Mean	SD
I can apply the training content to my job.	3.8	5.1	0.0	9.0	21.8	60.3	5.21	1.32
Viewing the training was a good use of my time.	3.8	2.6	1.3	3.8	20.5	67.9	5.38	1.22
The video vignettes in the training were helpful.	3.8	1.3	0.0	2.6	25.6	66.7	5.45	1.11
The audio vignettes in the training were helpful.	3.6	1.3	0.0	5.1	25.6	64.1	5.40	1.13
Compared to an in-service on a similar topic, I would learn more using a training website like this.	2.6	1.3	2.6	12.8	28.2	52.6	5.21	1.12
This training would be a value to help all non-direct care staff (e.g., administrators) interacts with residents with mental illness.	3.8	0.0	0.0	2.6	25.6	67.9	5.50	1.04
The training program presentation was at a level appropriate for direct care staff.	2.6	0.0	1.3	2.6	18.4	75.0	5.59	0.95
The training program will help direct care staff reduce behaviors associated with mental illness.	2.7	0.0	0.0	6.7	26.7	64.0	5.47	0.96

SD = standard deviation.

TABLE 7.

Descriptive Statistics for Impact on Employee/Employer Relationships and Job Performance and Satisfaction

	Strongly disagree %	Mostly disagree %	Somewhat disagree %	Somewhat agree %	Mostly agree %	Strongly agree %	Mean	SD
If my employer made this training available to all company employees...
I would feel like my company cares about me.	6.4	5.1	3.8	15.4	33.3	35.9	4.72	1.45
I would feel more positive about my company.	6.4	5.1	3.8	12.8	28.2	43.6	4.82	1.48
I would feel a greater commitment to my company.	5.1	5.1	5.1	21.8	32.1	30.8	4.63	1.38
I would be able to provide better care for the residents.	9.1	1.3	2.6	2.6	28.6	55.8	5.08	1.51
I would be more productive at my job.	6.5	5.2	3.9	13.0	29.9	41.6	4.79	1.48
I would feel safer on the job.	5.1	5.1	2.6	11.5	33.3	42.3	4.90	1.39
I would feel more satisfied with my job.	3.8	5.1	5.1	16.7	34.6	34.6	4.77	1.33

SD = standard deviation.

DISCUSSION

This research showed positive effects of a relatively brief behaviorally-focused mental illness training on the Internet. Given that 76% of the study participants were either RNs or LPNs, these findings suggest that the training can be an effective training tool for LTC nursing staff, which has been characterized in the literature as being inadequately trained about mental illness (Glaister & Blair, 2008; Institute of Medicine, 2008; Molinari et al., 2008; Morris & Mentes, 2006). Signifi-cant positive program effects were found on constructs of self-efficacy, knowledge, attitudes, and behavioral intention, which theory suggests are likely to positively influence adoption of new behaviors (Bandura, 1969, 1977; Fishbein, 2000). Further, significant dose response relationships were found for all four of these factors. In sum, this research indicates that the Internet mental illness training was beneficial for participants, most of whom were nursing staff.

The Internet is widely used by college students and for continuing education. Given that LHPs obtain their professional training in college and then with continuing education, these would be logical venues to provide the mental illness training tested here. While Internet research on effects of professional training is still preliminary, a recent meta-analysis indicates that Internet interactivity and feedback can enhance learning outcomes and potentially improve instructional efficacy (Cook, Levinson, & Garside, 2010). Based on the findings reported here, the Caring Skills: Working with Mental Illness program would seem to be a promising educational tool, if only because it is replicable; additionally, textbooks and continuing education courses are not often robustly tested. While the efficacy of Internet training for LTC workers has been documented for NAs (Breen, et al., 2007; Hobday, Savik, Smith, & Gaugler, 2010; Irvine, et al., 2003; Irvine, Billow, Gates, et al., 2011; 2012; Irvine, Billow, Bourgeois, et al., 2012; Irvine, et al., 2007; MacDonald, Stodel, & Casimiro, 2006), the only research we found on web training for LHPs was our pilot study (Irvine, Billow, Bourgeois, et al., 2012), and a stigmatization education program for psychiatrists (Bayar, et al., 2009). Our research provides evidence for 6-week maintenance of effects of a Web mental illness training. Two studies of an Internet training for NAs on dealing with resident aggression showed significant effects after eight-weeks (Irvine, Billow, Gates, et al., 2011; 2012), but we found no other reports on maintenance of Internet training effects for care workers.

Our research offered an opportunity to collect data on stigmatizing beliefs about residents by LHPs in LTCs. In the research reported here, a positive trend at follow-up in empathy, coupled with significant decreases on the Pity, Dangerousness, and Fear stigmatization subscales, suggests that the training helped the LHPs be more understanding and less condescending, and made them feel safer around residents with mental illness. These results support research that suggests that LHPs may benefit from additional sensitization to the mentally ill (Schulze, 2007). Even though our training was not specifically designed to address stigmas, the results suggest that it had positive effects on LHPs stigmatizing beliefs. Having said that, our measures of empathy and stigma did not provide an in-depth assessment of these constructs, nor is it clear what element(s) of the training were influential, so further research is needed to understand these results.

User acceptance of the training program (Tables 5–7) indicates a positive reception, even though it was designed for a relatively less educated audience of NAs. Overall, these results are comparable to program acceptance scores by LHPs in earlier pilot research and by nurses and non-caregiving staff of a nursing home (Irvine, Billow, Bourgeois, et al., 2012 and by nurses and non-caregiving staff of a nursing home (Irvine, Beatty, et al., 2012). The Program Impact ratings (Table 7) are of special interest because they assessed user perceptions of an employer who would provide the training, as well as how it would impact them at work. The results indicate that participants would view an employer positively for providing the training, and that they perceived that the training would be of benefit to their care of residents and to their job satisfaction and performance. These findings support research that suggests that training to improve job skills can increase job satisfaction and job commitment, and it may be an important element of the formula to decrease worker turnover (Cready, Yeatts, Gosdin, & Potts, 2008; Sengupta, Harris-Kojetin, & Ejaz, 2010; Yeatts et al., 2010). Crandall, White, Schuldheis, and Talerico (2007) studied the adoption of culture change by nine LTC facilities. They recommend that training in best practices be institutionalized as a means to sustain positive changes and as demonstration of administrative commitment to culture change. LTC facilities with culture change initiatives underway are realizing benefits in terms of better occupancy rates, staff retention, and a better competitive position in the marketplace (Doty et al., 2008) compared to LTC facilities that are not adopting culture change initiatives.

Limitations

While the research presented here is promising, it has several limitations. First, the training effects were measured over only six weeks, and while a benefit of such an Internet-based study is the recruitment of a national sample of participants, the study relies on self-report assessments only. More rigorous evidence of training effects would be obtained from longer-term follow-up and in vivo behavioral observations.

Second, participants self-identified as having limited mental illness training and lacking good self confidence to deal with resident behaviors associated with mental illness. Thus, the study findings may not be an accurate representation of all LHPs. Further, the majority of the participants were Caucasian nursing staff (i.e., RNs, LPNs), and we cannot know if the training would have similar effects on a more diverse population of LHPs. A larger sample size would allow examination of training effects by degree or license of the participants as well as race/ethnicity

Third, participants were screened to qualify and then randomized into either the Treatment or Control groups; they participated on the Internet without direct contact with the research team. We cannot verify that the screening criteria were fully met.

Finally, this research was constrained to examine the effects of only two fundamentals courses that focused on specific mental disorders. The impact of the entire set of 11 courses is not known.

Implications for Practice

This research demonstrates that a behaviorally-focused Internet-based professional training on how to effectively deal with patient behaviors that are driven by mental illness can be beneficial for LHPs working in LTC facilities. Taken together, the results suggest that the training had a clinically meaningful impact on the participants. This research extends previous reports about the effective use of Internet training and it has implications for educational programs that prepare health professionals for licensure or continuing education.