Supervisor Training

Supervisors are frequently overlooked when impairment-focused interventions are implemented in the workplace. There is a growing body of occupational & environmental medicine literature that points to the need to give supervisors the tools they need to succeed in protecting workers from a variety of factors contributing to impairment.

Abstract

Objective

This study presents a pilot evaluation of a new online training program created to improve supervisors’ ability to protect worker safety by recognizing and responding to signs of impairment from diverse causes.

Methods

The study design relied on a mixed-methods approach with two waves of surveys and interviews immediately after training completion and after 3 months to gauge changes in participants’ perceived knowledge and competencies.

Results

Participants had a positive reaction to training and found it relevant to their job responsibilities and aligned with existing organizational safety programs, practices, and policies. There was statistically significant improvement in perceived impairment knowledge and competencies. The study also provided early indications of training transfer.

Conclusions

The study offers encouraging results that may be used to improve the impairment recognition and response training for supervisors and support future investigations of its impact.

LEARNING OUTCOMES

• Understand preventable factors (eg, drugs, alcohol, prescription and over-the-counter medication, fatigue, stress) that may impair workers’ ability to maintain an adequate level of safety

• Review a mixed-methods approach to evaluating a training program for frontline supervisors

• Discuss the effect of online training on supervisors’ perceived knowledge and competencies in addressing workplace impairment

Worker impairment has a multifactorial etiology with several modifiable contributing factors and serious consequences. The effect of illicit drug use on worker health and safety has been a subject of debate in the literature. In the United States, the rate of workplace deaths due to drug overdose seems to be relatively low but has been growing.¹ Some researchers argue that drug use and misuse represent risk factors for occupational injuries, particularly among young workers, partly because individuals under the influence of drugs are less likely to follow safe work practices.^2–4 Cannabis is by far the most frequently detected substance in workplace drug testing.⁵ As of 2022, 37 US states and the District of Columbia have legalized medical or recreational cannabis, meaning a majority of Americans live and work in a state with some form of legal use.⁶ In 2017, the US Department of Health and Human Services declared the opioid crisis a public health emergency.⁷ Just as this epidemic ravaged the general population, it also spread into the workforce. One study found that 68% of adults who reported past-month prescription pain medication misuse are in the workforce, and those industries with younger, male-dominated workers reported two to three times higher rates of misuse when compared with the nation’s general workforce.⁸ The relationship between alcohol use and occupational injury is equally complicated, but it seems that excessive alcohol consumption increases the risk of injury in the workplace.^9,10 What is certainly alarming is that an estimated 2.3 million US adults drink before going to work, 8.9 million drink during the workday, and 2.1 million work while under the influence of alcohol.¹¹

Beyond the chemical causes of impairment, there is a multitude of other (often correlated) factors present in the work environment. A growing body of literature demonstrates the detrimental effects of fatigue in the workplace. Greater injury risks are shown to be associated with work schedules exceeding 60 h/wk and 12 h/d.¹² Nonstandard work hours and arrangements, including shift and night work, are becoming more prevalent in the modern workforce and have a considerable influence on workers’ productivity, safety, health, and quality of life.^13–15 Similarly, both acute and chronic health conditions (eg, asthma, migraine headaches, diabetes, concussion, allergies) may impair workers’ ability to maintain an adequate level of productivity and safety at work due to the effect those conditions have on one’s physical and mental health status. Impaired functioning has also been associated with stress, job burnout, depression, and other psychosocial factors in various occupations where safety is paramount.^16–18 Goetzel et al¹⁹ (2004) take a detailed look at the employer costs associated with some of the most common physical and mental health conditions affecting workers and conclude that heart disease, mental illness, diabetes, hypertension, and even headaches and allergy have a significant impact on the corporate bottom line but a large portion of the monetary losses are unaccounted for by employers.

Regardless of the cause, there are certain commonalities in the way impairment manifests itself. Changes in appearance, odor, behavior, gait, demeanor, work attendance, and performance can all be indicative of impairment.^20,21 Some signs of impairment can be quite subtle (eg, slower reaction time), whereas others are more obvious (eg, slurred speech). Not all signs and symptoms are immediate; sometimes, there may be a delayed onset. Learning to recognize the signs of impairment can help identify individuals who are in need of attention and assistance to protect their safety and the safety of those around them.

Conventionally, US employers’ safety efforts have been guided by the Occupational Safety and Health Act of 1970 to provide workers with an environment “free from recognized hazards that are causing or are likely to cause death or serious physical harm,” which does not directly address causes of impairment.²² The Drug-Free Workplace Act of 1988 is intended to prevent workplace substance use and misuse among federal employees, grantees, and contractors but does not extend to most private places of employment.²³ Many employers—particularly in safety-critical industry sectors—have implemented substance misuse prevention programs, stressing the importance of organizational policy, drug testing, employee education, employee assistance programs, and supervisor training.²⁴ However, the limited scope of such programs makes them ineffective in addressing causes of impairment unrelated to substance use. Now, there is a growing interest among Environmental, Health and Safety (EHS) professionals and advocates in broadening the traditional approach to addressing impairment in the workplace. Instead of exclusively focusing on the impact of illicit drugs and alcohol, guidelines have been developed to encourage employers to consider the effects of other potential causes of impairment.^21,25

It is noteworthy that in the context of workplace substance misuse prevention, supervisors are seen as a critical stakeholder group. Their role in promoting safety at work has also been widely recognized. In addition to being technical experts and team leaders, supervisors are expected to serve as safety champions by fostering a strong alignment of worker behavior with corporate safety policies and values. Supervisors are the lynchpin of organizational culture as they bridge the divide between the frontlines and the executive office. A recent systematic literature review demonstrates that supervisors can acquire the right knowledge and skills through training to perform their safety responsibilities successfully, and this type of training can improve both supervisor- and worker-level outcomes pertaining to safety knowledge, attitude, communication skills, and behavior.²⁶ Given supervisors’ unique position and ability to affect safety, the question becomes “can supervisors also be trained to deal with the various factors contributing to workplace impairment?”

Course Development and Formative Evaluation

Impairment Recognitions and Response Training for Supervisors (IRRTS) was created with the goal of improving the ability of supervisors to recognize and respond to impairment from diverse causes. Identifying early signs of impairment can ensure workers receive the support they need. The IRRTS defines workplace impairment as the inability to function normally or safely as a result of a number of factors including chemical substances, physical factors, and mental and physical health conditions. Approaching impairment from a more holistic perspective is what differentiates IRRTS from other, more traditional, training programs (eg, reasonable suspicion training). This broader definition of impairment sets the tone for the entire course by emphasizing a compassionate approach, rather than punitive action for misconduct, and switching the focus to worker safety and wellbeing.

A rigorous formative evaluation informed training development, which confirmed before implementation that the target audience found the training concept realistic, appropriate, and acceptable.²⁷ A needs assessment survey of 350 company decision-makers found that 80% of respondents believed that their corporate policies provided adequate coverage of substance use issues, but only 30% had written policies in place for fatigue management. The survey showed that while 77% of companies had some form of impairment training, only 40% of those courses targeted supervisors. Eighty-seven percent of respondents saw a need for supervisor training and, most importantly, 93% agreed with the new, holistic approach to addressing workplace impairment.²⁷ These findings suggest that employers view impairment as a safety priority but comprehensive resources are lacking to assist them in addressing the multifaceted nature of impairment.

An environmental scan conducted as part of the formative evaluation of IRRTS aimed to provide an overview of existing supervisor impairment training programs. The scan included 32 impairment courses targeting supervisors and managers and four additional professional courses identified through literature searches and consultation with a panel of industry and impairment research experts. The literature search resulted in zero peer-reviewed articles about impairment training programs for workplace supervisors. Three of the four nonworkplace courses, including the Drug Recognition Expert, Drug Impairment Training for Educational Professionals, and Advanced Roadside Impaired Driving Enforcement programs, have been scientifically validated via small-scale laboratory and field testing.^28–31

An iterative quality assessment of the identified 32 supervisor courses revealed a need for evidence-based interventions that follow the core principles of adult learning theory and are intended for workplace supervisor audiences. The quality assessment included a ranking exercise that considered each program’s inclusion of knowledge testing, skills application, breadth of topics covered (eg, confidentiality concerns, HR involvement, job accommodation), and evidence of training efficacy. In addition, a panel of industry experts ranked the programs and provided feedback on their strengths and limitations. The most basic finding of the environmental scan suggests that impairment training for supervisors is a viable solution but no suitable interventions currently exist, especially those that address the risk of impairment from multiple factors.²⁷

The IRRTS was developed as a self-paced eLearning program with the following learning objectives: (1) understand the relationship between impairment and workplace safety; (2) identify multiple causes of impairment; (3) recognize common signs and symptoms of impairment; and (4) identify appropriate actions to take upon observing potential impairment in the workplace. Initial testing indicated that IRRTS would take approximately 1 hour to complete. The course curriculum includes interactive features such as knowledge checks, scenarios on various forms of impairment within different industry settings, a knowledge-assessment test, and downloadable tools and resources for supervisors. Some of the major themes explored in the course include supervisor responsibilities, common causes and signs of impairment, responding to potential cases of impairment, and administrative and legal considerations. The course content focuses on commonly observed signs and symptoms of impairment broadly and breaks them into physical, cognitive, and performance indicators.

To coach supervisors on responding to suspected impairment, IRRTS establishes a step-by-step process to enable appropriate response: starting from identifying workers in safety-sensitive situations and proceeding through information gathering and incident or behavior documentation, through to creating an action plan including follow-up, where the impacted workers’ well-being, performance, and safety metrics are assessed. The majority of the practice scenarios integrated into the course center on applying these steps so that trainees can rely on a specific and actionable process that can fit within their existing workplace impairment policy framework.

The desired short-term outcomes of this training include improved knowledge and competence on impairment recognition and response as well as increased self-efficacy in addressing impairment. The desired long-term outcomes are increased suspicion and incident reporting, supervisory intervention, worker support during recovery and return to work, and team-based response to impairment. It is expected that the most impactful results will occur when IRRTS is implemented within a whole organization although it can also benefit individual supervisors.

The findings of the formative evaluation stage suggest that the inadequate scientific evidence base for impairment training provides little guidance on preferred outcome measures. The broader field of education and training evaluation, however, has come to rely greatly on Kirkpatrick’s model of training evaluation³² and typically focuses on measuring either perceived (ie, self-reported) or actual knowledge gains (ie, ascertained via tests and examinations). In this context, perceived knowledge refers to learners’ own appraisal of their growth and development as opposed to objectively measured learning. It is notable that perceived and actual measures of learning do not show a great deal of correlation and may even pertain to two different learning concepts.^33,34 Some researchers maintain that perceived knowledge plays a central role in metacognition, defined as leaners’ ability to understand, assess, and manage their own knowledge.³⁵ Metacognitive thinking encompasses such important aspects of learning as awareness of one’s own strengths and weaknesses, strategic thinking, problem solving, planning, applying knowledge in practice, and self-evaluation.³⁶ Several training evaluations performed in workplace settings have considered outcomes related to perceived changes in knowledge, competence, confidence, and skill acquisition.^37–40

Research Objectives

This pilot study aimed to measure the short-term effect of the IRRTS course on supervisors’ perceived knowledge and competencies related to impairment and to determine what individual- or organizational-level factors may affect their self-appraisals. The study also aimed to explore potential barriers and enabling factors that may affect the transfer of training to everyday work tasks of supervisors.

MATERIALS AND METHODS

Participants were recruited through advertising in professional safety publications and social media channels. Individuals were eligible to participate if they were able to communicate in the English language, resided in North America, had regular face-to-face contact with coworkers, had supervisory responsibilities over direct or indirect reports, and had Internet access on a computer or mobile device to participate in training and follow-up data collection.

The study received approval from the institutional review board of the host institution. Informed consent was obtained from all research participants before the study.

A version of the Kirkpatrick’s model of training evaluation adapted for use in workplace settings guided this study.²⁶ Outcomes were assessed at the following levels:

1. Participant’s reaction to training—supervisors’ opinion of their training experience, including satisfaction, usability, and relevance in relation to the training program’s delivery, materials, instructor, and format;

2. Change in knowledge or skills—acquisition of new or maintenance of existing job-related concepts, procedures, principles, competencies, or skills that pertain to workplace safety among supervisors, or changes in confidence to apply newly acquired knowledge or skills on the job; and

3. Change in behavior—changes in supervisors’ behavior in the workplace that occur as a result of training, including supervisors’ willingness, readiness, likelihood, or intentions to change behavior.

Immediately after training completion, participants were prompted to complete an online survey developed using the Qualtrics survey development platform (Qualtrics, Provo, UT). The survey included seven Likert-style statements pertaining to participant reaction to training (eg, Overall, this training met my expectations). Also included were two sets of seven questions each designed to gauge participants’ perceptions of current impairment knowledge and competency (eg, Based on what you have learned in this course, how would you rate your knowledge of the following… symptoms of impairment.) Then, they were asked to think back and provide ratings of perceived knowledge and competency in the same impairment areas before taking IRRTS (eg, Thinking back, how would you rate your knowledge prior to taking this course on the following… symptoms of impairment?). In other words, both pretest and posttest data on perceived knowledge and competency were collected at the same point. This so-called retrospective pretest design is known as a valid and convenient evaluative approach to measuring self-reported change. As Hill and Betz (2005) point out, “if the aim is to understand how participants feel about program effectiveness and their personal growth”—as opposed to precise measurement of program effect—“the retrospective test provides a more direct assessment of these factors” compared with the traditional pretest-then-posttest design.⁴¹ Compared with the traditional pretest-then-posttest design, retrospective pretests have been shown to be more appropriate to use in situations where response-shift bias may pose a threat to validity.⁴¹ Retrospective pretests based on self-reporting are not a perfect tool and can produce biased estimates of program effectiveness. Nonetheless, there is some evidence to suggest that retrospective pretests have a stronger correlation with objectively measured behavioral changes than simple pretests.^42,43 Retrospective pretest designs have been used in research on workplace safety training for frontline supervisors.^44,45

Once participants completed the survey, they were sent an email invitation to schedule a 30-minute interview. Two of the authors held the interviews via Zoom (Zoom Video Communications, version 5.11.3, San Jose, CA) within 7 days after training completion, using a semistructured interview guide to gather more in-depth information about participants’ experience with the training.

After a 3-month period, participants were asked to complete another online survey. This follow-up survey included eight Likert-type scale questions designed to gauge the experience of participants with applying the training content in the workplace. Upon survey completion, participants were contacted again via email to schedule another 15-minute Zoom interview. The second set of interviews was intended to gather more in-depth information about participants’ experience with applying the training content in their daily work responsibilities. All study participants who completed the study received a $50 Amazon gift card.

By combining quantitative survey data with qualitative interview data, the study relied on a mixed-methods approach to obtain a better understanding of the supervisor experience as well as the workplace context, in which the IRRTS course would eventually be deployed. Mixed-methods studies have become increasingly common across many disciplines, including occupational safety and health, in part because they help program evaluators simultaneously consider outcome measurement and intervention improvement perspectives.^46–48

The study analytics included descriptive and inferential statistical procedures. The univariate analyses included simple frequency distribution, percentage, and mean value calculations. Nonparametric Spearman correlation analyses were used with ordinal data and paired t tests to measure perceived knowledge and competency changes in support of the abovementioned retrospective pretest approach. A series of repeated-measures analyses of variance (ANOVA) tests determined whether there were differences among study participants based on their demographic or company characteristics. Statistical analyses were performed in SPSS (IBM SPSS Statistics for Windows, version 22.0, Armonk, NY).

Three research experts with backgrounds in impairment and occupational safety reviewed the questionnaire for content and face validity. Principal components factor analyses were conducted with Varimax rotation of the retrospective pretest items for knowledge and competency. The Kaiser-Meyer-Olkin Measures of Sampling Adequacy suggested that the sample size was sufficient relative to the number of items in the scales (0.870 and 0.817). Bartlett tests of sphericity indicated that the use of factor analysis was justified (P < 0.001). The correlations among the individual items were strong enough (>0.80) to suggest that the correlation matrices were factorable based on the measures of sampling adequacy. Visual inspection of the scree plot indicated a single factor, accounting for 84% of the variance for both knowledge and competency. The Cronbach αs of the perceived knowledge and competency scales were 0.89 and 0.87, respectively, suggesting adequate internal consistency reliability. The knowledge and competency items were averaged to create composite scales.

The interviews with participants were audio recorded in Zoom, which automatically generated verbatim transcripts of the interviews. The transcripts were edited for clarity and imported into the NVivo software (QSR International, version 10, Burlington, MA) for qualitative analysis. A codebook was developed by researcher-generated codes, followed by in vivo coding to categorize and generate thematic analysis findings. Transcribed data were independently coded by two of the authors with extensive experience with qualitative data analysis, resolving disagreements by consensus. Direct quotations from the interviews were included to reinforce the emerging themes.

RESULTS

Posttraining Survey—Quantitative Analysis

Sample Description

The sample size included 22 managers and supervisors who volunteered to take part in the study and complete the IRRTS course. Of those, 18 (81.8% response rate) completed the course satisfaction survey, 19 (86.4%) completed the first follow-up survey, 22 (100%) completed the first telephone interview, 16 (72.7%) completed the 3-month survey, and 15 (68.2%) completed the second interview.

Of the 22 individuals who volunteered to participate in the study, 10 (45.5%) were female. Half of the study sample (n = 11) reported having EHS responsibilities, while others were employed in other roles (eg, operations, procurement, HR). The sample was almost equally split between midlevel managers (12, 54.5%) and frontline supervisors (10, 45.5%). Fifty percent of the sample had between 1 and 10 direct reports, while three participants (14%) had no employees directly reporting to them.

Company size ranged from 10 employees to 50,200 (median company size of 707 employees). The sample included a mix of industries: manufacturing (36%), construction (18%), public administration and utilities (9% each), educational services, health care and social assistance, and waste management and remediation (5% each).

Participant Reaction

Overall, posttraining survey findings indicate that participants had a strongly positive reaction to the training (Table 1). A bivariate correlation analyses revealed some salient differences in the reaction to the course. Specifically, participants employed by smaller companies were more likely to report that the training improved their knowledge of impairment (Spearman correlation coefficient, 0.62; P = 0.006) and the training would help them keep themselves and others safe on the job (Spearman correlation coefficient, 0.57; P = 0.013). The number of direct reports did not seem to be correlated with course satisfaction and neither did position (ie, midlevel managers vs frontline supervisors), area of responsibility (ie, EHS vs other), or gender.

TABLE 1.

Participant Reaction to IRRTS Immediately After Training Completion*

	Mean (SD)
Learning environment was of high quality	4.2 (0.81)
Training content was applicable to me	4.2 (1.00)
Training will help me keep myself and others safe	4.2 (0.99)
Training improved my knowledge	4.2 (1.10)
I would recommend training to others	4.1 (1.13)
Training met my expectations	4.1 (0.96)
Course material presentation was engaging	4.0 (1.14)

*Responses on a 5-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree).

Self-Efficacy

Of the 19 individuals who completed the first follow-up survey, 10 (52.6%) said that they were extremely confident in their ability to apply what they had learned in the IRRTS course back on the job, eight (42.1%) were very confident, and one (5.3%) was somewhat confident.

Knowledge and Competency Gain

Changes in perceived impairment knowledge and competencies were investigated using a series of paired-sample t-tests. Statistically significant knowledge gains and competency improvements were found across all items included in the survey. The results of the paired t-tests are shown in Table 2.

TABLE 2.

Perceived Changes in Knowledge and Competencies Assessed Through a Retrospective Pretest Method

	Before Training, Mean (SD)	After Training, Mean (SD)	t (df)a
Knowledgeb
Expanded definition of imp.	6.95 (1.65)	8.79 (0.98)	5.49 (18)*
Multiple causes of imp.	7.00 (1.67)	8.79 (0.98)	5.04 (18)*
Symptoms of imp.	7.21 (1.58)	8.53 (1.26)	4.59 (18)*
Consequences of imp.	7.79 (1.55)	8.68 (1.20)	3.39 (18)*
Supervisors’ role in addressing imp.	7.53 (1.71)	8.74 (1.20)	4.46 (18)*
Steps supervisors can take to respond to imp.	7.26 (1.20)	8.95 (0.78)	7.76 (18)*
Steps others can take to respond to imp.	7.16 (1.46)	8.79 (0.92)	5.87 (18)*
Competenciesc
Identifying potentially impaired employees	7.42 (1.39)	8.63 (1.12)	4.30 (18)*
Gathering information when you suspect imp.	7.37 (1.42)	8.74 (0.93)	4.75 (18)*
Documenting cases of potential imp.	7.26 (1.41)	8.89 (0.94)	6.68 (18)*
Preparing to approach a potentially impaired employee	7.16 (1.61)	8.68 (1.00)	4.93 (18)*
Talking with a potentially impaired employee	7.16 (1.43)	8.79 (0.86)	5.87 (18)*
Following up with employee after a case of imp.	7.37 (1.21)	8.84 (0.77)	6.66 (18)*
Following your company policies on imp.	7.05 (1.90)	8.47 (1.22)	3.93 (18)*

*Statistically significant at P < 0.005.

^aResults of paired t tests.

^bResponses on a 10-point scale from not at all knowledgeable (1) to extremely knowledgeable (10).

^cResponses on a 10-point scale from cannot demonstrate (1) to can demonstrate effectively (10).

df, degrees of freedom; Imp, impairment; t, t statistics.

Eight repeated-measures ANOVAs were performed to examine how training participants’ gender, position, area of responsibility, and company size affected the before-after differences in the composite knowledge and competency measures. The before and after scores were entered as within-subjects factors, while gender, position, area of responsibility, and company size were entered as between-subject factors. The ANOVA tests for changes in knowledge and competency did not show statistically significant variation by the aforementioned variables.

Posttraining Interviews—Qualitative Analysis

In parallel to the survey findings, the telephone interviews provided additional information about the IRRTS course. Qualitative analysis of the participants’ comments yielded four major themes.

Theme 1—Generally Positive Reaction to Training

Echoing the survey results, there was an overwhelmingly positive response to training among participants. Virtually all participants said that they had gained new and useful information that would be relevant to their supervisory responsibilities in the workplace.

Training gave me the tools to be more active and do better observations and be more confident with that.

Participants shared a strong belief that impairment is a serious concern affecting both worker performance and safety, and the situation is further exacerbated by larger societal factors such as the growing trend of cannabis legalization in the United States and post–COVID-19 labor shortages.

We’ve had impairment cases, both from an alcohol standpoint, a prescription drug standpoint, and then from a narcotic standpoint all within the past 4 months.

Impairment is not getting any better as we all know with the loosening of the cannabis laws. It’s not an easy road nowadays for supervisors because certain things are legal and certain things are not.

Our company employees are dealing with a lot of fatigue issues right now. We’re having a really hard time finding employees. I can see that many of our people are reporting minor injuries because, I suspect, they are just exhausted.

In general, participants viewed impairment recognition training as appropriate for supervisors and a needed extra step that companies could take to be more focused on reducing the adverse consequences of impairment. Some acknowledged that the more traditional approaches to addressing impairment in the workplace (eg, pre-employment drug screening, drug-free workplace policies, employee assistance programs, and instructor-led drug and alcohol awareness training) have not produced the desired outcomes.

I thought the training was a good springboard, a good starting point for myself and my team.

We typically struggle to find good-quality reasonable suspicion training for our employees… and doing the standard five-panel drug screening for employment has not been effective.

Our mandatory training is a classroom-based, instructor-led course offered through the EAP provider. It spends a great deal of time on the different effects of drugs, medications, and alcohol. It gets to a point where I start feeling overwhelmed.

These findings validate the need for impairment training and the role supervisors can play in addressing impairment.

Theme 2—Beneficial Effect of IRRTS

Virtually all participants reported having learned something practical that they can take back to their companies. The emphasis on impairment as an issue encompassing three key aspects (ie, physical, cognitive, and performance) particularly struck a chord with many participating supervisors. Participants noted that training supervisors on impairment recognition and response techniques offers a considerable opportunity to prevent safety incidents in the workplace.

The course prompted me to bring the whole aspect of situational awareness back into my life vision. There are times when we get so caught up in the day-to-day activities that we miss those subtle indicators that something’s just not right around us.

I’m glad the course focused on safety sensitive positions—that is new to our company. The scenario about a forklift operator… to explain what “safety sensitive” actually means was useful.

I have not really thought about impairment until now. I have been on job sites before where there was some impairment—I did not realize it back then, but it makes perfect sense now.

Consistent with the IRRTS learning objectives, interviewees conveyed that the training had changed their orientation toward impairment as a concern that deals not only with illicit drugs and alcohol but also with worker fatigue, psychological well-being, and physical illness. This specific point was a key lesson learned point for many of the interviewees.

I really liked the expanded definition because when people think of impairment, they immediately think of some substance. But there are so many other factors and I just never really correlated to impairment.

I really liked how you started off by busting the myth that impairment is just drugs… that was my first “aha” moment.

We have a substance abuse department that deals with impairment issues. However, as a company, we fail to do the due diligence to deal with the mental health aspects of impairment.

What if he was not under the influence? Maybe it was something else? Maybe it was sleep deprivation, exhaustion… like working in the sun all day can impair your ability to function normally. This course can help with overcoming that fear of falsely accusing somebody.

Participants presented a range of views indicative of their increased comfort level and willingness to intervene in workplace situations involving a potentially impaired colleague, and doing so in a constructive way that does not automatically assign blame for misbehavior. The following quotes illustrate that this was one of the main points for many interview participants.

I typically have not overly involved myself in impairment assessments… if I look at the application of the course, that will be the most useful element for me—knowing how you can as a supervisor use these tools to safely have this discussion with somebody who may be impaired.

What I really enjoyed about the course was instead of yelling at somebody or getting pissed and having an argument on the job site, you work through the problem as a team and figure the problem out and then move forward so that the people can ultimately continue doing their job.

Supervisors need help verbally expressing exactly what they observe. I appreciate that the course teaches to avoid saying things like “You’re drunk” and instead focuses on communicating the behaviors and signs actually observed by the supervisor.

Theme 3—Good Fit With Existing Organizational Programs and Practices

One of the themes emerging from the interviews dealt with participants’ assessment of the fit between what their companies already had in place to address impairment and the IRRTS course. Specifically, participants noted ample parallels the IRRTS course had with other online educational opportunities to which they had been exposed, company programs dealing with impairment, and their current supervisory responsibilities.

I could see the parallels between investigating impairment-related incidents and what I typically do with safety incident investigations and follow-up.

It would be helpful to have a training resource for people out in the field. This course could be a requirement for new hires or newly promoted foremen in additional to an annual requirement.

… online training does have its place and a quick online refresher course—in addition to the classroom-based mandatory course that our company already offers—might help with information retention.

Theme 4—Barriers to Implementing IRRTS

Participants saw a number of factors potentially preventing their companies from making the IRRTS course broadly available to supervisors. Several of them pointed out a lack of clear guidance in corporate policy statements pertaining to the supervisor’s role in investigating cases of impairment, which could be a rather common occurrence in corporate settings.

In general, our company does not have a policy on impairment. We have a policy that talks about impairment while driving on company time, of course.

Our safety manual was written many years ago so we update it as we come across new issues. Impairment is one section that we could definitely update.

In addition, a small number of participants lamented that completing an hour-long training session online may not be a feasible option for their companies, an issue that particularly affects supervisors at remote company locations and older workers. They expressed their challenges as follows:

At our company, we have very seasoned people who were not brought up with computers.

We’ve got two foremen—one of them is not necessarily a computer guy. He′ll be the one to struggle with it.

Time is money and a one-hour course is a considerable undertaking. For people who are out in the field—like our guys in construction—it would need to be no more than 20 minutes.

Not surprisingly, one of the greatest challenges participants saw in implementing this kind of training was striking the right balance between safety training and production-related demands for frontline supervisors because of the time commitment required to complete the training and, then, put it into action.

To be a front line supervisor you have to follow the rules and regulations and still hit the numbers and everything else that management wants you to do. It’s not an easy job anymore, let alone the shortage of the workforce that we’re having right now.

As we all know, we’re always under the gun to get the production out, hit the numbers, and all that kind of stuff.

Three-Month Follow-up Survey—Quantitative Analysis

The follow-up survey conducted 3 months after training completion revealed that participants were successful with real-life applications of the training lessons learned. Seven participants (43.8%) said that they were able to apply what they had learned immediately after the training and six (37.5%) said that it took them 1 to 3 months to start applying what they had learned. Examples of follow-up action reported by the participants in the survey included discussing company policy on impairment with their colleagues (50.0%), working with other departments on suspected cases of impairment among workers (37.5%), identifying potential cases of impairment (25.0%), identifying employees in safety-sensitive positions (25.0%), and talking with an employee after a potential impairment incident (25.0%).

Three participants (18.8%) answered that they had not been able to use any of the knowledge or competencies acquired in training back on the job. They were unanimous that they simply did not have the opportunity to do so since training completion.

Figure 1 illustrates responses to a set of questions asked on the 3-month follow-up survey to gauge participants’ views of the training and training transfer.

FIGURE 1.

Three-month survey results, N = 16. ¹Includes strongly agree and agree responses. ²Includes strongly disagree and disagree responses.

FOLLOW-UP INTERVIEWS—QUALITATIVE ANALYSIS

The second stage of interviews provided an opportunity for training participants to share more in-depth information about their experience, which produced further insights into their views toward the training beyond that obtained from the survey. Three themes emerged from the interviews as outlined hereinafter.

Theme 1—Training Transfer

Participants’ responses to the question “Have you been able to apply what you learned from the course in your job?” can be grouped into two subcategories—observing cases of impairment and barriers to training transfer. Of the 15 supervisors who completed the follow-up interviews, three said that there had been opportunities for them to witness someone exhibiting signs consistent with the taught definition of impairment over the 3 months since training completion. Of those three, two were able to intervene using the tools and resources provided in training, whereas one participant felt uncomfortable doing so.

One of our staff came to work drunk, got in a car, and totaled it. Since I had not conducted an investigation like that before, the step-by-step process that was provided in training—even though to be honest with you, I thought I wasn’t going to need it—proved very, very helpful.

An individual, who overdosed on prescription opioids, was running the forklift and someone reported erratic behavior. We were able to quickly assess that situation and make sure they didn’t present a risk to themselves or others.

I did have a very specific example here at the office within the last month. The person was a field employee who currently does not report to me and I have very little interaction with him. I could not approach the employee myself and say something to him. I went straight to the top of the organization. We are a very small company, and it is more effective coming from the president, rather than a supervisor.

Some interviewees made a point of acknowledging that observable cases of impairment are extremely rare in their workforce, or they may not have had many chances to be directly involved in impairment incident investigations, which presents a barrier to transferring training to everyday practice. Other barriers mentioned were supervisors’ lack of time or competing priorities. For example:

I have been lucky that the subcontractor’s people show up with great attitude on the job. It has been great but, unfortunately, I have had no real chance to use the course materials.

Fortunately, we rarely see employees exhibiting signs of impairment.

We have paused all things new besides COVID. Can’t continue impairment conversation until COVID issues are under control.

It has been somewhat difficult because a large portion of our workforce is still remote.

Theme 2—Support From Others

During the follow-up telephone interviews, several participants reported that they had shared information and materials included in the course with important others, including their immediate supervisor, senior management, human resources, and environmental health and safety staff. The main reason for this posttraining behavior can be described as seeking support, buy-in, and approval from those in the position of power to use the newly acquired knowledge and skills in impairment recognition to affect real change in the workplace. However, some participants seemed pensive, perhaps even skeptical, about their ability to influence others.

I have discussed [the training] with my directors and other managers, and I definitely think it is something that our company needs to address. I have discussed impairment in general, how we deal with this issue and what we should do to improve in the future. But I don’t know exactly what follow-up action we are going to take.

During my last 2–3 safety meetings, I have used the course materials on impairment to give an overview presentation … This particular issue is very difficult for me because it is a touchy subject. I also followed up with my company’s EHS/HR director after I took the course.

I have been trying to explain to my company that fatigue is a form of impairment, and this lit a bunch of eyes up because they never thought of being overworked and tired as a form of impairment.

Many participants commented that they had received a great deal of support from others within their companies to continue focusing on impairment prevention in an effort to improve worker safety. In most cases, participants reported positive reaction; for other companies, however, the reception was less than encouraging as illustrated hereinafter:

I have discussed the impairment training with my colleagues and there was a lot of curiosity and interest, enough to encourage us to review the prescription medication reporting procedures.

I reached out to the specialist that leads the disability management group in our organization to share my thoughts about the training. I think they should consider how it would apply to our actual work duties. Their initial response was positive.

When I started talking about this training, I immediately became a pariah. That just tells me that [the training] is needed.

In terms of corporate policy implications, many participants agreed that the training had encouraged them to work collaboratively on exploring how their companies’ policies and procedures approach worker impairment and what role people managers play in addressing it. For nearly everyone, this was a significant next step to take. These are some of the thoughts shared during the interviews:

The training has also given me an opportunity to review some of our procedures for reporting prescription medication use…. In the review of our procedures, I found out that it wasn’t very clear how medication use is supposed to be reported and recorded, so we have started an internal effort to look at that.

I incorporated some of the course materials into the regular training that we do on reasonable suspicion.

One of the items that was on my list of improvements was our drug and alcohol policy. We’re in the final stages [of updating the policy]. We have it with our attorney for legal review right now, so we’ve gotten pretty far with it.

Theme 3—Areas for Improvement

While the general reaction toward IRRTS was positive, some interviewees stressed the importance of additional content, features, or improvements that could make the course more effective and relevant to supervisor audiences. Specifically, some participants expressed a desire for continual training, more focus on employee confidentially during incident investigations, limitations of online training delivery, additional hard-copy materials, and improved access to training updates and modifications. Examples of this are described below:

We are a smaller company where everybody knows everybody. When I think about gathering incident information, protecting confidentiality is one of the hardest things that I have to focus on.

Personally, I am not a big fan of online training because, I think, you lose something with online training.

If I could recommend one thing, it would be adding a couple of videos to show some scenarios of potential impairment.

It would be good to have some type of website to go to where you can get updated information. Because these things are always changing, especially with cannabis laws. There’s nothing worse than doing a training and seeing a statistic from 5 years ago.

DISCUSSION

In this study, a rigorous multistep, multistakeholder evaluation was undertaken of a newly developed training program for supervisors, who can be viewed as the first line of defense, so to speak, in identifying and responding to incidents where impairment may jeopardize workplace safety. Evaluation data were collected via surveys and personal interviews with training participants immediately after training and, then, 3 months later. The study results suggest that witnessing signs of physical, cognitive, or performance impairment among workers is not uncommon, and supervisors are keenly aware of the collective impact substance use, physical illness, fatigue, psychological distress, poor work-life balance, and other factors can have on members of their work crews. There is a need for educational programs that define impairment from a broader perspective (ie, beyond the effects of substance use and misuse) and emphasize appropriate and coordinated response, rather than punitive measures. The IRRTS course explored in this pilot study is one example of such programs.

It is important to acknowledge that IRRTS does little to tackle underlying causes of impairment but it works to detect signs of impairment early enough to enable an effective organizational response. Perhaps, its value should be seen in contributing to an integrated approach combining occupational safety with health promotion based on a combination of primary, secondary, and tertiary methods of impairment prevention.⁴⁹

The primary objectives of the study were to pilot test the IRRTS course and to explore ways to enhance training content and delivery. Borrowing training evaluation outcomes suggested by Kirkpatrick (1996),³² it was found that the training was well received by supervisors and instrumental in promoting awareness. It was particularly encouraging to see that study participants found the training relevant to their needs and were willing to recommend it to others. The study demonstrated positive changes in supervisors’ perceived knowledge and competencies in addressing impairment despite the very high scores on the retrospective pretest, which left little room for improvement. Statistically significant improvements were observed across learning outcomes as a result of exposure to various training components (eg, verbal instructions, case study presentations, downloadable tools). The high level of self-efficacy reported posttraining completion will help supervisors apply the lessons learned in everyday work activities. The results indicate that the IRRTS course inspired learning and self-reflection.

These results suggest that even a low-intensity online training program can be effective in achieving positive short-term changes, which is expected given the body of literature that, albeit small, has consistently demonstrated the potential of supervisory training interventions aimed at safety improvement. A previously published review of the literature on workplace safety training for supervisors identified 22 studies, of which 10 measured changes in attitudes and perceptions and changes in knowledge and skills as advised by Kirkpatrick’s model of training evaluation (see level 2 outcomes).²⁶ Consistent with the present findings, these studies find that supervisors find safety training important and relevant to their regular job responsibilities. Notably, one of those studies found positive changes in attitudes and perceived competencies among supervisors trained on proper response to worker injuries using a retrospective pretest.⁴⁵

Although the sample used in the study was relatively small, it did include both female and male supervisors representing a variety of industries, company sizes, positions, and job functions, suggesting that the findings may be generalizable beyond the study sample. No differences were found in perceived knowledge or competency gains across gender, position, area of responsibility, and company size, which could speak to the potential applicability of the training content to a variety of supervisor populations. However, it is quite possible that the small sample size used in the study did not allow to detect differences between these variables. Studies with small sample sizes are more prone to type II error. Future IRRTS studies would indeed benefit from a larger sample size and have a higher likelihood of detecting differences across various participant groups.

The study brought to light several enabling factors that might help with training delivery and several challenges in relation to the ability of supervisors to apply the IRRTS materials on the job. It also highlighted some opportunities for future IRRTS applications that might be useful to explore.

In terms of enabling factors, most participants seemed comfortable with the online delivery format and were largely in agreement that the virtual learning environment and course materials were conducive to learning. There were no technical issues reported with completing the training online. In fact, many participants pointed out that the flexibility of online training delivery, ability to control the training pace and pause, if necessary, and the ability to switch attention to other, more pressing work tasks helped them compete the training successfully. This finding is in line with previously published research, pointing to user autonomy, convenience, content adaptability, low cost, and engagement as important advantages of eLearning in the field of safety.^50,51

Another central finding from the study is that IRRTS seems to have potential to fit in well with existing policies, programs, and practices applicable to supervisors. Based on prior research, it is noteworthy to mention that training characteristics, including providing training content that is relevant and consistent with trainees’ job requirements, is a key factor affecting training transfer.^52,53 Supervisors who participated in the present study found themselves able to connect the IRRTS content to their personal workplace experience. Several admitted that they had been directly involved in dealing with difficult situations where additional guidance on how to assist impaired crewmembers would have been helpful. Thus, being able to recognize and respond to cases of impairment can be seen as a good fit for supervisory safety responsibilities. Furthermore, IRRTS did not seem to contradict, and may have even reinforced, the messages and guidance provided to participants in previous workplace training opportunities. All these factors combined may help improve the rate of adoption and future impact of the IRRTS course.

A widely cited study by Baldwin and Ford⁵⁴ (1988) draws attention to the impact of work environment on training transfer and, more specifically, to the role of management support as an important factor in the work environment affecting how skills are transferred from the classroom to the job. Cromwell and Kolb⁵⁵ (2004) demonstrated that higher levels of support from management facilitated transfer of training among participants of a supervisory skill-training course, a finding particularly germane to the present investigation because many IRRTS participants attempted to seek support, guidance, and approval from their own managers after training completion. This finding also brings to mind what Holton et al⁵³ (2000) describes as a dual-path mechanism, through which senior managers can influence the transfer of training among supervisors by acting as gatekeepers (eg, setting rules and expectations, creating opportunities for training transfer) but also as a source of reward and recognition. This is undoubtedly relevant to the ability of supervisors to apply impairment recognition and response skills on the job. Logically, then it follows that supervisors would be unlikely to intervene in real-life impairment situations without the say-so of their managers.

Seeing training participants proactively reaching out to their important others (eg, EHS, human resources, legal counsel) was a somewhat unexpected finding, considering IRRTS did not include an explicit call to action with regard to this. It highlights the importance of supervisors’ position as the bridge between the frontlines and organizational decision-makers, but it may also suggest a possible barrier to training transfer. Rowlinson et al⁵⁶ (2003) note that supervisors may lack proper authority to perform safety assignments even if they are aware of their safety responsibilities, possess the right knowledge and attitude toward safety, are willing to take on additional safety tasks, and possess a sufficient level of self-efficacy. From this perspective, it is also not surprising that many participants took the initiative of finding out what their corporate policies say about supervisors’ role as it pertains to worker impairment in follow-up to training. Corporate policy can establish a plan of action and delineate responsibilities of frontline supervisors, midlevel managers, and top executives for dealing with potentially impaired workers, whereas not having clear policy guidelines, or not knowing what the policy says about dealing with impaired workers, can further hinder supervisors’ ability and willingness to exercise their authority. Thus, a company-wide impairment policy tied to EHS management principles may be essential to effective implementation of IRRTS.

The task of applying theoretical knowledge in day-to-day practice may also be facilitated by having opportunities to do so. A brief overview of this topic by Burke and Hutchins⁵⁷ (2007) indicates that not having sufficient opportunities to apply newly acquired knowledge and skills is a major barrier to training transfer in workplace settings. By the conclusion of the present study, some supervisors had not observed any cases of impairment and, thus, were unable to intervene. Presumably, the most obvious explanation has something to do with the length of the follow-up period (3 months), but other contributing factors also may have been involved such as company size (some companies in the study were as small as 10 employees) and work arrangements (study took place during the COVID pandemic of 2020–2022 when remote work was common and supervisors’ face-to-face interaction with their crewmembers may have been limited). Under certain circumstances when cases of impairment are infrequent, supervisors will be challenged to maintain their competence and confidence in impairment recognition. In this study, some participants were amenable to taking a refresher course on impairment recognition, and there is some evidence that repeated safety training is associated with greater training transfer.⁵⁸

The study findings should be interpreted with a number of methodological limitations in mind. We used a nonprobability sampling approach to recruit participants into the study, which potentially introduced some measure of selection bias and, thus, limited external validity. The high pretest scores might suggest that supervisors who took part in the study were already quite conscious of the workplace impairment issues, providing further support to the selection bias concern. In general, self-reporting is prone to cognitive biases.⁵⁹ There are validity concerns with retrospective pretest data collection as research participants have to rely on recall of their past cognitive states, which may lead to either upward or downward estimation of change.⁶⁰ However, self-reports seemed appropriate for this evaluation of perceived knowledge changes occurring as a result of a short-duration intervention. The retrospective pretest design is prone to a number of other important threats to validity—social desirability bias (presenting one’s self in a more favorable light), subject acquiescence (tendency to agree with survey statements), effort justification (overstating self-improvement to justify the time and effort spent on program participation), cognitive dissonance (resolving internal conflict by overstating improvement to meet one’s own expectations for change), and implicit theory of change (overstating improvement due to an intervention assuming it had the desired effect).⁴¹ The possibility of biased self-assessment scores either on the retrospective pretest cannot be completely ruled out. The decision to use a retrospective pretest was driven by the desire to have an easy-to-administer method that would limit participants’ burden associated with study participation, reduce incomplete survey data, and reduce response-shift bias (changes in measurement standard before and after an intervention). For a training program intended to redefine impairment for supervisors, it seemed particularly important to use an alternative to the traditional pretest capable of minimizing the response-shift bias problem. In our view, the advantages of a retrospective pretest seemed to outweigh the disadvantages in this particular case.

Taking the abovementioned limitations into account, this study makes an important contribution as the first step on the path toward creating a much needed solution for a serious workplace safety concern. The study findings and the IRRTS course itself are relevant to a variety of workplace settings where supervisors interact with their crewmembers individually or in groups and are in a position to intervene directly when signs of impairment are observed and safety is at risk. This could include a wide variety of workplaces such as manufacturing facilities, warehouses, offices, garages and auto repair shops, construction sites, restaurants and school cafeterias, hospitals and clinics, detention centers, among several others. The supervisors’ viewpoints presented in this study provide invaluable insights for future investigations that are needed, as many important questions about the program remain. Recruiting for a larger, intervention-control study is currently underway.