Evaluation of safety management and leadership training using mobile technologies among logging supervisors

Abstract

Background:

Logging is recognized as one of the most dangerous industries in the United States (US), ranking among those with the highest occupational injury and fatality rates. Although logging operations in the Southeastern US have lower rates of injuries and fatalities compared to other regions of the US, due in part to the use of large machinery to fell timber as opposed to chainsaw felling, safety hazards continue to persist. The hazards present in the logging cut sites in which loggers operate may result in worker injury, illness, or fatality. Our objective was to develop, deliver, and evaluate a safety management and leadership training among logging contractors and supervisors using mobile tablets as a personal learning environment.

Methods:

A safety leadership and management training vignette was developed based on previously collected focus group needs assessment data. A non-random sample of 31 male logging supervisors received the safety leadership and management training on a mobile tablet. Kirkpatrick Levels 1, 2, and 3 training effectiveness evaluations were performed.

Results:

A statistically significant large effect size suggests safety knowledge was gained among training participants when comparing post-test scores to pre-test scores (Level-2). Participants rated their training experience favorably (Level-1), and applied knowledge gained from the training throughout their weekly work activities three months after training (Level-3).

Conclusion:

Our findings suggest the utilization of mobile learning techniques can be an effective means to deliver safety management and leadership training content to logging contractors and supervisors. Future trainings should be linguistically and literacy-level appropriate, as well as comprehensive in nature, including meaningful and relevant content. Our observations support the use of mobile devices as just one component of a more comprehensive health and safety management program for workers in the logging industry.

Introduction

Logging has consistently been one of the most hazardous industries in the United States (US). In 2016, the logging industry experienced a fatal injury rate of 135.9 fatalities per 100,000 full-time workers, which exceeded the national rate of 3.6 fatalities per 100,000 workers.¹ In the same year, 106 deaths occurred nationally within the US logging industry, an increase from 80 deaths in the prior year. This high rate of work-related fatalities in the logging industry highlights a need for continued health and safety efforts to prevent injuries and fatalities among logging workers.

Prior studies revealed that logging workers are well aware of their profession’s dangerous nature.^2,3 However, the degree to which workers comprehend the risk for a work-related safety incident varies. As part of a broader study to characterize and address perceptions of risk and safe work practices among logging supervisors and crew members in Arkansas, Louisiana, and Texas (Ark-La-Tex), our qualitative research found opposing opinions related to risk for worker fatalities among loggers who acknowledged a risk associated with logging work.³ Some loggers believed fatal events were rare in the South, while others felt that safety incidents usually resulted in fatalities.³ In addition, some loggers in that study also characterized the risks of logging as unpredictable, while others believed their risk for injury was equal to that of any other workplace. A common belief reported by study participants was that logging in the southern US is safer compared to other regions because of the mechanization of timber harvest operations.³

Currently, timber producing states have some form of logger training or certification available. These programs have multiple goals with safety on logging sites being secondary to best management practices. One existing training developed by the West Virginia Division of Forestry began incorporating a safety module with a video component into a safety certification program, which yielded promising results.² After viewing a video addressing logging safety, which was incorporated into the training, there was an increase in knowledge among training participants regarding logging safety hazards and how to mitigate them. Improvement in attitudes and positive changes in self-reported work practices were also reported by training participants at a 6-month post-training evaluation.

While prior safety training studies have focused on safety awareness among workers, no trainings have been developed and evaluated at the management level. The primary function of leadership and management is to produce change among workers in a business organization.⁴ More specifically, safety leadership is the process of interaction between business leaders and workers, through which leaders can influence workers to achieve business safety objectives and promote a positive safety culture. Safety management practices include the policies, strategies, procedures, and activities implemented or followed by the management of an organization addressing worker safety.⁵ Six safety management practices have been identified as predictors of safety performance that include: (1) management commitment, (2) safety training, (3) worker involvement, (4) safety communication and feedback, (5) safety rules and procedures, and (6) safety promotion.^6–12 Safety management practices not only improve working conditions but also positively influence worker attitudes and behaviors with regard to safety, thereby reducing injuries in the workplace.⁵ To date, no safety management and leadership training program has been made available to logging supervisors in the Ark-La-Tex region.

Because logging operations often take place in remote locations, delivery of any type of training can be challenging. Recent advances in mobile technologies have provided new opportunities for training delivery as compared to traditional classroom-based training. Mobile-learning (m-learning) refers to the use of mobile or wireless devices for the purpose of learning.¹³ Any type of electronic mobile device including smartphones, tablets, laptops, or personal media players can fall within this scope. The utilization of mobile devices for training has the potential to improve the delivery and effectiveness of safety training, as well as overcome many challenges associated with safety training on remote logging sites.

Arcury et al.¹⁴ recommend that a variety of training delivery formats (visual, hearing, hands-on) should be utilized when providing health and safety training to those in the agriculture and forestry workforce. Our objective was to develop, deliver, and evaluate a safety management and leadership training module among logging supervisors using mobile tablets as a personal learning environment. For this safety training project, we (1) developed a safety management and leadership training vignette to be deployed on mobile devices and (2) examined the effectiveness of the developed training vignette among logging supervisors.

Methods

General methods

Safety management and leadership training materials were developed using previously collected data in focus groups comprised of logging supervisors.³ The training vignette was produced and administered using Articulate Storyline software¹⁵ using iPad devices as a mobile platform.

Researchers partnered with three state logging associations in the Ark-La-Tex region to facilitate recruitment of training participants. Logging supervisors, who provided logging site supervisory oversight of workers and harvest operations, were recruited to participate in the training while they attended logging association monthly meetings. Inclusion criteria for each participant included being 18 years or older and self-identified as a logging supervisor of logging harvest operations in the Ark-La-Tex region. All trainings took place in conference rooms at logging association headquarters. Prior to training, study objectives were explained to logging supervisors and informed consent was obtained. All trainees expressed fluency in mobile device operation, including iPad tablets, which were used as the training platform. After informed consent was obtained, training participants provided demographic information including age, race, ethnicity, years of experience in the logging industry, and their number of years of experience as a logging supervisor. Once demographic information was collected, participants began the training. The training vignette lasted approximately 30 min, which included pre- and post-test knowledge evaluations. Participants completed the training at their own pace and had the ability to go back to previous content for review. The training project was approved by the University of Texas Health Science Center at Houston, Committee for the Protection of Human Subjects and the Institutional Review Board of the University of Texas Health Science Center at Tyler.

Training content

Training content included the current scope of the logging safety situation by presenting the most recent morbidity and mortality statistics for the Agricultural, Forestry, and Fishing (AgFF) industrial sector, as well as logging-specific injury and fatality statistics. Because this training was delivered exclusively to logging supervisors working in the Ark-La-Tex region, emphasis was placed on safety risks specific to mechanized logging practices. In order to emphasize the severity and reality of tragic events within the industry, the training reviewed actual injury and fatality events reported in the media that occurred on logging sites in Southeastern US. The training vignette included interactive opportunities which consisted of scenario-based problem-solving exercises.

Several safety management and leadership concepts were introduced to supervisors throughout the training, including the importance of complying with current Occupational Safety and Health Administration (OSHA) standards for safe work practices. The safety management portion of the training reviewed the importance of supervisor commitment to safety, worker participation, hazard identification and control on logging sites, and safety training.¹⁶ Supervisors were presented with methods to increase the frequency at which they discussed safety with their workers and provided supervisors with suggestions to improve safety communication. The vignette included the importance of rewarding safe work practices and addressing unsafe behaviors to prevent work-related injuries. The training emphasized the responsibility of the supervisor in ensuring workplace safety and worker accountability related to protecting themselves and coworkers while on the job, as well as the importance of being able to identify, report, and address safety hazards.

Evaluation methods

To evaluate the effectiveness of the training, the Kirkpatrick Four-Level Training Evaluation Model was utilized.¹⁷ Prior to viewing the training vignette on the iPad devices, workers completed a pre-test as part of the Level-2 evaluation (learning). This testing tool asked training participants 15 questions presented in multiple choice or true/false formats. After viewing the training vignette, a post-test, which replicated the questions on the pre-test (but in different order), was administered. Upon completing the post-test, participants completed a Level-1 evaluation (reaction) in which they provided feedback about their training experience utilizing a Likert scale. Participants also received immediate feedback on their Level-2 pre-and post-test scores on the mobile device. Level-3 evaluations (behavior) were administered via phone 3 months post-training by a member of the study team. All evaluations were administered using Qualtrics Mobile Survey Software®, which allowed responses to be saved offline for subsequent download and analysis. Budgetary and time limitations precluded the necessary data collection from being performed to enable Level-4 (results or injury reduction) and Level-5 (cost-benefit analysis) effectiveness evaluations.

Statistical analysis

Statistical analysis included descriptive statistics of subject demographic characteristics, test score results and training effectiveness evaluations. Paired t-test determined differences between pre- and post-test mean scores, and Hedges’ g estimate was used to compute effect size based on a comparison of pre- and post-test mean scores relative to pooled variances.¹⁸ All statistical analyses were performed using Stata v.14 [Stata Corp LP, College Station, TX].

Results

A total of 31 male, English-speaking, logging supervisors received the safety management and leadership training. The median age reported by workers was 46.5 years (SD = 12.3 years). The majority of workers reported having a high school education (75.0%). The entire sample self-identified as being white, with 77.4% identifying as Non-Hispanic, 3.2% identifying as Hispanic, and 19.4% identifying as other ethnicity (Table 1).

Table 1.

Demographics of logging supervisor training participants (n=31).

Characteristics	Mean (SD) or Frequency (%)
Gender (%)
Male	31 (100.0)
Female	0 (0.0)
Age	46.5 (12.3)
Years working in the logging industry	25.1 (12.6)
Years working as a supervisor	18.0 (13.3)
Highest education level achieved (%)
No Education	0 (0.0)
Elementary School	0 (0.0)
Middle School	1 (3.6)
High School	21 (75.0)
Higher Education	6 (21.4)
Race (%)
White	31 (100.0)
Ethnicity (%)
Non-Hispanic	24 (77.4)
Hispanic	1 (3.2)
Other	6 (19.4)

Level-1 evaluation

The majority of participants rated their training experience favorably in their Level-1 evaluations. Most agreed that the training environment and using the iPad device to complete the training were enjoyable experiences and simple to navigate. More than half of training participants reported having learned new techniques to use on the job (Table 2).

Table 2.

Training effectiveness evaluations.

	Strongly Disagree (%)	Disagree (%)	Somewhat Disagree (%)	Neither Agree or Disagree (%)	Somewhat Agree (%)	Agree (%)	Strongly Agree (%)
Level-1 (n=31)
Q1. It was easy to use the iPad device to complete the training.	3.1	0.0	0.0	0.0	6.3	59.4	31.3
Q2. I enjoyed participating in the training using the iPad device.	0.0	0.0	0.0	6.3	6.3	59.4	28.1
Q3. The quiz questions were easy to understand.	3.1	3.1	0.0	0.0	0.0	75.0	18.8
Q4. I enjoyed the training area.	0.0	0.0	0.0	3.1	3.1	68.8	25.0
Q5. I learned new techniques to use on the job.	3.1	0.0	3.1	9.4	18.8	50.0	15.6
	Mean (SD)	ttest p-value
Level-2 (n=31)
Pre-test	78.9% (9.9)	0.0015
Post-test	86.2% (12.7)
		Never (%)	Once a week (%)	2–3 times a week (%)	4–6 times a week (%)	Daily (%)
Level-3 (n=10)
Q1. Since having completed the training how often have you applied any of the safety management practices that you learned from the training?	Pre	-	-	-	-	-
	Post	0.0	20.0	30.0	0.0	50.0
Q2. How often do you talk to your workers about safety in the typical work week?	Pre	0.0	40.0	30.0	0.0	30.0
	Post	0.0	40.0	10.0	0.0	50.0
Q3. How often do you reward workers for good safety behaviors or practices?	Pre	10.0	40.0	40.0	0.0	10.0
	Post	10.0	40.0	10.0	10.0	30.0
Q4. How often do you discipline workers for unsafe behaviors or practices?	Pre	40.0	0.0	40.0	0.0	20.0
	Post	20.0	20.0	0.0	20.0	40.0
Q5. How often do you perform safety inspections on cut sites?	Pre	0.0	10.0	60.0	0.0	30.0
	Post	0.0	10.0	0.0	10.0	80.0

Level-2 evaluation

A total of 31 responses were collected during both the pre- and post-tests. The average pre-test score of correct responses by participants was 78.9% (SD = 9.9) (Table 2). The average post-test score was 86.2% (SD = 12.7), resulting in a statistically significant difference in pre- vs. post-test mean scores [t(30) = −3.50, P = 0.0015]. Overall, we observed a medium to large difference in pre- and post-test scores relative to the pooled standard deviation, resulting in an effect size estimate of 63.98. These results indicate a medium to large learning effect, suggesting safety management and leadership knowledge was gained among training participants. Logging supervisors showed the most improvement in their recognition of the inherent danger in the logging industry as evidenced by high fatality rates, the role of hazard identification and control in safety management, and the importance of prioritizing worker safety.

Level-3 evaluation

Of the 31 logging supervisors who completed the training, 10 were available to complete the Level-3 follow-up evaluation via phone call 3 months post-training. Of these supervisors, all reported adopting new safety management and leadership practices by increasing their discussions with workers about logging site safety to a minimum of once per week following training completion. Most noteworthy, the percentage of supervisors reporting as performing daily safety inspections increased from 30% to 80%. As a result of the training, most supervisors reported rewarding workers for correct safety practices, addressing their observed unsafe behaviors, and performing safety inspections on logging sites (Table 2).

Discussion

According to the most recent Recommended Practices for Safety and Health Programs released by the Occupational Safety and Health Administration (OSHA), recommended elements of safety and health management programs are safety leadership, worker participation, hazard identification and assessment, hazard prevention and control, education and training, and program evaluation and improvement.¹⁶ Our training presented these safety management and leadership components to logging supervisors in a novel training format that was easy to understand and deliver, and overall was well-accepted by training participants. We found significant increases in knowledge gained immediately after training completion. Among those who participated in behavioral evaluations 3 months post-training, positive changes were reported, with all participants reporting as having applied new techniques learned from the training, as well as increasing the frequency of safety discussions with workers.

Logging is physically demanding and dangerous work that often requires heavy labor and places workers in hazardous environments. Typically, workers spend the majority of their time outdoors, often in isolated areas. While working, loggers are subject to several factors that put them at increased risk for injuries including: hazardous terrain conditions, poor weather conditions, exposure to biological agents, the use of potentially dangerous machinery and tools, heavy loads, physical agents, such as noise and vibration, dust, and exhaust gas among many others.¹⁹ By the year 2050, annual timber harvest in the US is expected to increase by 24%, with the majority of this increased harvest coming primarily from the southern portion of the nation.²⁰ This expected increase, identified safety risk factors, and the high occupational injury fatality rate in the logging industry suggests a serious need for improved safety training strategies that more effectively address the hazards associated with employment in this occupation.

Currently, few safety training initiatives for loggers have been evaluated for effectiveness of safety awareness or knowledge gains, behavioral change, or injury reduction.^2,21,22 Those that have been evaluated have provided mixed results, with some showing no evidence of significant injury reduction and others suggesting positive behavioral changes. These inconsistencies have been thought to result from high turnover rates in workers who do not receive trainings in their entirety.² Among the successful trainings where injury behavioral changes have occurred, there has been a notable interest in safety and improvement in safety behaviors and practices as reported by workers after training completion.²

This is one of the few studies that attempts to evaluate behavioral changes/improvements after the completion of safety training developed for and delivered through a mobile device platform. Increased knowledge was demonstrated immediately following training completion. Safety management and leadership behaviors assessed at a three-month follow-up demonstrated positive behavior changes as participants reported applying concepts learned in the training.

There are several limitations to this study, one of which is the limited and non-random study sample. Though the statistical analysis showed a significant positive effect of knowledge gained and subsequent improved safety management and leadership behavior, future training interventional studies should recruit a larger sample of logging supervisors to more adequately evaluate training effectiveness. In addition to limited sample sizes (Levels 1, 2, and 3 effectiveness evaluations), it is important to note that the sampled group was homogenous when it came to race/ethnicity and reported relatively high levels of educational attainment. Training outcomes may have differed among logging supervisors with lower education levels or different races/ethnicities as compared to supervisors included in our sample. Additional limitations are the possibility of a nonresponse bias due to the limited Level-3 sample size, as well as reporting bias as a result of dependence on self-reported behaviors in our Level-3 post-training evaluation. Thus, respondents may have been incentivized to report positive training results.

Our training effectiveness evaluation revealed that safety management and leadership training has the potential to increase safety knowledge and result in positive behavioral changes. However, future effectiveness evaluations are needed to determine if safety training delivered in this format results in a reduction in worker injuries and fatalities (Level-4 evaluation). Additionally, the evaluation of such training in the field on actual logging sites should be conducted. Our findings suggest that video trainings delivered via mobile devices are not only capable of addressing safety topics and behaviors but are also fairly easy to deliver. This method of training delivery in the logging industry holds promise in effectively disseminating relevant training content on logging sites. Our observations support the use of mobile devices as just one component of a comprehensive safety management and leadership training program for logging supervisors. This training should not be viewed as a replacement of other traditional training methodologies that can be included in future training curricula, such as live classroom-based lectures, on-the-job mentorship or coaching, or group discussion. Use of this mobile learning may prove fruitful for future safety training effectiveness studies. Future development and evaluation of mobile learning safety training is needed to ensure the cultural, linguistic, and literacy needs are met among logging workers, as well as their supervisors.

Conclusion

Our findings suggest the utilization of mobile learning techniques may be an effective means to deliver safety management and leadership training content to logging supervisors. Supervisor safety training should be linguistically and literacy-level appropriate, as well as comprehensive in nature, including meaningful and relevant content. Our observations support the use of mobile devices as just one component of a more comprehensive health and safety management program for supervisors in the logging industry.