Evaluation of toolbox safety training in construction: The impact of narratives

Abstract

Background:

Construction is a dangerous industry with a large number of small businesses. Because they require minimal resources to deliver, toolbox talks may be an ideal training format for small construction contractors.

Methods:

Eight toolbox talks were developed, each with two versions. One version of each toolbox talk was standard and one version included a narrative and discussion questions. Participants were randomly assigned to receive the standard or the narrative version. Pre- and post-intervention surveys measured demographics, workplace safety climate, and knowledge. The post-intervention survey also measured training impact.

Results:

Including narratives with discussion questions significantly increased knowledge gain and led to increased training impact. Less experienced workers were more likely to gain knowledge and training impact compared to more experienced workers. There were no significant changes in workplace safety climate.

Conclusions:

The results suggest that including a narrative and discussion questions increases toolbox talk effectiveness.

1 |. INTRODUCTION

Construction is one of the most dangerous industry sectors in terms of mortalities and morbidities. In 2015, roughly 9 million workers were employed in construction in the United States, representing 6% of the total workforce.¹ There were more fatal injuries in construction than any other industry in the United States, accounting for 19% of the 4836 work-related deaths that year.² The rate of injuries requiring days away from work in the construction industry was 134.8 injuries per 10 000 FTEs in 2015, a rate higher than all private industries combined (93.9 injuries per 10 000 FTEs).³

From an occupational safety and health (OSH) perspective, one of the most prominent characteristics of the construction industry is the large number of contractors involved in nearly any project. Most of these contractors are very small enterprises. It is estimated that approximately 90% of construction contractors employ 20 or fewer workers.⁴ Smaller construction firms experience higher rates of fatal injuries than larger firms.⁵ For example, in 2010, 56.3% of construction deaths occurred in establishments with fewer than 20 employees, yet such establishments employed just 41.4% of the construction workforce.⁵ Among the challenges to meeting the OSH needs of small businesses is that they typically have very limited resources to apply to OSH training needs.⁶ There are often no dedicated safety staff in these firms, rather it is among the several “hats” worn by the owner, who is often also the office manager, a field supervisor, and works alongside employees on projects. Because they require minimal resources and no professional training to deliver, toolbox talks may provide an ideal OSH training format for small construction contractors.

2 |. LITERATURE OVERVIEW

Toolbox talks are brief (10–15 min) OSH instructional sessions held on the worksite or at the contractor’s office. A number of sources^7–9 recommend that toolbox talks should focus on a specific topic that is relevant to the immediate worksite and which can be fully discussed in the limited amount of time available. Some sources have suggested that toolbox talks can be made more effective by including short narratives describing relevant scenarios (typically OSH failures) and questions about the scenarios.¹⁰ Unfortunately, the rationale for the inclusion of narratives does not go far beyond suggesting that workers will find narratives more interesting and therefore should pay greater attention to them. As discussed later in this paper, strong rationales for the effectiveness of narratives may be found in the concept of training engagement¹¹ and as a relevance modifier in the Elaboration Likelihood Model.¹²

Toolbox talks are widely used in the United States^{7–9,13–15} and internationally,^16–17 however, there is nothing in the literature addressing issues such as timing and frequency of presentation. Indeed, as pointed out by Olson et al,¹⁵ there is little in the empirical literature demonstrating either the effectiveness of toolbox talks as a training device or specifically testing how to tailor them for greater effectiveness. Olson et al found that using line drawings as opposed to photographs significantly increased the distance from which workers could identify hazards. In addition, they developed a new format for toolbox talks that was preferred by supervisors and decreased the amount of preparation time required for the trainings. Kaskutas et al¹⁴ investigated the impact of having foremen participate in an 8-h fall prevention and safety communication training focusing on brief safety interventions such as toolbox talks. Worksite evaluations of both foremen and crewmembers found significant improvements in fall prevention knowledge, safety behaviors, and safety communications that lasted for at least 6 months following the training. Although this study showed that improving the communication skill sets of foremen can increase the effectiveness of the training they provide for their crewmembers, it did not directly address the effectiveness of toolbox talks as a stand-alone teaching device.

This paper presents the findings of a study that investigated the effectiveness of toolbox talks to increase OSH knowledge, to increase the impact of training safety, and to improve worksite safety climate. This study also investigated whether the addition of narratives with discussion questions increased the effectiveness of toolbox talks in terms of knowledge gain, behavioral intentions, and safety climate.

3 |. MATERIALS AND METHODS

3.1 |. Materials development

Toolbox talks on eight common construction OSH concerns were developed by NIOSH researchers for the purposes of this study. A list of the most high-risk activities was generated by consulting mortality and injury statistics for the construction industry. This list was reviewed by NIOSH subject matter experts and stakeholder partners. The stakeholder partners included representatives from the construction industry, the Occupational Safety and Health Administration (OSHA) and adult education specialists. Eight topics were selected for development of toolbox talks. The topics selected were: Preventing falls from roofs, Preventing falls from extension ladders, Preventing deaths from improper tool use, Preventing falls from equipment or loads (forklifts), Preventing falls through holes in roofs and floors, Preventing electrocutions: Overhead power lines and boom cranes, Preventing deaths from crushing: Building materials, and Preventing deaths from skid-steer loaders. The NIOSH subject matter experts and stakeholder partners guided, reviewed, and approved the content of the toolbox talks developed for this project. Two versions of the toolbox talks were produced. The first version consisted of a brief discussion of the OSH concern in general, followed by a bulleted list of appropriate safety measures accompanied by illustrations of relevant safety equipment. The second version contained the same OSH content as the first version, but also included a short narrative (1 paragraph) accompanied by questions asking (1) whether the participant had ever suffered such an accident and its consequences and (2) how the participant could help to prevent such an accident on their current worksite. The narratives were adapted from Fatality Assessment and Control Evaluation (FACE) Program reports (available at https://www.cdc.gov/niosh/face/default.html) of actual workplace safety mishaps related to the OSH topic of the particular toolbox talk. Both versions were not more than one page in length. The back page of each had a sign-in list for attendees. See Figure 1 for an example of a toolbox talk containing a narrative. The non-narrative toolbox talks looked the same, but without the narrative paragraph and discussion questions.

FIGURE 1.

Sample toolbox talk containing narrative and discussion questions

3.2 |. Participants

Participants were recruited from among the employees of general construction companies operating in the Greater Cincinnati Metropolitan Area. Potential companies were suggested by NIOSH subject matter experts and local construction industry stakeholders. Each company was contacted by NIOSH researchers and the purpose of the study was explained. The companies were asked to review the toolbox talks to ensure the safety topics were relevant to their workers and worksites. Companies agreeing to participate were asked to identify individuals who would be trained as toolbox talk presenters and to select worksites for training and data collection. The presenters received 1-h of training to familiarize them with the toolbox talks and the data collection requirements of the study. This training was intended to standardize presentation and data collection across all sites participating in the study. Companies were randomly assigned to study conditions. To prevent contamination between worksites, it was decided that all worksites from participating companies needed be assigned to the same treatment condition. Working within this constraint and the need to balance the number of workers assigned to each condition, the companies were randomly assigned to either the control (toolbox talk alone) condition or the treatment (toolbox talk and narrative) condition. This approach is sometimes referred to as urn randomization in the clinical trials literature.¹⁸

Nine companies with 16 worksites agreed to participate in this study. Seven companies with eight worksites were assigned to the control condition and two companies with eight worksites were assigned to the treatment condition. A total of 351 individuals completed the baseline questionnaire (163 in the control group and 188 in the treatment group). At the end of the 8th week, a total of 207 individuals completed the post-intervention questionnaire (107 in the control group and 100 in the treatment group). This represents an attrition rate of 41.0% for the entire sample (34.4% for the control group and 46.8% for the treatment group).

3.3 |. Intervention

The toolbox talks were presented one per week, for 8 consecutive weeks. Typically, the talks were given at the beginning of the work shift on Monday mornings. Prior to the first toolbox talk presentation, to establish a baseline, the workers were asked to complete a brief questionnaire. The questionnaire began with six basic demographics items. Workplace safety climate was assessed using a short three-item scale (Cronbach’s alpha 0.75) developed for the purposes of the study. A sample question is “If an inspector showed up today, how well would your site do?” Response choices were a five-point Likert scale ranging from (1) poor to (5) excellent. Scores ranged from 3 (participant only responded “poor”) to 15 (participant only responded “excellent”). OSH knowledge was assessed with eight multiple choice items. These items reflected the content of the toolbox talks used in the study and were developed for the purposes of this study. A sample question is “Which of the following is NOT a correct way to use an extension ladder?” There were four response options for each question, and answers were coded as either correct or incorrect. Scores ranged from 0 (all incorrect) to 8 (all correct). After the presentation of the 8th toolbox talk, the participants were asked to once again complete a questionnaire to assess post-intervention changes related to participating in the study. The post-intervention questionnaire included the same demographics, safety climate, and knowledge items from the baseline assessment. In addition, participants were asked about current training impact. A five-item scale assessing training impact (Cronbach’s alpha 0.73) was developed for the purposes of this study. A sample question is “My coworkers work more safely than they did before the training.” Response choices were a four-point Likert scale ranging from (1) strongly disagree to (4) strongly agree. Scores ranged from 5 (participant only answered “strongly disagree”) to 20 (participant only answered “strongly agree”).

• H1:This study was designed to test the following five hypotheses:
• H2:As compared to baseline, all participants will show significant gains in OSH knowledge.
• H3:As compared to baseline, all participants will report significant improvements in workplace safety climate.
• H4:On the post-treatment questionnaire, the participants receiving the narrative toolbox talks will show significantly greater gains in OSH knowledge compared to participants in the non-narrative condition.
• H5:On the post-treatment questionnaire, the participants receiving the narrative toolbox talks will report significantly greater improvements in workplace safety climate compared to participants in the non-narrative condition.
• H6:On the post-treatment questionnaire, the participants receiving the narrative toolbox talks will show significantly greater gains in training impact compared to participants in the non-narrative condition.

3.4 |. Statistical analysis

Results were analyzed using IBM SPSS Statistics Version 24. Paired samples t-tests were used to determine significance pre- and posttraining, and independent samples t-tests were used to determine significance between the treatment and control groups. This work was reviewed and approved by the NIOSH Institutional Review Board.

4. |. RESULTS

Three of the participants in the control group reported that their native language was Spanish. Given that the study materials were all in English, it was decided to drop these three individuals from the study to reduce the possible influence of language fluency upon results. Therefore, the following analyses were conducted on a total sample size of 204 (104 in the control group and 100 in the treatment group). Missing data were deleted pairwise.

Statistical analyses were conducted comparing the study dropouts with those retained in terms of demographics, OSH knowledge, and safety climate. No significant differences were found. Additional analyses were conducted comparing the treatment and the control groups at baseline on OSH knowledge and training impact and no differences were found. However, the groups did differ significantly on safety climate. The control group rated the safety climate of their worksites significantly lower than the treatment group (2-tailed t = −2.250, P = 0.026). See Table 1 for relevant pre-intervention demographics.

TABLE 1.

Demographics reported on the pre-intervention questionnaire

	Control group		Intervention group		Total sample
Characteristic	N	%	N	%	N	%
Age
Less than 22 years	8	7.7	9	9.1	17	8.4
23–30	25	24.0	27	27.3	52	25.6
31–40	28	26.9	27	27.3	55	27.1
41–50	30	28.8	23	23.2	53	26.1
51–60	12	11.5	8	8.1	20	9.9
Over 60 years	1	1.0	5	5.1	6	3.0
Total	104	99.9	99	100.1	203	100.1
Length of time in trade
Less than 1 year	9	8.7	12	12.1	21	10.3
1–5 years	32	30.8	28	28.3	60	29.6
6–10 years	19	18.3	13	13.1	32	15.8
11–20 years	24	23.1	21	21.2	45	22.2
Over 21 years	20	19.2	25	25.3	45	22.2
Total	104	100.1	99	100.0	203	100.1
Educational background
Have not graduated from high school	5	4.9	14	15.1	19	9.7
High school graduate or equivalent (GED)	55	53.9	47	50.5	102	52.3
Some college	33	32.4	19	20.4	52	26.7
College graduate	9	8.8	13	14.0	22	11.3
Total	102	100.0	93	100.0	195	100.0

4.1 |. Hypothesis testing

Hypothesis 1: As compared to baseline, all participants will show significant gains in OSH knowledge. The mean baseline knowledge score for the entire sample was 6.97 (SD = 1.05), as compared to a post-intervention mean knowledge score of 7.46 (SD = 0.749). This increase was a statistically significant increase, in the predicted direction, in knowledge scores (paired sample, one-tailed t = 5.724, df =167, P <.001). Hypothesis 1 was supported.

Hypothesis 2: As compared to baseline, all participants will report significant improvements in workplace safety climate.

The mean baseline safety climate score for the entire sample was 11.37 (SD 1.76), as compared to a post-intervention mean safety climate scale of 11.28 (SD 11.28). This was not a statistically significant increase in safety climate scores (paired sample, one-tailed t = 0.709, df = 178, P = .239). Hypothesis 2 was not supported.

Hypothesis 3: On the post-treatment questionnaire, the participants receiving the narrative toolbox talks will show significantly greater gains in OSH knowledge as compared to participants in the non-narrative condition. The mean postintervention knowledge score of the non-narrative group was 7.337 (SD = 0.799), as compared to the mean post-intervention knowledge of the narrative group of 7.728 (SD = 0.750). This was a statistically significant increase, in the predicted direction, in knowledge scores (one-tailed t = 1.689, df = 187, P = .046). Hypothesis 3 was supported.

Hypothesis 4: On the post-treatment questionnaire, the participants receiving the narrative toolbox talks will report significantly greater improvements in workplace safety climate as compared to participants in the non-narrative condition. The mean post-intervention safety climate score of the non-narrative group was 11.21 (SD = 1.939), as compared to the mean post-intervention safety climate score of the narrative group of 11.43 (SD = 1.562). This was not a statistically significance difference in safety climate scores (one-tailed t = 0.87, df = 192, P = .194). Hypothesis 4 was not supported.

Hypothesis 5: On the post-treatment questionnaire, the participants receiving the narrative toolbox talks will show significantly greater gains in training impact as compared to participants in the non-narrative condition. The mean postintervention training impact score of the non-narrative group was 11.474 (SD = 1.926) as compared to the mean postintervention training impact score of the narrative group of 10.047 (SD = 1.588). This is a statistically significant difference (one-tailed t = 3. 272, df =81, P = .003), in the predicted direction in safety attitude scores. Hypothesis 5 was supported.

4.2 |. Additional analyses

Given the mixed findings from the initial hypothesis testing, additional analyses were conducted to further understand the findings from this study. Anecdotal accounts of previous NIOSH training studies suggested that less experienced workers might be more amenable to OSH interventions than more experienced workers. There is some support for this phenomenon in the literature. For example, Flynn and Sampson¹⁹ report that less experienced workers complained that they were frequently prevented from implementing better OSH practices by the more experienced workers who dominated the safety culture in their company. In an excellent ethnographic study, Paap²⁰ describes a construction culture dominated by entrenched hierarchies. To be considered a “good worker” by more established peers, newer workers must be willing to place productivity ahead of safety and accept being injured as an integral part of working in construction. Consequently, it was decided to investigate the findings from this study by comparing less experienced with more experienced workers.

The participants in this study were asked to identify as belonging to one of five categories of work experience (see Table 1). For the purpose of the following analyses, the two least experienced groups and the two most experienced groups were collapsed together, yielding two groups: 5 or fewer years of experience (n = 81) and 11 or more years of experience (n = 90). A series of analyses paralleling those conducted in testing the hypotheses above were conducted for each experience grouping, examining whether all participants learned and saw improvements in workplace safety climate, and if participants receiving the narrative toolbox talks had greater improvements in knowledge, safety climate, and training impact than those receiving the non-narrative toolbox talks. These analyses are referred to as H1a-H5a for the more experienced worker group and H1b-H5b for the less experienced worker group.

For the workers with 11 or more years of experience, the only significant finding was baseline to post-training knowledge gain (H1a; two-tailed t = 2.901, df =66, P = .005). Analyses testing the remaining hypotheses (H2a, H3a, H4a, and H5a) did not yield significant results. For the workers with 5 or fewer years of experience, there was also a significant finding for baseline to post-training knowledge gain (H1b; two-tailed t =4.878, df = 73, P < .001). The workers with 5 or fewer years of experience in the narrative condition had significantly higher post-training knowledge scores as compared to the workers in the non-narrative condition (H3a; two-tailed t = 2.102, df = 74, P = .042). In addition, workers with 5 or fewer years of experience in the narrative condition had significantly higher post-training training impact scores as compared to the workers in the non-narrative condition (H5a; two-tailed t = 2.514, df = 38, P = .016). Analyses testing H2a and H4a did not yield significant results.

Any study proposing to influence practice must address the difference between statistical and practical significance. One way of doing so is through calculation of Cohen’s D,²¹ a standardized effect size, and using the associated interpretations of effect size. Testing of Hypothesis 1, contrasting the knowledge gain post-treatment across the entire study sample against baseline, yielded D = 0.545. Testing of Hypothesis 3, contrasting the post-treatment knowledge gains between the control (non-narrative) and the treatment (narrative) groups, yielded D = 0.50. Cohen’s guidance on interpretation of D would categorize both effect sizes as being medium. A medium effect size is expected to be robust enough to be noticeable and meaningful.

The results from testing Hypotheses 2 and 4 indicate that toolbox talks, with or without narratives, did not affect worksite safety climate. However, it is possible that the participating companies might have had more positive safety climates than non-participating construction companies, making it more difficult to improve upon their safety climates. As such, this study may have encountered a ceiling effect due to the relatively high safety climate scores found in the participating companies. Support for this possibility is found in the relatively high mean safety climate and OSH knowledge scores at study baseline. The control group had a mean baseline safety climate score of 11.09 (SD 1.93) and the intervention group had a mean baseline safety climate score of 11.65 (SD 1.47). These scores indicate that both groups considered the safety climates of their companies to be above average. A recent report by Dong et al²² found that 46.6% of construction workers thought their workplace was very safe. This percentage was lower than all industries except agriculture. Thus, the sample for this study was likely somewhat biased with a safety climate that exceeds the industry average. It is worth noting that the significantly lower safety climates reported by the control group, which was comprised of seven smaller businesses, as compared to the treatment group, which was comprised of two larger businesses, was consistent with observations in the literature that smaller businesses can be expected to have poorer safety climates.⁶

The results from testing Hypothesis 5 indicate that toolbox talks with narratives were significantly better than toolbox talks without narratives in increasing training impact. This comparison yielded D = 0.57, another medium effect size. The additional analyses contrasting workers with 5 or fewer years of experience with those having 11 or more years of experience suggest that the group effect for training impact tested in Hypothesis 5 was entirely due to less experienced workers. There were no significant differences in training impact between baseline and post-treatment for workers with 11 or more years of experience nor was there a treatment effect for this group when comparing narrative and non-narrative conditions. This finding suggests that more experienced workers are more entrenched in their safety behaviors and attitudes than less experienced workers. The Cohen’s D for the comparison of workers having 5 or fewer years of experience in the narrative and the non-narrative conditions was 0.836, interpreted as a large effect size. On one hand, it is very hopeful that less experienced workers are more amenable to safety attitude interventions. On the other hand, attention must be given to the fostering of this openness to change lest it be snuffed out by the dominant attitudes of the more experienced workers. These findings are in keeping with both Flynn and Sampson¹⁹ and Paap.²⁰ Unfortunately, due to an error in duplication of project materials on the post-intervention questionnaire, many individuals were not administered the complete training impact questionnaire. Consequently, the findings related training impact are based upon much smaller samples than the remainder of the study measures and must be interpreted with caution.

5. |. DISCUSSION

The results from testing Hypothesis 1 indicate that toolbox talks, both with or without a narrative, facilitate learning. This learning occurs across all levels of worker experience. The results from testing Hypothesis 3 indicate that adding a narrative with discussion questions increases knowledge gain as compared to toolbox talks without a narrative. One possible explanation for the narrative effect may be found in the concept of engagement,¹¹ which refers to the extent to which an individual is actively involved in any given training activity. Burke conducted a meta-analysis of occupational safety and health training studies and concluded that, holding content constant, training that is higher in engagement is more effective than training that is lower in engagement. A traditional classroom lecture with little opportunity for interaction between student and instructor would provide low levels of engagement. In contrast, a “hands on” training with close mentorship by the instructor would provide a high level of engagement. The toolbox talks alone, presented without any discussion, would represent a training with a relatively low level of engagement. The short narrative with discussion questions that was used in this study would represent a training with a medium level of engagement. Therefore, the findings of this study are consistent with Burke.

Another explanation for the increased effectiveness of the narrative condition in increasing training impact is found in the Elaboration Likelihood Model (ELM).¹² Like other communication models, ELM recognizes message relevance to be an important factor related to message impact. The use of brief narratives, developed from actual construction fatality reports, arguably should increase the relevance of the safety message by contextualizing it in terms that the worker can readily identify with. ELM proposes that attitude change occurs through two paths—affective and cognitive. The simple bullet listing of safety facts and best practices in a toolbox talk unaccompanied by a narrative represents a cognitive path approach. The brief narrative of an actual fatal incident represents an affective path approach. Either approach can effectively sway attitudes, at least in the short term. However, to ensure longer-term gains, ELM argues that that individuals need to engage in some level of effortful processing of the material. The more new thoughts an individual is encouraged to generate about a topic, the more likely that the attitude shift will be robust. Inclusion of a narrative in the toolbox talk not only adds an affective appeal to the instruction, but the discussion questions provide an opportunity for individuals to engage in some degree of effortful processing of the information. More effortful processing yields greater safety attitude changes, yielding greater behavior changes further downstream.

5.1 |. Study limitations

One significant limitation of this study was the large attrition rate between baseline and post-treatment. In retrospect, it is now clear that the fluidity of the workforce on even a large commercial construction site was underestimated. Many workers have moved on to other sites in less than8 weeks.Although statistical analysis of baseline measures found no significant differences between completers and non-completers, an attrition rate of 41% must certainly raise some concerns. Another limitation is that the control group was comprised of small businesses, while the treatment group was comprised of large businesses. While the groups were divided this way to get an equal number of worksites for each, the statistical validity may have been impacted.

It is also unknown how similar the companies that were willing to participate in this study are to those that were unwilling or simply construction companies in general. As discussed earlier, it is possible that the participating companies had more positive safety climates than construction businesses in general. The elevated baseline safety climate scores indicate it is possible that failure to find significant improvements in safety climate may be attributable to a ceiling effect. Similarly, the knowledge baseline scores were also relatively high, indicating that the workers had a good OSH knowledge prior to the intervention. Although significant differences (in the hypothesized directions) were found in the knowledge scores, a similar ceiling effect might actually lead to an underestimating of the efficacy of toolbox talks as a training device. Another limitation of this study is that it did not continue to track study impact beyond the immediate postintervention assessment. Therefore, it is not known whether any positive treatment effects were sustained and, if so, for how long. Finally, all of the participants in this study were native English speakers. Given that an increasing number of construction jobs are performed by Latin American immigrants,⁵ English language interventions, no matter how effective with native English speakers, are not likely to be of much use to native Spanish speakers.

5.2 |. Summary

The findings of this study are consistent with the guidance provided in much of the literature discussing toolbox talks. However, it has not previously been empirically demonstrated. Toolbox talks are an effective teaching device for all construction workers and are particularly effective for newer workers. Including a narrative and discussion questions increases their effectiveness. It is important to note that both explanations provided for the increased effectiveness of the narrative condition toolbox talks emphasize not just the narrative, but also the discussion questions as being key to increasing engagement and processing of information. Calculation of Cohen’s D for the findings of this study suggest that these effect sizes are meaningful and robust. There is a need to continue to demonstrate effectiveness among other samples within the construction industry to investigate not only knowledge gain and behavior change, but also the impact on safety climate.

5.3 |. Future directions

The most obvious next research steps are those addressing the study limitations. Although logistically more demanding than the method used by this study, future studies that use workgroups rather than worksites for data collection. Attrition may be reduced by following workgroups from site to site. Opportunities should be sought for testing the effectiveness of toolbox talks with the employees of companies with less positive safety climates to address concerns of possible ceiling effects. In addition, it would be of great interest to assess the effectiveness of the toolbox talks for at least several months after the intervention. This assessment would not only speak to the robustness of the treatment effect, but could guide recommendations for frequency of trainings. Spanish language toolbox talks should be developed and tested with native Spanish-speaking construction workers to determine if this approach is effective with them, and whether narrative toolbox talks are more effective than non-narrative toolbox talks.