Associations between a safety prequalification survey and worker safety experiences on commercial construction sites

Abstract

Background:

While assessment of subcontractors’ safety performance during project bidding processes are common in commercial construction, the validation of organizational surveys used in these processes is largely absent.

Methods:

As part of a larger research project called Assessment of Contractor Safety (ACES), we designed and tested through a cross-sectional study, a 63-item organizational survey assessing subcontractors’ leading indicators of safety performance. We administered the ACES Survey to 43 subcontractors on 24 construction sites. Concurrently, we captured the safety climate of 1426 workers on these sites through worker surveys, as well as injury rates, for the duration of the project.

Results:

At the worksite level, higher average ACES scores were associated with higher worker safety climate scores (p<0.01) and lower rates of injury involving days away (p<0.001). Within subcontracting companies, no associations were observed between ACES and worker safety climate scores and injuries.

Conclusions:

These results suggest the overall and collective importance of the construction project and its worksite in mediating worker experiences, perhaps somewhat independent of the individual subcontractor level.

Introduction

For construction projects, pre-qualification processes are used by project owners, general contractors, and construction management companies as part of the bidding process to select contractors and subcontractors to reduce the project’s overall risk. These procedures typically include assessment of a company’s quality of work, financial stability, and project scale experience.^1–5

Increasingly, prequalification processes include safety performance to minimize risk of workplace accidents and worker injuries.^2,3,6 Historically, safety performance has been assessed through health and safety (injury) records, usually recordable injuries as required by the Occupational Safety and Health Administration (OSHA recordables) and the company’s Experience Modification Rate (EMR).^7,8 EMR is a measure used by insurance companies as a measure of an employer’s past workers’ compensation claims and future risk. These lagging indicators, however, have several limitations.⁹ Counting past injuries assumes that all of the injuries that occurred were actually reported. Underreporting of injuries occurs frequently, especially in smaller construction companies.^10,11 In addition, given the measurement of injury rates as per full time equivalents, injuries will occur less frequently in smaller companies compared to larger companies, providing a less reliable assessment over a certain length of time.

The use of organizational surveys assessing leading indicators of a subcontractor’s safety performance in prequalification procedures has increased considerably.¹² Leading indicators of safety are antecedents of injuries and seek to identify the upstream root causes of work-place incidents that can lead to injury and illness. They include the presence of uncontrolled hazards as well as the organizational policies, programs, and practices that monitor, control, and/or eliminate these hazards.

Our recent review of organizational surveys used for safety prequalification procedures identified several limitations of these instruments, mainly not having essential elements of safety management systems or being validated. The surveys identified during the review included a mixture of leading and lagging indicators, with many having some measure of a company’s safety management systems. However, most surveys were missing an assessment of essential elements of safety systems and none had been validated in terms of the survey’s association with lagging indicators (e.g. injury rates) and worker safety climate.¹³ The constructs that received little attention in many of the surveys were program evaluation and improvement, employee involvement, and safety communication. The review found no validation studies associated with these surveys.

The goal of this paper is two-fold – to describe the development of a comprehensive organizational survey for construction contractors that captures the leading indicators associated with best practices of injury and illness prevention programs that can be used as part of a prequalification procedure^14–19 and to examine the association between scores from the organizational survey and worker safety climate and worker injury rates by construction worksites and by subcontractor. We tested the hypotheses that workers on construction worksites with higher (better) organizational survey scores, created by averaging the organizational survey scores of the onsite subcontractors, have higher (better) worker safety climate scores and experience lower recorded worker injury rates. We also tested the hypothesis that workers for subcontractor companies with higher organizational survey scores have higher (better) worker safety climate scores and experience lower injury rates.

METHODS:

This study is part of the ACES (Assessment of Contractor Safety) research study examining prequalification assessment procedures in the construction industry to promote safer construction sites and supported by the Center for Construction Research and Training (CPWR). All study procedures were reviewed and approved by the Northeastern University’s Office of Human Subject Research Protection (Protocol Number 140610).

Development of the ACES Survey

We compiled a new organizational survey, the ACES survey, to assess construction contractors’ safety performance through several steps, including formative research, vetting, and testing the survey. The ACES survey focused only on leading indicators and did not include injury, EMR, or other commonly used lagging indicators. The focus on leading indicators aimed to include antecedents of workers injuries and illnesses; that is upstream root causes of work-place incidents and worker injury and illness. These include organizational policies, programs, and practices that monitor, control, and/or eliminate hazards in the workplace.

For the formative research step, we first compiled lists of questions about leading indicators from existing surveys of organizations assessing safety and health factors within a company. Our review of 52 existing prequalification surveys identified 102 unique questions; however, these surveys had significant gaps related to worker participation and program evaluation components.¹³ Therefore, we examined organizational surveys outside of the construction industry-exclusive sources including the Workplace Integrated Safety and Health Assessment from the Harvard T.H. Chan School of Public Health Center for Work, Health, and Wellbeing;²⁰ the US Occupational Safety and Health Administration’s Safety and Health Assessment Worksheet (OSHA 33 Form);²¹ the Center for Construction Research and Training’s (CPWR) Safety Culture Assessment Tool (S-CAT);²² and the Institute for Work and Health Organizational Performance Metric.²³ In addition, we included constructs for an existing survey used for prequalification related to safety management systems and alcohol and drug related policies and practices associated with injury rates.²⁴

As part of the formative research, we also completed interviews with a convenience sample of safety managers, construction industry safety experts, and labor safety advocates from the Boston metropolitan and New England region to capture other best practices constructs to be considered in the ACES survey. These interviews included 14 safety directors and one purchasing manager currently employed by commercial construction general contractors, five professionals representing risk services or safety for project owners, three insurance experts, and two representatives from community-based worker safety advocacy organizations.

Second, through an iterative process we reduced the numerous survey constructs and items into a draft survey that would be feasible to complete in a reasonable amount of time. The process first categorized the questions into constructs based on the Injury and Illness Prevention Program essential elements ¹⁹ and examined redundancies in and across these constructs. The constructs included management commitment and employee involvement, hazard identification and assessment, safety communication, special programs (including drug and alcohol programs), emergency preparedness, training and information, evaluation and surveillance, outcomes and demographics. We then evaluated the face validity of all the questions based on our own experiences and knowledge followed by an additional round of vetting with general contractor safety directors and subcontractor company personnel to confirm that the survey was appropriate (as per their knowledge of company programs and policies) and feasible to complete within a reasonable time period.

Third, we further refined the ACES survey and its response categories through three rounds of cognitive testing interviews with safety and project managers of several subcontractors on a partnering construction worksite.^25,26 The cognitive testing ensured that the final items effectively measured the intended constructs and were consistently understood by the contractor companies. Each item in the survey was assessed with regards to comprehension, information retrieval, judgment/estimation, and selection of response category.

Evaluation of the ACES Survey

Performance of the ACES survey was evaluated through a worksite-level cross-sectional study where ACES survey responses for companies at each site and for each company were compared to workers’ experiences including worker safety climate collected by surveys and recordable injury rates.

Data collection for the evaluation occurred with a convenience sample of active commercial construction worksites in the Boston-metro area between January 2017 and August 2018. Eligibility criteria included an injury/incident reporting system, project duration of at least 6 months, and a project budget of at least $5 million. Local construction companies and project owners were contacted to discuss if any of their current projects would meet our study inclusion criteria. Additionally, construction project search tools like http://www.bldup.com were used to locate active construction sites in the region. Twenty-three unique general contractors were contacted, resulting in 10 partnerships.

ACES survey data were collected from managers of the worksite’s subcontractors. To recruit subcontractors, research staff worked with safety managers of the general contractors on the worksites and attended project management meetings. There, research staff explained the research purpose to project managers from on-site subcontractors. Project managers (or another eligible person within their company, often safety managers) then either agreed or did not agree to complete the ACES survey on site or at their company’s headquarters. Where feasible, ACES survey data were collected from more than one type of manager within the same company. While participants were not given monetary incentives to complete the survey, they were offered reports on our findings, as well as a report of their own ACES score upon the completion of our project. All participants completing the ACES survey provided informed consent. For each worksite we calculated a subcontractor ACES score by averaging the ACES scores of the subcontractors we were able to collect from that site.

Additional data collected from the subcontractors included lagging indicators of safety performance and company demographics including past OSHA reportable injuries; injuries resulting in days away, restricted, or transferred (DART); hours worked, and Experience Modification Rate (EMR).

Safety climate was collected via worker surveys on these construction sites at the same time ACES Survey recruitment occurred. Research staff worked with site contacts, generally the site’s safety manager, to identify and establish key locations to sample workers and request their participation directly. Safety directors and other site managers and foremen also assisted by distributing flyers noting the day, time, and location of survey data collection by research staff. Only apprentices, journeymen, and foremen completed the worker survey (not upper management positions). Average completion time for our worker survey was 10 – 15 minutes, and participants were given a $10 gift card for participating. All workers who were surveyed did so voluntarily and provided informed consent. No worker names were recorded to protect participant anonymity.,.

The worker surveys contained two safety climate scales capturing the basic two level organizational structure of construction worksites with general contractors as top management and subcontractors providing the direct supervision of workers.^27,28 The worksite level safety climate scale consisted of four questions pertaining to the worksite’s general contractor (Supplementary Table 1). The subcontractor level safety climate consisted of 5 questions pertaining to the subcontractor employing the worker. These questions each had a 5-point Likert response. Scores from these two scales were averaged across workers on the same worksite providing the two safety climate scores for each worksite. Scores from these two scales were also averaged across workers employed by the same subcontractor company, providing the two safety climate scores for each subcontractor.

The other main outcomes utilized for the study were the recordable injury rate for the duration of the worksite’s project and for each subcontractor. Recordable injury data for the worksites (worksite’s hours worked; OSHA reportable injuries; injuries resulting in days away, restricted, or transferred -DART) were collected from our primary contact person for each site (the site’s safety manager or project manager). This occurred 12 to 18 months after the worker survey and the ACES survey data collection in order to assure that the building project was completed. In addition, we collected site specific data such as size, cost, and the progression of the worksite at the time of the ACES and worker survey data collection.

We calculated injury rate for both the worksite and subcontractor by number of medical treatment cases divided by total hours worked then multiplied by 200,000 to obtain the injury rate per 100 full time equivalent (FTE). Similarly, we calculated DART rates by number of lost and restricted workday cases divided by total work hours then multiplied by 200,000 to obtain the number of DART cases per 100 FTE.

Statistical Models

We tested the hypotheses that (1) workers on construction worksites with subcontractors with higher (better) ACES scores will report higher (better) safety climate scores and experience lower injury rates and that (2) workers in subcontractor companies with higher ACES scores report higher safety climate scores as well as experience lower injury rates through univariate correlation and multi variable regression analyses. For the correlation analyses we calculated Pearson correlation coefficients for worksite average ACES scores and subcontractor ACES scores with worksite and subcontractor level safety climate scores, injury rates, and DART rates.

For the regression analyses, we fitted regression models to estimate associations between ACES scores and safety climate (linear multivariable regression) and injury and DART rates (log-linear multivariable models) at the site level. Covariates considered to potentially confound the association between ACES score and worker outcomes at the site level include square footage of project, project cost (in dollars), project completion percent, presence of a full-time safety manager, and total worker hours. In models for injury rates, total number of worker hours were used as an offset term in the model, not as a covariate. Company-level associations between ACES scores and safety climate were adjusted for presence of female workers on site and previous participation in a prequalification survey (other than ACES). Regression analysis testing associations between ACES scores and injury rates at the company level could not be employed due to an overwhelming amount of missing data in total worker hours or hours billed reported by the subcontractor companies. All significance testing was two-sided and at the 5% significance level.

RESULTS:

The final ACES survey consisted of 63 items covering the essential elements of best practices associated with safety and health performance of companies. Given that a majority of the questions had five-point Likert scale responses with either agreement (strongly disagree to strongly agree) or frequency (never to always) verbal markers, and about a quarter were binary yes/no response, the responses were all normalized to a 0–1 scale. The five-point Likert scales were assigned values of 0, ¼, ½. ¾, and 1 and the binary yes/no responses were assigned values of 0 (no) or 1 (yes). The ACES Survey score was then simply the summation of each question’s normalized response proving a possible range from 0 to 63 points, with 63 representing better safety. The complete ACES survey can be found in the supplement materials and instruction can be found on the study’s webpage, http://www.acesprequal.org/.²⁹

To evaluate the ACES Survey, we collected worker and subcontractor data from 25 sites in the metropolitan Boston area (Table 1). Two adjacent sites managed by the same general contractor employing many of the same subcontractors and consisting of different portions of the same project were combined into one site for analysis, yielding 24 sites. On average, worksites were 245,850ft² (Standard Deviation (SD) = 358,790), involved 116 FTEs (SD = 124), and cost $116.80M (SD= $215.82M). At the time of worker survey completion, worksites were an average of 77% complete. Six worksites reported the presence of a full-time safety manager on site.

Table 1:

Descriptive data of the worksites and the companies completed the ACES survey. Values reported are mean (standard deviation) unless otherwise noted.

	Worksite[N]	Subcontractor[N]
DEMOGRAPHICS
Total enrolled	24	43
Square Footage (1000s)*	246 (359) [21]	--
Full Time Equivalents (FTEs)	116 (124) [19]	181 (208) [15]
Project Cost ($ Millions)*	116.80 (215.82) [20]	--
% Project Completion*	77% (30.57) [21]	--
Dedicated safety person [N (%)]*	6 (29%) [21]	--
ACES Score	43 (4) [23]	42 (7) [43]
Experience Modification Rate (EMR)	--	0.86 (0.27) [32]
OSHA VIOLATIONS [N (%)]
0	9 (45%)	18 (95%)
1	5 (25%)	1 (5%)
2	0	0
3 Or More	6 (30%)	0
WORKER EXPERIENCES
Safety Climate
Worksite level	4.1 (0.3) [24]	4.1 (0.4) [43]
Subcontractor level	4.1 (0.3) [24]	4.1 (0.4) [43]
OSHA recordable injuries (per 100 FTEs)	3.1 (4.9) [18]	2.0 (2.31) [12]
DART injuries (per 100 FTEs)	1.5 (2.8) [17]	2.4 (3.3) [13]

^*Item not applicable for subcontracting companies

From subcontractors working at these 24 sites, 64 ACES surveys were completed from 43 unique companies from several trades (Supplementary Table 2). Subcontractor ACES Survey scores were on average 42 points (SD = 6) and ranged from 20 to 50 out of a possible 63 total points.

Across these sites, 1426 workers completed the surveys (Supplementary Table 1). These workers were mostly male (94%), white (72%), and members of unions (88%). Within each worksite, individual workers reported average worksite-level safety climate scores of 4.1 (SD = 0.3) and subcontractor-level safety climate scores of 4.1 (SD = 0.3) on a scale of 1 to 5 (Table 2). These workers were employed by 162 different subcontracting companies.

Table 2 –

Pearson correlations coefficients ([95% confidence interval]) between ACES Score and company indicators and worker experiences.

	Worksite ACES	Subcontractor ACES
N	24	43
EMR	--	−0.26 [−0.56, 0.10]
OSHA violations	0.17 [−0.31, 0.58]	−0.37 [−0.70, 0.10]
Worker Experiences
Safety Climate
Worksite level	0.58 [0.23, 0.80]**	0.03 [−0.28, 0.32]
Subcontractor level	0.58 [0.22, 0.80]**	0.12 [−0.18, 0.41]
OSHA recordable injuries (per 100 FTEs)	0.14 [−0.37, 0.58]	−0.41 [−0.79,0.22]
DART injuries (per 100 FTEs)	−0.07 [−0.55, 0.44]	0.35 [−0.25, 0.75]

^*p<0.05

^**p<0.01

^***p<0.001

Of these 1426 workers who completed the surveys, 635 were employed by the 43 subcontracting companies that completed the ACES Survey (Supplementary Table 3). These workers were similar in demographics to the other 791 workers who were not employed by these companies. These workers reported a company-level average safety climate score of 4.1 (SD= 0.4) and worksite-level average safety climate score of 4.1 (SD =0.4).

Recordable injury data were available from 21 of these sites. Across worksites, the average OSHA recordable injury rate was 3.3 per 100 FTEs (SD = 5.1) and the average rate of injuries involving days away, restricted, or transferred (DART) was 1.6 per 100 FTEs (SD = 3.0).

Positive and significant correlations were found between worksite average ACES scores (for subcontractors on that site) and worksite-level and subcontractor-level safety climate scores averaged across workers (Table 2, Figure 1). Correlations between ACES scores for each subcontractor and their average worker safety climate scores were not significant, suggesting no linear association between these measures.

Figure 1.

Associations between the average Assessment of Contractor Safety (ACES) prequalification organizational survey score on a worksite and the worker safety climate at the subcontractor level (group) and the worksite level (organization). The lines represent the regression analysis presented in Table 3

Adjusted linear models indicated statistically significant associations between worksite-averaged ACES scores and worksite-level and subcontractor-level safety (Table 3). Each one-point increase in worksite ACES score is associated with a 0.072 (95% Confidence Interval [CI: 0.02 0.12]) increase in worksite safety climate, an equivalent of a 1.5% increase. A one-point increase in worksite ACES score is also associated with a 0.052 (95% CI: [0.005, 0.098]) increase in subcontractor level safety climate, representing a 1.0% increase. No significant associations were found in models relating subcontractor ACES scores to worksite and subcontractor-level safety climate.

Table 3 –

Regression coefficients ([95% confidence interval]) from linear models testing for associations between ACES and Worker Experiences. Models are adjusted for square footage, project cost, project completion percentage, FTEs, and presence of a dedicated onsite safety person.

	Worksite ACES	Subcontractor ACES
TOTAL	24	43
Safety Climate
Worksite level	0.072 [0.02, 0.12]*	0.002 [−0.018, 0.022]
Subcontractor level	0.052 [0.01, 0.10]*	0.005 [−0.012, 0.023]
OSHA recordable injuries (Per 100 FTEs)	−0.11 [−0.24, 0.034]	--
DART injuries (Per 100 FTEs)	−0.32 [−0.46, −0.18]***	--

^*p<0.05

^**p<0.01

^***p<0.001

At the worksite level, ACES scores were not significantly correlated with injury outcomes (Table 2, Figure 2). However, adjusted linear models indicated statistically significant associations between worksite-averaged ACES scores and worksite-level and subcontractor-level safety (Table 3). Each one-point increase in worksite ACES score is associated with a 0.32 (95% CI [0.18, 0.48]) decrease in injury rates.

Figure 2.

Associations between the average Assessment of Contractor Safety (ACES) prequalification organizational survey score on a worksite and the recordable and days away, restricted or transferred (DART) injury rates. The lines represent the regression analysis presented in Table 3. FTE, full time equivalents

DISCUSSION

The objective of this study was to describe and evaluate the ACES survey for construction subcontractors that captures the leading indicators associated with best practices of injury and illness prevention programs for use in prequalification procedures.^14–19 The results support the hypotheses that workers on construction worksites with subcontractors with higher (better) scores on their ACES surveys have higher (better) safety climate scores and experience lower injury rates. However, the results did not support the hypothesis that workers in subcontractor companies with higher ACES scores also have higher safety climate scores.

Overall, the ACES survey does capture worker experiences, based on these results. Considering that the ACES survey is completed by a manager of the company, this capture is a positive finding. Often managers reported different levels of safety performance compared to workers, with mangers reporting better practices compared to workers.³⁰ However, relatively speaking the managers’ reports appear to be correlated with worker experiences on these sites.

The results indicate that construction worksites play an important role in improving worker safety outcomes, more so than the individual subcontracting companies. This finding suggests that characteristics of the project impact worker safety and health more so than an individual company. These characteristics can include the collective combination of the subcontractors as measured by averaging the ACES of the onsite subcontractors, the standards set by the project’s general contractor, and/or the commitment of the project owner towards worker safety. This finding reflects what we heard through our formative research with safety managers – it is the worksite and team of contractors (and managers/foreman/workers) collectively that drives the safety experiences for the workers.

The lack of correlations between a subcontractor’s ACES score and worker experiences is similar to what we observed in our previous research that used the subcontractor as the unit of analysis.³¹ In that study, we did not examine worksite level associations – only company level. We also used a different organizational survey that did not contain an assessment of some essential elements of safety management programs, such as employee involvement.^13,19 This lack of association at the subcontractor level may also be a limitation in the company level sample size. When a larger number of companies can be studied, for example over 2000, we have observed trends with a different survey.²⁴ When considered in the context of the results of the construction site experience, the large sample size needed for subcontractors demonstrates again the importance of the project in setting the expectations for subcontractor safety performance.

The results are limited by the cross-sectional design of the study and hence no causality can be concluded. The associations, especially at the worksite level, may be influenced by a selection bias due to our engagement with contractors and worksites who were willing to participate. These companies are often leaders in the field. However, we did have a large variability in worksite ACES averaged scores as denoted by the standard deviations and the figures. Perhaps the ACES survey may be a proxy for other organizational factors such as a financial commitment to safety and health. The worksite data were collected only when a subset of subcontractors was on site, yet our worksite data (injury rates) represented the entire project duration including months before and after survey data collection periods. In the end, the significant associations were found and were informative. And of course, the small sample size in the number of construction sites can be susceptible to outliers; however, when we removed outliers the associations remained significant. In addition, these associations match the results of a much larger data set with a different prequalification survey.²⁴

The design was further limited by the low response rate of the subcontractors in completing the ACES survey. Only about a third of the subcontractors that we identified through the worker surveys, which asked for the primary employer for the workers, completed the ACES survey, despite our strategic efforts. Our best success was through in-person meetings with the subcontractors on site following project management or safety meetings. Numerous attempts to contact company representatives after visits to worksites were fairly unsuccessful. We tried to incentivize companies to complete the survey through an offer of providing them feedback on their scores, which resulted in one additional participating company. Moreover, the ACES survey data completion required knowledge of policies, programs, and field practices. Few individuals within the firm were fluent in all three realms, which may indicate the degree of integrating safety and health into the day-to-day operation of the subcontractor. Some of the specific data concerning the number of injuries and man-hours billed required further resources and was often not immediately available to the person completing the ACES survey for a research study. Within the context of a bidding process, the respondent may have a more effective incentive to find the data. As a result, injury data at the company level was missing in our data, which directly limited our analysis of associations with these data.

In addition, the ACES score gave equal weight to all 63 items in the scale through normalizing the response scales and simply summing these responses. Most likely some of the questions have more impact than others and could have been weighted accordingly. There are analytical methods to determine weighting and also factor analysis to better understand the importance of specific variables and constructs. At this time the number of contractors who have completed the survey is small for such analytical approaches.

Nonetheless, we did have a large range of variability in ACES scores and at the worksite we did see important and hypothesized associations. However, to test associations within companies, a larger sample size is needed with greater variability in the ACES scores. Furthermore, the results are limited to the commercial construction industry with a heavily unionized workforce in the Boston/New England area. Both the ACES survey and safety climate scales were self-reported; however, they were from different populations and when compared to each other and injury data proved to have some reliability.

Overall, these results support the idea that construction projects that engage in prequalification processes with the ACES survey and select subcontractors with higher scores can help in setting an improved safety climate on a worksite and reduce injury rates. While the study design and the capability to measure all important factors were limited, the outcome supported the expected hypotheses. More work needs to be done to better understand and validate the relationships between prequalification procedures, organizational surveys used in these procedures, safety performance, and worker experiences on construction sites to overcome the limitations of this study.