Testing the associations between leading and lagging indicators in a contractor safety pre-qualification database

Abstract

Background

Safety prequalification methods lack validation in predicting construction company’s worker injuries.

Methods

Safety assessments from 2198 construction contractors including Safety Management (SMS), Safety Programs (e.g. falls, hearing protection), and Special Elements (drug testing, return to work) scales were compared to their recordable injury cases (RC) and injuries involving days away, restricted, or transferred (DART).

Results

SMS and Special elements were related to injury rates. Each one-point increase in SMS values was associated with 34% reduced odds of a recordable case rate greater than zero (Odds ratio (OR): 0.66, 95% Confidence Interval (CI): (0.57, 0.79)), and a 9% reduced recordable case rate, if one occurs (Risk Ratio (RR): 0.91, 95% CI: (0.88, 0.94)). A one-point increase in SMS was associated with 28% reduced odds of a DART (OR=0.72, 95%CI (0.56, 0.91)), and 9% reduced DART rate, if one occurs (RR=0.91, 95%CI (0.87, 0.95)).

Conclusions

These results suggest the need for safety management systems in subcontractors.

INTRODUCTION

During the contract bidding process, there is need among owners and general contractors to evaluate the ability of potential subcontractors to provide safe work environments. Partnering with contractors that share similar safety related goals and priorities may increase the success of creating a safer and healthier work environment. Having access to a pre-qualification assessment of a company’s safety systems and procedures may, therefore, help job owners select subcontractors aligned with their safety values.

Many available tools for assessing and prequalifying subcontractors for safety rely on lagging indicators such as injury rates and Experience Modifying Rate (EMR), a measure of a company’s past lost experience. ^1–3 While these lagging indicators are important safety outcomes, they provide minimal information about a company’s systems of safety and safety management policies, programs, and practices. In addition, these lagging indicators may be prone to mismeasurement due to underreporting of work-related injuries in the construction industry. ⁴

More recent prequalification tools have included leading safety indicators, such as safety programs and policies ⁵ as leading indicators are thought of as predictive antecedents to accidents and injuries. In loss prediction models and in other work and safety frameworks, organizational policies, programs and practices are associated with lagging indicators such as worker behavior and worker injuries. ^6–10 Elements of safety management systems include leadership commitment, regular and frequent hazard recognition and controls, employee training, safety communications, and program evaluation. ^11,12 A recent review of the literature concludes, that in the construction industry, the question remains: Do pre-qualification tools that measure systems of safety components predict the reported injuries for the same companies? ¹³

Therefore, our goal is to provide the first test of the associations of safety management systems and other leading indicators with lagging indicators as collected by a prequalification company with over 2000 construction companies. We present and evaluate a subcontractor pre-qualification assessment procedure informed by various leading indicators related to company-level safety policies and programs.

METHODS

Data:

All data in this study were extracted from the ConstructSecure, Inc (Marlborough, Massachusetts) contractor safety assessment program (CSAP) of self-reported and validated company-level safety data. Registered and active companies who subscribe to the ConstructSecure service have a safety manager complete an annual survey capturing measures of safety management systems, special programs, and special elements as well as lagging indicators including OSHA recordable injuries and EMR. Additionally, the survey includes demographic questions about the company such as number of employees, number of person-hours worked, and trade. Many general contractors and owners of construction projects require companies that are bidding on work to be registered with at least one CSAP, which allows for the evaluation of a company’s safety performance alongside the bid details.

Construct Secure has two parts, a survey that includes leading and lagging indicators and a proprietary algorithm that utilizes the survey as well as a copy of the companies written safety policies and program to create a score. In this paper we are interested in describing the relationships between the leading and lagging indicators collected in the survey alone. The result of this survey is a score from zero to one hundred, with higher scores being indicative of safer companies. We do not include this score in our analysis as it is based both leading and lagging indicators of safety making it inappropriate to use as a predictor of lagging indicators.

For this study the lagging safety indicators, which will be dependent variables in all models, are 1) all recordable injury cases (RC) and 2) injuries involving days away, restricted, or transfer (DART). The number of employees was not a reliable measure to create calculate injury rates as 1,809 companies report 0 employees. Therefore, injury rates for both RC and DART, are calculated per 100 full-time equivalent (FTE) person-hours billed.

Safety measures considered as leading indicators included a series of questions summarizing five components of a subcontractor’s organizational programs and policies (Table 1): Safety Management Systems (SMS), Safety Program Elements: Hazards; Safety Program Elements: Programs; Special Elements not related to drug and alcohol programs; and drug and alcohol related Special Elements. The Safety Management System (SMS) section contains 17 questions related to safety culture, communication, and incident investigations, such as “Does your company have a full-time safety manager on staff?,” and “Does your company have a defined budget for safety?” Safety Program Elements: Hazards (SPE.H) consists of 16 questions pertaining to potential hazards that workers may be exposed to on a worksite, e.g. using ladders, working in trenches, or using power tools. Safety Program Elements: Programs (SPE.P) has 17 questions regarding programs addressing specific safety practices related to worksite hazards, such as eye protection or fall protection programs. The non-drug and alcohol program Special Elements (SE.ND) consists of 4 questions covering a contractor’s return-to-work programs, OSHA partnerships, and participation in OSHA’s Safety and Health Achievement and Recognition Program (SHARP) and Voluntary Protection Programs (VPP). The drug and alcohol program Special Elements (SE.D) consists of four questions pertaining to drug and alcohol testing policies. Additionally, companies are asked to self-report the number of OSHA citations received over the past 3 years, which we have categorized as 0, 1, 2, or more than 3. Since data on citations are retrospectively collected, we treat them as leading indicators of an injury occurring in the current year.

Table 1:

Company Descriptive Statistics

Statistic (N = 2,148)	Maximum Score	Mean	St. Dev.	Min	Pctl(25)	Median	Pctl(75)	Max
Safety Management System (SMS)	17	14.2	3.0	0	13.0	15.0	16.2	17.0
Safety Program Elements Hazards (SPE.H)	16	4.6	4.0	0	1.1	4.0	7.0	17.0
Safety Program Elements Programs (SPE.P)	17	14.3	2.4	0	13.9	15.0	16.0	16.0
Special Elements Non-drug and alcohol (SE.ND)	4	1.1	0.9	0	1.0	1.0	1.0	4.0
Special Elements Drug & Alcohol Screening (SE.D)	4	3.0	1.2	0	2.0	4.0	4.0	4.0
OSHA Citations	3	0.4	0.9	0	0.0	0.0	0.0	3.0
Recordable Cases (RC) per 100FTEs		3.0	4.4	0	0.0	1.5	4.2	61.3
Days Away Restricted Transferred (DART) per 100FTEs		1.8	2.9	0	0.0	0.6	2.5	32.3
Full Time Equivalents (FTEs)		296.8	1194.5	10	31.0	70.4	191.8	34989.3

Company specific trade (70 categories) was extracted, but for modeling purposes, we combined certain trades (for example, “Concrete cutting/drilling,” “Poured concrete foundation and structure”, and “Concrete delivery” were grouped together as “Concrete”), reducing the number of categories to 33 (see Supplemental Table 1). ¹⁴ ConstructSecure automatically verifies all data by comparing survey responses to each company’s written safety programs and manuals, and OSHA 300 Summary Form, which are uploaded to the ConstructSecure website.

On October 1^st 2015 we downloaded all the safety data from the ConstructSecure database, which consisted of nearly 5000 companies; however, these downloaded data contained both incomplete and expired data (data from companies that have not updated their surveys within a year). For this study we only included companies with complete and non-expired data. By including only complete and non-expired data, all companies in this study have completed the ConstructSecure prequalification tool between September 30, 2014 and September 30, 2015. We excluded companies reporting less than 10FTEs, as these companies are not subject to OSHA reporting guidelines ¹⁵. In addition, due to their small rates, a single injury adds a lot of variability to an annual injury rate. Lastly, we excluded companies reporting extreme outliers for values relevant to calculating lagging indicator rates, such as an EMR > 2, Hours Worked > 110,000,000, a recordable case rate higher than 2800 per 100 FTEs, and a DART rate higher than 240 per 100 FTEs.

Northeastern University's Office of Human Subject Research Protection reviewed and approved all procedures and methods for the study (IRB Approval #14-06-10).

Statistical Models:

To examine the associations between the leading and lagging indicators we first calculated Spearman correlation coefficients between all measures. We then fit models for each outcome, the company specific count of DART or RC incidents reported, offset by 100*FTEs, to create the respective rates (DART/100FTEs, RC/100FTEs). To appropriately model company specific DART and RC rates, accounting for the large number of companies reporting zero DART and zero RC events, we use Zero-Inflated Poisson (ZIP) models with robust standard errors. The zero-inflation piece of the model first considers a logistic regression model to determine if the DART or RC rates equal zero or if it they are distributed according to a Poisson distribution. If, according to this logistic model, the rate is distributed as a Poisson, then a log-linear model is fit. ¹⁶ Robust standard errors control for mild deviations from the assumption that the mean and variance of the Poisson distributed outcomes are equal. ¹⁷

Covariate selection:

Covariates expected a priori to affect both DART and RC rates included organizational systems of safety such as SMS, SPE.H, SPE.P, SE.ND, and SE.D,. Crude models were first examined to determine predictive power of each individual organizational system of safety on injury outcomes.

The number of OSHA citations reported in the previous 3 years, evaluated as both a categorical and linear term, and trade were also included in the model. Due to the large number of categories, trade was included only in the Poisson portion of the model, not in the logistic regression piece. Final multivariable models were constructed using the cross-validation methods described below.

Model Validation:

Ten-fold cross validation (CV) methods were used to build, compare, and assess the fit of all predictive models. With 10-fold CV, the full dataset is split in to 10 separate sets, models are built and fit using one of the 10 distinct data sets, while predictions are then generated from the remaining 9 left-out sets. Observed outcomes are compared to the predicted outcomes and models which minimize the square root of the mean squared error between these outcomes are preferred. Predictive performance of the DART and RC models was determined by calculating the square of the Pearson correlation coefficient between the held-out observations and their predictions, the cross-validation R². Multivariable adjusted models, selected for maximizing the cross-validated R² between observed and predicted outcomes were the same for both the recordable case rate and DART rate outcomes.

We note that comparison between the performance of this prequalification tool and the ConstructSecure score is impossible, as the outcomes of interest for our model (the lagging DART and RC rates) are used to calculate the ConstructSecure score.

Data summaries and analyses were conducted using the zeroinfl() function in the pscl library of R v3.4.0.¹⁸ Robust standard errors were calculated using Stata SE version 14.

RESULTS

Study population:

At the time of our data download, the ConstructSecure database contained complete, partial, or expired safety information on 4,877 companies. For each of the 2,512 companies had complete unexpired record, information related to leading and lagging safety indicators. After removing companies according to the criteria described in the Methods section above, prequalification data were available for 2,148 companies (Table 1), located in 46 states (MS, MT, SD, and WY were not represented), the District of Columbia, Alberta, Ontario, Quebec, and New Brunswick. The average company reported a safety management score (SMS) of 14.2 (Standard deviation (SD)=3.0, Maximum (max)= 17), 4.6 hazards (SPE.H) (SD=4.0, max=17), and 14.3 safety programs (SPE.P) (SD=2.4, max=16). Companies also reported an average of 1.1 safety special elements (SE.ND) (SD=0.9, max=4) and 3 drug and alcohol screen programs (SE.D) (SD=1.2, max=4). While over 75% of companies reported zero OSHA citations over the past 3 years, the average number of citations equaled 0.4 (SD=0.9, max=3).

The distribution of FTEs was skewed towards large values, evidenced by the company average of 296.8FTEs, but median FTEs equaling 70.4. Once scaled by the FTEs, the average DART rate among companies was 3.0 per 100FTEs (SD=4.4), with 914 companies reporting 0 DART events. The average recordable case rate was 1.8 per 100FTEs (SD=2.9), with 701 companies reporting 0 recordable cases. Histograms display the non-normal distributions of DART and RC rates and highlight the observed excess zeros, which motivated our use of zero-inflated Poisson models. (Figure 1).

Figure 1:

Distributions of Recordable case (L) and DART rates per 100 FTEs associated with each of the construction subcontracting companies. For clarity, the horizontal axis has been restricted between 0 and 30.

All leading indicators studied in this analysis were significantly correlated with each other (Table 2). Safety Management Systems score was most highly correlated with safety programs (r=0.61), and inversely correlated with safety hazards (r=−0.33). The presence of drug and alcohol screening programs was inversely correlated with both RC rates (r=−0.11) and DART rates (r= −0.10). The fewer safety hazards was also significantly inversely correlated with both RC rates (r=−0.03) and DART rates (r=−0.03). The number of OSHA citations reported showed small, but significant positive correlations with both RC rate (r=0.05) and DART rate (r=0.04).

Table 2:

Correlations (N=2,148)

	SMS	SPE Hazards		SPE Programs		SE Non-Drug and Alcohol		SE Drug		OSHA Citations		RC per 100FTEs		DART per 100FTEs
SMS	1	−0.33	*	0.61	*	0.34	*	0.42	*	0.10	*	−0.04		−0.07
SPE.H	-	1		−0.52	*	−0.25	*	−0.19	*	−0.14	*	−0.03	*	−0.03	*
SPE.P	-	-		1		0.30	*	0.35	*	0.09	*	−0.05		−0.05
SE.ND	-	-		-		1		0.33	*	0.05	*	−0.05		−0.06
SE.D	-	-		-		-		1		0.05	+	−0.11	*	−0.10	*
OSHA Citations	-	-		-		-		-		1		0.05	*	0.04	*
RC per 100FTEs	-	-		-		-		-		-		1		0.77	*
DART per 100FTEs	-	-		-		-		-		-		-		1

^*: p < 0.05

SMS: Safety Management Systems (17 questions)

SPE.H: Safety Program Elements, Hazards (16 questions)

SPE.P: Safety Program Elements, Programs (17 questions)

SE.ND: Special Elements not related to Drug and Alcohol (4 questions)

SE.D: Special Elements related to Drug and Alcohol (4 questions)

RC per 100FTEs: Recordable Injury Case rate per 100 full-time equivalents

DART per 100FTEs: Days Away Restricted or Transferred rate per 100 full-time Equivalents

Crude ZIP models showed consistent statistically significant associations between Safety Management Systems and Drug and Alcohol Special Elements and both recordable case and DART rates (Table 3 and Table 4). Each one-point increase in a company’s SMS score, was significantly associated with a 34% reduction in the odds of a recordable case event occurring (Odds ratio (OR): 0.66, 95% Confidence Interval(CI): (0.57, 0.79)), and a 9% reduction in the rate of recordable cases, if one does occur (Risk Ratio (RR): 0.91, 95% CI: (0.88, 0.94)). Similarly, each one-point increase in SMS score was significantly associated with a 28% reduction in the odds of DART incident (OR=0.72, 95%CI (0.56, 0.91)), and a 9% reduction in DART rate, if one does occur (RR=0.91, 95%CI (0.87, 0.95)). Associations in models that were fully adjusted for all leading safety indicators were generally consistent to those from crude models (Table 3 and Table 4). Trade specific rate ratios are presented in Supplemental Table 2.

Table 3:

Recordable Cases Models (N=2,148)

	Crude				Multivariate				Multivariate + Trade
Leading Indicator	Count Model (RRs)		Inflation Model (ORs)		Count Model (RRs)		Inflation Model (ORs)		Count Model (RRs)		Inflation Model (ORs)
SMS	0.91	(0.88, 0.94)	0.66	(0.57, 0.79)	0.92	(0.89,0.95)	0.78	(0.63, 0.97)	0.95	(0.92, 0.98)	0.81	(0.66, 0.99)
SPE Hazards	1.01	(0.98, 1.04)	1.32	(1.04, 1.68)	1.01	(0.98, 1.03)		--	0.99	(0.97, 1.01)		--
SPE Programs	0.98	(0.93, 1.04)	0.89	(0.61, 1.28)	1.04	(1.00, 1.09)		--	1.00	(0.97, 1.04)		--
SE Non-drug and alcohol	0.89	(0.76, 1.04)	0.33	(0.12, 0.96)	1.00	(0.86, 1.15)		--	0.98	(0.88, 1.09)		--
SE Drug and Alcohol	0.78	(0.72, 0.84)	0.35	(0.21, 0.59)	0.81	(0.75, 0.87)	0.48	(0.28, 0.84)	0.83	(0.78, 0.89)	0.45	(0.27, 0.76)
OSHA Citations	1.03	(0.94, 1.13)	0.20	(0.06, 0.60)	1.07	(0.98, 1.16)	0.15	(0.03, 0.83)	1.05	(0.98, 1.12)	0.12	(0.02, 0.72)
Observed vs Predicted R2					0.72				0.80
Cross-validation R2					0.71				0.74

--: Covariate not included in model

Table 4:

DART Models (N =2,148)

	Crude				Multivariate				Multivariate + Trade
Leading Indicator	Count Model (RRs)		Inflation Model (ORs)		Count Model (RRs)		Inflation Model (ORs)		Count Model (RRs)		Inflation Model (ORs)
SMS	0.91	(0.87, 0.95)	0.72	(0.56, 0.91)	0.90	(0.86, 0.94)	0.86	(0.66, 1.12)	0.93	(0.90, 0.96)	0.90	(0.74, 1.09)
SPE Hazards	1.00	(0.95, 1.05)	1.15	(0.84, 1.58)	1.01	(0.98, 1.03)		--	0.98	(0.96, 1.00)		--
SPE Programs	1.01	(0.92, 1.10)	0.89	(0.61, 1.31)	1.08	(1.02, 1.15)		--	1.03	(0.99, 1.08)		--
SE Non-drug and alcohol	0.88	(0.72, 1.08)	0.39	(0.02, 9.21)	0.99	(0.84, 1.17)		--	0.97	(0.86, 1.10)		--
SE Drug and Alcohol	0.75	(0.70, 0.81)	0.44	(0.28, 0.69)	0.79	(0.74, 0.84)	0.55	(0.32, 0.97)	0.82	(0.77, 0.87)	0.53	(0.29, 0.97)
OSHA Citations	1.01	(0.90, 1.14)	0.83	(0.06, 0.41)	1.07	(0.97, 1.18)	0.12	(0.03, 0.49)	1.05	(0.98, 1.13)	0.75	(0.06, 8.67)
Observed vs Predicted R2					0.58				0.70
Cross-validation R2					0.51				0.62

--: Covariate not included in model

In crude models, each additional Drug and Alcohol Special Element is associated with a 65% reduction in the odds of a recordable case event occurring (OR: 0.35, 95% CI: (0.21, 0.59)), and a 22% reduction in the expected number of RCs (RR: 0.78, 95% CI (0.72, 0.84)). In fully adjusted models, the effect of a one-point increase in SE.ND is slightly attenuated, but still significantly associated with 17% reductions in the odds of a recordable case (OR = 0.83, 95%CI (0.78, 0.89)), and a 55% reduction in the RC rate (RR = 0.45, 95%CI (0.27, 0.76)). DART rate models showed similar effects, where fully adjusted models showed a one-point increase in SE.ND was associated with an 18% reduction in the odds of a DART event (OR = 0.82, 95%CI (0.77, 0.87) ) and a 47% reduction in the DART rate (OR = 0.53, 95%CI (0.29, 0.97) ).

The final ZIP models for RC and DART included terms for each leading indicator, while the zero-inflation portion of the model only included terms for SMS, SE.ND, and OSHA citations. Further adjusting models for trade did not influence the inclusion of any leading indicators. Final cross-validated R² for the multivariable model for recordable cases was 0.71, while further adjusting for trade increased this value to 0.74. Multivariable and trade-adjusted DART models had cross-validated R² values of 0.51 and 0.62, respectively. In a sensitivity analysis, our results remain robust to the exclusion of the seven Canadian companies.

DISCUSSION

Through our cross-sectional analysis of this CSAP database, we have identified several organizational leading safety indicators that are associated with safety performance in the construction industry, as measured by recordable cases and days away restricted or transferred. More safety management systems were inversely associated with RC and DART rates, and companies with higher SMS scores were more likely to report no incidents. Similarly, the inclusion of drug and alcohol policy elements was associated with improved recordable case and DART rates.

Our study is unique in that within the database we observed a (inverse) dose-response relationship between safety management systems and injuries when they occur. The finding that safety management systems is associated with the injury data further supports the need for such organizational commitment to safety.⁶ The efforts to put forth safety management programs has been supported by many safety advocates including the OSHA proposed program rule of 1999 ¹⁹and their white paper in 2012. ¹¹ The questions in this section capture a range of components that are often referenced as essential elements for safety and health management systems. ⁹ These questions assess management commitment through evidence of, and an involvement plan for, organizational safety and health goals. Other factors include inspection and hazard recognition, infrastructure for worker involvement through committees and feedback opportunities, accountability within all levels of management, evaluation plans, incident investigations, and emergency plans. Overall the recorded associations here are in support of having strong safety management systems in place in these contractors.

The fact that the safety program scores were not significantly related to the outcomes was surprising and may be due to several reason. One is the belief that simply complying with OSHA standards may not be enough to reduce injuries ²⁰. Another is the existence of these programs and their hazards may vary a great deal from company to company perhaps due to the existence of different trade-specific hazards. As a result of this variability, a tally of the number of programs may not be enough to see relationships within this sample. However, understanding which programs are in place may help users of pre-qualification data to plan appropriately when dealing with specific contractors for specific projects. It also helps to ensure appropriate compliance issues for a general contractor and an owner when hiring contractors.

The finding that having drug and alcohol screening program reduced the rate of reported injuries is consistent with other studies that have examined such associations; however, the quality of these studies prohibits conclusions about cause and effect of such programs ²¹. In their review Pidd and Roche ²¹state the need for more rigorous research to evaluate such programs and that better dissemination of the current evidence (or the lack there of) is needed. Given the expense of implementing a drug and alcohol screening program, companies with such policies may have better financial performance, and thus the presence of these programs may be associated with other unmeasured company characteristics. The horse may have left the barn as more and more companies are using drug testing as part of their safety programs based on cross-sectional studies including our own. ^22,23 Given the high rates of alcohol and drug use among construction workers^24,25 the industry should consider employing substance abuse programs for their employees. The use of peer-based programs along with testing can reduce injuries by 33% and is very cost effective. ²⁶

Our study does have some limitations. First, the cross-sectional design of our analysis does not allow us to identify temporal or causal relationships between the organizational programs and policies around safety, and the lagging safety outcomes. This design was chosen as it best fit our goal of developing a safety prequalification tool using current leading indicators, and it accommodated the data we had access to. Additionally, the leading indicators that we investigate (such as safety management systems and job hazards), are likely to be fairly stable over time. Also, because of the self-report of most included measures, if there are inaccuracies in these metrics it is likely that they will be directional. We assume that companies would be more likely to overstate their safety programs and perhaps underreport their injuries. Further, many companies may rely on non-contract employees and other contractors to perform work, which could also bias results regarding the relationships between company policies and injury outcomes, as those non-contract employees may not be fully aware of all company safety policies and programs and when injured, may not report it to the prime contractor. Finally, the results of this work may not be generalizable to contractors that do not voluntarily participate in a safety-prequalification program, such as ConstructSecure, as well as smaller companies with less than 10 FTEs in billable hours.

Overall these data provided a unique opportunity to explore associations between leading (safety management systems and programs) and lagging (injuries) indicators in a large number of construction contractors across the United States and Canada. The associations documented here through using statistical approaches accounting for the large number of zeros in the data support the need for the implementation of safety management systems across the broad range of contractors.