The gap between tools and best practice: an analysis of safety prequalification surveys in the construction industry

Abstract

This paper characterizes safety prequalification surveys currently in use in the construction industry to identify approaches that include leading indicators of worker safety performance. We collected prequalification surveys available in the public domain from internet searches, construction company websites, published literature, and construction industry partners. We utilized a conceptual framework, based on safety theory and best practices, to categorize survey questions. Fifty-two prequalification surveys were identified containing 112 unique questions. Most included questions related to lagging indicators (83%), safety management leadership (75%), and worker training (60%). Safety management system elements such as hazard prevention and control, program evaluation and improvement, coordination and communication were notably absent in 90 percent of the surveys. There was little consistency in the surveys available concerning leading indicators of safety. Only a small number of surveys currently in use incorporate all the elements of best practices associated with robust safety management systems.

Introduction

In an effort to reduce risk, building project owners, general contractors, and construction management companies utilize subcontractor prequalification procedures as part of the bidding process to select contractors and subcontractors for a construction project.^1–3 These procedures can evaluate numerous criteria to assess the capabilities of the subcontractor candidates.^{3, 4} In addition to cost, quality, and time, health and safety performance is one of the key attributes in subcontractor selection.^{2, 5, 6} Subcontractor selection is also claimed as one of the most effective risk reduction elements among safety management programs.⁷ However, the actual practices and criteria of the procedures to assess a subcontractor’s safety have not been described or characterized, especially within the framework of best safety practices.

In terms of assessing a company’s safety performance, examining injury records would appear to suffice; however, there are several issues associated with simply measuring injury outcomes that limit the information they provide about the work environment.⁸ Counting past injuries assumes that all of the injuries that occurred were actually reported. Underreporting injuries occurs frequently, especially in smaller and in construction companies.^{9, 10} In addition, assuming the same injury rate per full time equivalent, injuries will occur less frequently in smaller companies compared to larger companies. The other measure frequently included in the prequalification process is the Experience Modification Rate (EMR), an insurance premium indicator based on the assumption that prior (3 years) injury claims will predict future costs. Historically, health and safety records and EMR have been treated as effective prequalification criteria for construction project success.¹¹

In contrast, leading indicators are antecedents of injuries and seek to identify the upstream root causes of work-place incidents. They include the presence of uncontrolled hazards as well as organizational policies, programs, and practices that monitor, control, and eliminate these hazards. These leading indicators are present every day within an organization and can be measured and hence may be better indicators of safety performance. There is a large range of leading indicators described in the research literature including management commitment, employee involvement, subcontractor management, safety climate, and many others.^{7, 12–14} From a best practice point of view these indicators follow common themes that can be organized into a similar structure.¹⁵ For example, the Occupational Safety and Health Administration’s (OSHA) proposed injury and illness prevention program (I2P2), which was withdrawn and revisited in a 2012 white paper,¹⁶ describes major elements of successful programs, which include management leadership, worker participation, hazard identification and assessment, hazard prevention and control, education and training, and program evaluation and improvement.

The interest in prequalification surveys that use leading indicators has gone up considerably in recent years.¹⁷ Until recently, the methods to examine a contractor’s organizational policies, programs and practices had long been done through non-standardized, time consuming and costly procedures such as one-on-one interviews during the bidding and contracting process. Organizational a surveys that provide a measures of leading indicators such as the organizational safety management systems can help initiate and contribute to the process.

The goal of this paper was to characterize, describe, and document current prequalification surveys to assess a subcontractor’s safety performance. Our approach was to find all publicly available surveys with a focus on comprehensive programs that address safety management systems. These surveys were identified through discussions with safety professionals and researchers, literature reviews, and internet searches. From the identified tools, we catalogued all relevant safety questions first by leading and lagging indicators and then into the framework of safety organizational policies, programs, and practices. Through this process, our other goal was to identify gaps between prequalification survey metrics and leading indicators grounded in organizational theories.

METHODS

Identifying tools and procedures

We first identified prequalification surveys currently in use in construction firms in the United States. These surveys were developed and used by general contractors, owners, as well as third party companies providing the prequalification service for owners and subcontractors.

To identify surveys used by commercial contractors, we requested prequalification surveys from large commercial contractors in the United States or downloaded them from their website. The largest 100 commercial contractors were identified through 2015 construction industry reports.¹⁸ The number of companies was reduced as a result of company mergers and bankruptcies, and a total of 86 companies remained. Additional companies were added to the list from area construction firms and their parent companies identified through interviews of local safety managers. An internet search ensued to identify company websites and the presence and availability of prequalification surveys online for subcontractors seeking to obtain work with this firm. When a barrier existed, such as a required password, the company was both emailed and telephoned to request access to the prequalification surveys in support of our research study.

We identified prequalification surveys utilized by governmental (federal, state, municipal) agencies and non-profit organizations. These surveys were identified through internet searches or interviews with construction safety advocates who worked with government agencies. Two third-party providers of commercial construction prequalification provided us with the questions within the surveys that they include in their services.

We also sought to identify surveys through a search of the peer-reviewed safety literature; however, the articles found did not provide enough detail to be included in the review. The search applied a set of keywords (Appendix A) to ScienceDirect, EBSCOHost, Web of Science, Pubmed, GoogleScholar and ProQuest databases. We found five articles that included safety elements. The specific safety-oriented questions were absent in these surveys; hence, were not included in this review.

Identifying, describing and categorizing questions

Once collected, we identified all the unique questions contained in all the tools and procedures. To do so, we created a list of questions from the longest survey. Then we examined each question from the other surveys, comparing the item to the list of questions to see if the question or similar question was already present. If not, it was added to the list. Once we had a complete list of question, we identified the use of each question across all prequalification surveys. Once identified, we categorized each unique question using the hierarchy structure (Figure 1).

Fig. 1.

Document hierarchy of safety and health

The top level of the structure is policies, which ensure that all important factors such as finance, quality, schedule, environment, safety, and health are aligned with applicable regulatory and corporate requirements as well as the values of the organization. Procedures, the second level, are specific methods to meet the policy requirements in day-to-day operations of the organization. Work instructions, the third level, are the specific activities related to procedures and programs in the form of technical manuals or guidelines. The lowest level, records, includes performance and inspection results.¹⁹ From the safety and health perspective, the corresponding structure of documentation consists of safety management systems (SMS), specific safety programs, and safety outcomes.

Within the safety management systems questions, we further categorized the questions into sub-categories based on elements of successful programs identified in the OSHA proposed Injury and Illness Prevention Program (I2P2). These sub-categories included management leadership, worker participation, hazard identification and assessment, hazard prevention and control, education and training of all levels of employees, program evaluation and continual improvement, and coordination and communication for employers on multiemployer worksites (Table 1).

Table 1.

Sub-Categories for safety management systems derived from successful programs

Sub-Category	Description
Management leadership	Management provides the leadership, vision, and resources needed to implement an effective safety and health program.
Worker participation	Workers are involved in establishing, operating, evaluating, and improving the safety and health program. All workers at a worksite should participate, including those employed by contractors, subcontractors, and temporary staffing agencies.
Hazard identification and assessment	A set of processes used to identify and evaluate both existing and potential hazards on a worksite.
Hazard prevention and control	The methods used to control or eliminate the hazards are identified. Effective controls can protect workers from hazards; help avoid injuries, illnesses, and incidents; minimize or eliminate safety and health risks; and help employers provide workers with safe and healthful working conditions.
Education and training	Activities for informing workers and managers about workplace hazards and controls so they can work more safely and are more productive. In addition, providing workers and managers with a greater understanding of the safety and health program itself, so that they can contribute to its development and implementation.
Program evaluation and improvement	Safety and health program should be evaluated initially to verify that it is being implemented as intended. Periodically or annually, employers need to step back and assess what is working and what is not, and whether the program is on track to achieve its goals.
Coordination and communication	Within a multiemployer environment, companies need to consider how work and safety activities affect the safety of other employers and workers at the site.

For the safety program category, we included questions regarding specific hazard and control programs. These programs meet compliance with 29 CFR Part 1910 “Occupational Safety and Health Standards” and Part 1926 “Safety and Health Regulations for Construction.” Examples of such programs include fall protection; ladder safety; signs, signals, and barricades; materials handling and many other programs.

Finally, we identified questions or data within safety outcomes based on the lagging indicators typically used to describe a company’s safety performance. These included data from OSHA’s Form 300A, Summary of Work-Related Injuries and Illnesses, which includes recordable cases; days away, restricted or transferred (DART); fatalities; and injuries and illnesses. In addition to OSHA’s Form 300A, first aid injuries, near misses, jobsite safety inspections, EMR, and worker safety perception surveys were also considered as safety outcomes²⁰. Once categorized, we compiled the prevalence of question categories across the different surveys to identify gaps across this framework in the current tools.

Results

A total of sixty-three prequalification surveys that evaluated contractor safety were identified. Some surveys assessed subcontractor safety performance post-hire and were not intended for use during the prequalification process. Subsequently, we removed these surveys leaving a total of fifty-two. On average the surveys had fourteen questions with over half of the surveys having more than ten. The longest survey had sixty-three questions.

Among these fifty-two prequalification surveys, a total of 112 unique questions were identified (Appendix: Table A-I). Fifty of the 112 questions identified were leading indicators associated with safety management systems (Table 2). There were fourteen questions related to worker safety training. Other questions addressed leadership and hazard identification and assessment. Questions categorized into management leadership included questions regarding a written safety policy, defining safety goals, allocating resources, and safety accountability assignment²¹. The corresponding questions identified in the surveys included: Do you have a safety policy statement endorsed by top management? Do you have corporate safety goals and objectives? Do you have a defined budget for safety, and Do you have safety a accountability program? Furthermore, questions like Do your supervisors hold safety meetings? and Do you employ full time safety supervisors on all job sites were also presented in half of the surveys. The responses to almost all of the questions were simply yes/no.

Table 2.

Prequalification questions distribution

Categories	Number of relevant questions	Proportion of identified questions
Safety Management Systems (Appendix: Table A-I)	50	44.6%
Management Leadership	11	9.8%
Worker Participation	6	5.4%
Hazard Identification and Assessment	8	7.1%
Hazard Prevention and Control	4	3.6%
Education and Training	14	12.5%
Program Evaluation and Improvement	3	2.7%
Coordination and Communication	4	3.6%
Specific Safety Programs (Appendix: Table A-II)	39	34.8%
Safety Outcomes	6	5.4%
Other(Appendix I)	17	15.2%
Human Resource	11	9.8%
Environmental	3	2.7%
OSHA Partnership	3	2.7%

The lagging indicators included seven safety outcomes: OSHA citations, EMR, recordable cases, fatalities, days away, restricted or transferred (DART), and violations. Thirty-nine questions addressed specific safety and hazard control programs (Appendix: Table A-II). Many of these were related to adherence to specific OSHA standards, such as lockout/tagout (LOTO), hearing conservation programs, and falls.

In addition, prequalification surveys included questions related to human resources hiring policies, environmental programs, and OSHA partnerships. The questions found in the human resource category asked about return-to-work programs, drug and alcohol policies, sexual harassment programs, and work fitness assessments. The questions related to the environmental category asked about environmental protection programs and Environmental Protection Agency (EPA) violations. The last category concerned participation of OSHA programs such as the Voluntary Protection Programs (VPP), the Safety and Health Achievement Recognition Program (SHARP), and the OSHA Strategic Partnership Program (OSPP).

The fifty-two prequalification surveys came from six different types of users (Table 3). About half were from commercial construction companies and were available on their webpages. Twenty-eight PQ surveys from construction companies were identified. Most of those companies were medium to large, and nine are in the top one hundred construction companies by revenue in the United States²¹. Most public agencies with available prequalification were departments of transportation, which accounted for 15 percent of the surveys identified. An additional 27percent of source were other owners, such as energy companies, universities, and public agencies. Two prequalification surveys were provided by third-party companies.

Table 3.

The number of surveys distributed by user type.

Source of Prequalification survey	Number	Percent
Construction Company (GC or CM)	28	53%
Transportation (owner)	8	15%
Energy Company (owner)	6	12%
Academic Institutions (owner)	5	10%
Public Agency (owner)	3	6%
Third-Party Service	2	4%
Total	52

The overall categories included in the surveys varied from survey to survey (Tables 4–6). No single prequalification survey included all twelve categories. In terms of safety management systems, the prequalification surveys differed across the six user types with third-party agencies having the most questions (Table 4). Twelve of the fifty-two prequalification surveys (23%) did not include questions concerning safety management systems (Table 4). Less than 10 percent of prequalification surveys had questions about hazard prevention and control, program evaluation and improvement, coordination and communication.

Table 4.

Number of safety management system questions

User Type	Median	Maximum	Number with no SMS
General Contractors (28)	5	13	3 (11%)
Transportation Agency (10)	0	8	6 (60%)
Energy company (6)	7.5	17	0
Academic institution (5)	4	10	2 (40%)
Public agency (3)	1	8	1 (33%)
Third-party Agency (2)	18.5	21	0
Total	5	21	12 (23%)

Table 6:

Number (and percent) of surveys with non-safety management systems questions

Most surveys (83%) included items pertaining to lagging indicators or safety outcomes (Table 6). Ninety-three percent of the surveys from construction companies included safety outcomes. However, only 50 percent of surveys required by transportation agencies had questions about safety outcomes, and the remaining 50 percent (four tools used by departments of transportation) that had no question related to any of the twelve safety categories.

Discussion

The main goals of this paper were to characterize the prequalification surveys currently in use and to identify gaps between these surveys and best practices grounded in organizational safety theories. We compiled a list of questions from the prequalification surveys and characterized them with respect to lagging or leading indicators within a best-practices framework. Almost every survey included lagging safety indicators such as injury rates, OSHA citations, and EMR. Many surveys also included leading indicators; however, there was a large number of surveys that did not include any questions regarding safety management systems.

Lagging indicators were reported fairly consistently across the surveys. Measurements such as EMR and injury rates are well defined concepts and metrics. Standardized methods exist to calculate and to report them in a prequalification survey. In contrast, there were many more and different questions regarding leading indicators.

The large variability in the design and wording of questions addressing leading indicators across the different surveys suggests that there is little consistency in how to measure them when and if a survey attempts to capture these factors. This may be due to several issues including different attitudes and beliefs about these indicators. Questions regarding training elucidate this idea. For example, a typical question about training and education is Do you have a new employee-training/orientation program? But training is not only for front-line workers as another question from a different survey asks: Does your company conduct additional health & safety training for supervisors, or foremen?

While many of the prequalification surveys did contain leading indicators, there were some noticeable gaps when compared to best practices and essential components of safety and health programs. The items that received little attention in many of the surveys were program evaluation and improvement, employee involvement, and safety communication. Safety communication is an important aspect of establishing a good safety climate.^{22, 23} Additionally, communication and coordination are essential within multiple employer sites to ensure that all organizations are on board with setting safety as a value.^{24, 25} Furthermore, worker participation and involvement has demonstrated improved outcomes for safety.^{26, 27} Finally, none of the surveys describe measuring worker perceptions of the work environment that can be used to measure the efficacy of safety communications as well as provide an evaluation of the safety program.

Nearly all of the surveys used by construction companies included lagging indicators of safety performance. These indicators provide important data, especially when they indicate higher injury rates.²⁸ Lagging indicators also provide motivation to improve safety management systems and can inform through root cause analysis how to improve the systems. Using both leading and lagging indicators can provide a better overall assessment of a company’s performance.

The differences observed across the different types of organizations (contractors, third-party and owners) may also be reflective of their different priorities. The surveys from the two third-party organizations had the largest number of questions related to safety management systems. These organizations appear to be leading the way for including more leading indicators of safety.. The surveys from the energy companies may be more associated with these groups being more focused on process safety.²⁸ Similarly, departments of transportation had very few questions regarding safety management systems.,.

Only a few prequalification surveys were described in the peer-reviewed literature, and their emphasis was on financial and performance measures in lieu of safety. There were no published studies demonstrating the capability of these surveys to predict the future safety performance of the companies. While it may be unclear if such surveys actually provide a prediction, they may provide a method to assess a company’s current capabilities and identify weaknesses that can improve worker safety and health.²⁹

From a theoretical point of view, leading indicators should by definition predict future safety performance. However, the variability in the types of leading indicators included in these organizational surveys indicate that there is not consensus on their capability of predicting future performance. Larger scale studies are needed to examine which factors can indicate future safety performance and, if so, what might be the best way to measure them.

By no means is this descriptive review comprehensive. The survey items evaluated were available in the public domain or obtained through research partners. Other tools could not be obtained due to internet firewalls, lack of response from construction companies contacted by telephone, or due to proprietary constraints. Therefore, other tools may have different items and constructs not captured in this review.

The overall assumption of this study and in the commercial construction industry is that contractor prequalification aids in the selection of safer contractors. Additional research to support this assumption is merited. Our anecdotal experience with general contractors suggests that a thorough prequalification process enables them to leverage the quality of the subcontractor’s safety programs. Thus, the prequalification process helps the subcontractors to identify lapses in their safety programs and presents opportunities for enhancement. We did not systematically explore how contractors themselves view these different organizational surveys and if they feel the surveys are useful.

Furthermore, prequalification surveys can be one part of the prequalification process. Other methods exist, such as the review of safety manuals, interviews, and detailed meetings between contractors. Using multiple assessment methods, especially when their results converge, provides a more comprehensive overview of a company’s safety performance. Nonetheless, organizational surveys that include leading indicators can provide a good overview of an company’s commitment to safety and health.^{30, 31}

Conclusion

Current and available safety prequalification surveys include a mixture of leading and lagging indicators. Most include measures of injury rates and company EMR. Many did include some measure of a firm’s safety management systems; however, several elements of best practices were notably absent in many of the surveys. Only a couple surveys incorporating all the elements considered important leading indicators in best practice guidelines. Policy initiatives by private and government sectors to require safety prequalification processes for their construction contractors¹⁷ or through the issuing of building permits,³² should include more leading indicators in their assessment of contractors.

Table 5:

Number (and percent) of surveys with questions in related to each of the sub-categories of safety management system questions

Biographies

Kang-Hung Liu is a doctoral candidate in Department of Industrial Engineering and Engineering Management National Tsing Hua University, Hsinchu, Taiwan, ROC. His research identifies potential work risks using safety climate technique based on the organizational ergonomics approach. He holds industrial engineering degrees from the National Tsing Hua University.

Jamie Tessler is Occupational and Environmental Health Training Manager at Northeastern University. She has 29 years of experience researching, designing and conducting occupational health and safety training programs, with the last 20 years focused on ergonomics programs in various industries. She holds degrees from the University of Massachusetts Lowell, Hunter College, and Hampshire College.

Lauren Murphy is an assistant clinical professor in the Department of Physical Therapy, Movement, and Rehabilitation Sciences in the Bouvé College of Health Science at Northeastern University. As an Industrial/Organizational Psychologist Dr. Murphy has advanced knowledge in quantitative research (e.g., survey development, testing/administration), qualitative research methods (e.g., focus groups, interviews), and data analysis. She holds degrees from Portland State University and the University of Massachusetts Amherst.

Chien-Chi Chang is a professor in the Department of Industrial Engineering and Engineering Management at the National Tsing Hua University in Hsinchu, Taiwan.

Jack T. Dennerlein is a professor in the Department of Physical Therapy, Movement, and Rehabilitation Sciences in the Bouvé College of Health Science at Northeastern University. He is also adjunct professor of ergonomics and safety at the Harvard T. H. Chan School of Public Health and associate director for their Center for Work, Health, and Wellbeing. Dr. Dennerlein holds degrees from the State University of New York at Buffalo, Massachusetts Institute of Technology, and the University of California, Berkeley.