Development of the St. Louis Assessment of Fall Risks at Residential Construction Sites

Abstract

We describe the development and pilot testing of the St. Louis Assessment of Fall Risks, a worksite audit to assess fall prevention safety practices on residential construction sites. Feedback from carpenters and safety instructors regarding work tasks associated with falls from heights as well as surveillance data were used to develop the audit instrument. Domains of the audit focus on the framing process, including general safety climate/housekeeping, floor joist/sub-floor installation, walking surfaces/edges, wall openings, truss setting, roof sheathing, ladders, scaffolds, and personal fall arrest equipment. The audit was tested at sixteen residential construction sites documenting excellent inter-rater reliability (kappa = 0.93). Results suggest that the audit has good face and content validity and is a reliable instrument for measuring fall safety risks at residential construction sites. It is practical, easy, and safe to administer, making it a potentially useful instrument for field research as well as regular safety monitoring by foremen and crew.

1. Background

Construction workers encounter many work situations that place them at risk of injury on a daily basis. In 2006 the construction industry experienced more fatalities than any other industry. ¹ Falls account for over one-third of all construction fatalities; however, in residential building construction, almost one-half of the fatalities are due to falls. Falls from heights to a lower level rose 39% among residential construction workers in the two-year period between 2003 and 2005. ¹ Unfortunately, injuries to construction workers are probably higher than the United States Bureau of Labor Statistics’ (BLS) suggest. ^2-6 This under-estimation likely results from a combination of factors including under-reporting of injuries, and the over-estimation of person time at risk; many in the construction trades do not work 2000 hours per year. ⁷ In the construction industry, Welch ⁸ and colleagues found that at least 14% of the employers in Massachusetts misclassify workers as independent contractors; therefore injuries sustained to these workers are not recorded by the BLS. In addition, BLS data does not capture the experiences of very small employers, a factor of particular relevance in the homebuilding industry.

Although work occurs at lower elevations in residential construction compared to commercial, the exposure to uncontrolled fall risks may be greater than at commercial sites. On site safety professionals are a rarity, particularly among small residential contractors, and safety innovation in residential construction has lagged behind commercial construction. For example, during home construction, solid anchor sites to fasten fall protection devices do not exist during the early construction phases ⁹ and often do not exist at later stages of construction. The residential work environment is ever changing and the work crews are often small and dispersed. The ratio of experienced to inexperienced crew can be low and there is an increasing immigrant face in residential construction in the U.S. as well. ⁷

Contemporary home design may expose construction workers to greater risks than in past years due to steeper pitched roofs, open floor plans, vaulted ceilings, and other design elements requiring greater work at heights. New home construction is a very fast-paced and competitive sector of the construction industry, with significant time pressures on most jobs. Recognizing that some of the building practices described in 29 CFR 1926 OSHA Construction Industry Regulations ¹⁰ are difficult to comply with at residential construction sites, OSHA released Directive STD 3.1A Safety and Health Interim Residential Guidelines ¹¹ to describe alternatives to conventional fall protection when working at heights 6’ or greater. These guidelines are more realistic to protect workers from falls from height in this sector, but since guidelines do not have the same legal enforcement as regulations, there is less incentive for adherence.

In order to prevent falls on residential construction sites, the factors contributing to falls and the fall risks at the worksite must be identified. Much has been learned from targeted active and passive surveillance activities in residential construction; specifically, based on the nature of the work, carpenters work from many elevated work surfaces and they essentially fall from all of them. ^{7, 12} However, assessing the frequency of exposures associated with falls is more challenging.

Worksite audits are a means of evaluating current levels of exposure to fall (and other) hazards, but most of the audit instruments identified in the literature are designed to assess commercial construction sites. ^{13, 14} The building materials, construction methods, and equipment vary significantly between commercial and residential sites; therefore commercial audits do not address the fall risks present at most residential sites. Bigelow and colleagues’ ¹⁵ designed a residential construction audit to assess exposures and priorities for training and incentive programs intended to reduce injuries and illnesses among home builders in the Denver area. While their audit assessed common safety hazards at residential sites, it did not focus on fall risks. Lipscomb and colleagues ¹⁶ used an audit on residential sites as part of an active surveillance effort designed to assess some broad measures of safety climate and to identify circumstances surrounding recently reported fall incidents, however this tool did not provide a comprehensive assessment of fall risks The reliability and validity of existing audit instruments has not been described. The purpose of the current research was to develop a reliable and valid observational audit to comprehensively assess the level of fall risks on residential construction sites. We sought to design a simple and practical tool that could be administered by construction workers at worksites.

2. Methods

Audit development

This research was conducted as part of a larger project focused on improving and subsequently evaluating fall prevention training for apprentice carpenters. The work took place in metropolitan St. Louis, Missouri, U.S.A., which has the largest unionized residential workforce in a single geographic area in the country. The audit development process included several steps. First, we reviewed existing federal regulatory standards designed to prevent falls from height, including 29 CFR 1926 OSHA Construction Industry Regulations and Directives STD 3.1A Safety and Health Interim Residential Guidelines ^10-11, as well as existing surveillance data specific to falls in residential carpentry. ^{12, 16-19} Focus groups helped us identify the current fall prevention work practices reported by apprentice carpenters and their fall prevention concerns. ²⁰

We next searched the safety, prevention, injury management, and construction literature to identify existing self-report and observational instruments that measure worker behaviors and safety conditions, including Halperin and McCann’s scaffold audit of commercial construction sites. ¹³, West Virginia University’s pocket PC fall protection audit for commercial construction sites ¹⁴, Lipscomb’s residential worksite audit ^{16, 19}, Stafford and Cameron’s worksite audit ²¹, and Bigelow’s Home Safe behavior-based safety audit. ^22,15 A list of 24 potential domains for the worksite audit was generated by integrating feedback from the focus groups ²⁰ , along with variables identified in the literature review, from review of existing regulations ^10-11, and audit instruments. ^{13-16, 19, 21-22}

Eight journeymen carpenters with expertise in residential construction, worksite safety, and apprentice training, independently rated the 24 domains for importance on a 5-point scale (very low, low, medium, high, and very high). We computed the mean ratings of importance for each domain and calculated the intra-class correlation coefficients to assess the inter-rater reliability between the subject matter experts on the importance of each domain. Domains with the highest mean ratings of importance were included on the audit tool. Items specific to fall prevention were developed for these domains based upon previous audit instruments, construction standards and guidelines, focus group feedback, and subject matter expert input. Decisions regarding scaling were made based upon methods used in other worksite audits. The subject matter experts also helped us develop detailed procedures for the administration and scoring of audit items, which were outlined in an administration manual. After development of the audit and administration manual, we proceeded to pilot testing of the audit, named the St. Louis Assessment of Fall Risks (SAFR).

Pilot Audits & Establishment of Inter-rater Reliability

Two experienced carpenters were recruited as audit administrators for piloting the SAFR. These auditors each had over twenty years of prior work experience as journeymen carpenters, experience evaluating worksites using research protocols, and knowledge of governmental health and safety standards, including construction OSHA 500 training. We trained these auditors in administration procedures for the SAFR, including the procedure for contractor informed consent, safety at the worksite, item administration, and scoring criteria. A convenience sample of large (over 250,000 carpenter hours/year) residential contracting companies that employ apprentices and carpenters trained at the St. Louis carpenter apprenticeship program were identified as potential participants in the pilot testing. We contacted these contractors, explained the project, and received informed consent. If requested, we allowed a contractor representative to accompany the audit team during pilot audit testing. Since exposure to heights is more frequent in the framing process than other phases of construction, worksites performing framing operations were targeted.

The audit pilot testing team included the two experienced carpenters and one of the audit developers (VK). The team followed the audit administration and scoring procedures described in the administration manual. Each auditor simultaneously and independently completed the audit. Work tasks were not repeated or simulated for audit purposes. Only actual work processes, worker behaviors, and construction phases observed during the audit were assessed and scored. After the audits were completed, the audit team discussed their individual ratings to ensure that the scoring procedures were followed consistently and to identify modifications that needed to be made to the audit and administration manual to improve consistency and clarity.

Data were encoded and entered into spreadsheet using double entry to avoid errors. Data analysis was performed using SPSS (version 14.0, SPSS Inc., USA). Descriptive statistics were used to summarize demographic and audit items. In order to establish inter-rater reliability, we computed the kappa statistic to measure level of agreement between the two independent journeymen auditors’ for all audit items and also for only the items observed during the audits. We also computed the inter-rater reliability between journeymen auditors for both versions of the audit and between each journeyman auditor and the audit developer (VK). The Institutional Review Board at Washington University School of Medicine approved all research procedures including the worksite audit, recruitment script and informed consent procedures.

3. Results

The Audit

The nine domains receiving the highest mean ratings of importance by the subject matter experts were included in the final SAFR instrument. These include general safety climate and housekeeping, floor joist and sub-floor installation, walking surfaces and edges, wall openings, truss setting, roof sheathing, ladders, scaffolds, and personal fall arrest equipment. The eight subject matter experts showed excellent agreement in their ratings of the importance of the domains of the audit, with an intra-class correlation of 0.87 (95% CI 0.77-0.94). Mean ratings for the variables ranged from 1.63 to 5.0. Domains with mean ratings in the top two tertiles were included on the audit. The audit instrument and administration manual were modified once during the pilot testing, with items deleted due to difficulty in rating, reworded for clarity, or added due to importance. The first twelve audits were performed using the initial version of the audit, and the last four used the final version of the audit. The specific items on the final version of the audit are listed in table 1.

Table 1:

St. Louis Audit of Fall Risks Pilot Observation Results (n=16)

Domains / Items	Observed Met	Observed Not Met	Not Observed
General Safety Climate & Housekeeping
All workers wear hard hats	9	7	0
All workers wear safety glasses/eye protection	13	3	0
Pathways/access point free of materials/debris	10	5	1
Floor Joist & Sub-floor Installation
Floor joists are set/secured from ladder/ground/ scaffold, not beam/top plate	2	1	13
First sheet of sub-floor installed from ground/ ladder/scaffold, not joist	1	0	15
Workers install subsequent sub-flooring standing on established deck, not joist	1	0	15
Walking Surfaces & Edges
Walking surfaces >6’ above lower level are protected by guardrail or erected wall	6	1	9
All walking surfaces >6’ above lower level that are not protected by guardrail or wall are identified with a warning line painted 6’ from leading edge	2	2	12
All areas with unprotected walking surfaces are designated Control Access Zones; boundary is clearly marked, workers monitored, and access restricted	1	3	12
Guardrails protecting openings are constructed sturdily (200# force) with 2 x 4’s, top rail 42”, mid-rail 21”	5	4	7
Holes >6’ above lower levels are covered; a hole is a gap > 2” in a pathway commonly accessed	0	0	16
Stairwell has sturdy handrail on at least one side	1	1	14
Wall Openings (window/door)
Walls > 6’ above lower levels that have openings with bottom edge <39” from floor are protected by guardrails	5	0	11
For walls >6’ above lower levels, guardrails are constructed sturdily (200# force) with 2 x 4’s, top rail 42”, mid-rail or lower wall at 21” from ground	4	2	10
Truss Setting
Lay out for trusses is performed from sub-floor or ladder, not from top plate	0	0	16
For walls up to 8’, trusses are installed from ladder or scaffold along interior wall	0	0	16
For walls >8’, first 2 trusses are set from ladder or scaffold along interior wall	0	1	15
For walls > 8’, common trusses are set & secured from ladder, scaffold, or interior top plate using stable truss for support; not standing on exterior top plate	1	0	15
Worker removes chain/webbing from truss while standing on ladder/secure truss	0	1	15
Workers lift boards/stand trusses only when using stable truss for support	0	0	16
Roof Sheathing
Bottom row of roof sheathing installed from truss web, ladder, or scaffold	0	0	16
Workers install slide guard on 1st row of sheathing before installing next row	1	0	15
Slide guards are ≥ 2 x 4 boards, bottom guard is perpendicular to sheathing	2	0	14
Slide guard intervals: pitch up to 9 in 12 at 13’ intervals, > 9 in 12 at 4’ intervals	2	0	14
Slide guards are installed across full width of the roof & on all sides of roof	3	0	13
Roof is clear of sawdust, debris & dew/snow/ice if workers are on roof	2	0	14
If slide guards are not used, fall arrest is properly used by all workers on roof	0	0	16
Ladders
Straight, free of cracks/broken parts, free of mud/ice, side locks on step ladder	12	0	4
Set up on level & solid base, securely set at the bottom	8	3	5
Extension & job-built ladders are secured at the top in appropriate manner	1	5	10
Step ladders fully opened & side locks engage, not leaned on structure like straight ladder	6	3	7
Extension & job-built ladders are set at correct angle of 1:4 ratio (palms of hands reach side rails if toes at base)	6	0	10
Extension & job-built ladders extend 3’ past upper landing surface	4	1	11
Workers do not work from top 3 rungs of extension & job-built ladders & top rung or platform of step ladder	7	1	8
Workers maintain 3 points of contact while climbing ladders & do not carry supplies while climbing ladder	2	3	11
Workers always keep belt buckle region within side rails & both feet on ladder	4	1	11
Workers drag excess mud off of shoes before climbing ladder	0	0	16
Scaffolds
All Scaffolds: Fall protection used if > 10’ tall (personal fall arrest/guardrail/net)	1	0	15
Ladder Jack: Ladders are safely secured at both the top & bottom	1	1	14
Ladder Jack: Maximum height is 20’	2	0	14
Ladder Jack: Walk board is 12” wide	2	0	14
Ladder Jack: 3rd ladder present to access if walk board is outside of ladders	0	0	16
Ladder Jack: If access ladder is present, it extends 3’ above walk board	1	0	15
Pump Jack: Set on secure/stable base	1	0	15
Pump Jack: 4 x 4 posts are properly braced & secured to building	1	0	15
Pump Jack: Maximum height is 50’	1	0	15
Pump Jack: Workers only disengage 1 brake at a time	0	0	16
Job-Built: Set up on level, stable footing	1	0	15
Job-Built: Platform is secure & stable	1	0	15
Job-Built: Platform is 18” wide	1	0	15
Personal Fall Arrest
Workers wearing fall arrest use approved harness that is worn properly	2	0	14
Lanyard is properly attached to secure anchorage point/lanyard length is correct	2	0	14

Items on the audit describe the safe levels of performance based upon subject matter expert opinion and applicable federal safety standards and guidelines ^10-11. All items on the audit are based on these federal standards and guidelines, except for one item, which is based on ladder manufacturer recommendations. For example, the criteria for controlled access zones and truss setting are based on Directive STD 3.1A Safety and Health Interim Residential Guidelines ¹¹, and the criteria for the ladder and scaffold domains are based upon the 29 CFR 1926 OSHA Construction Industry Regulations. ¹⁰

After reviewing other audit instruments, we decided to use an all-or-none scoring method for reliability and ease of administration. Bigelow and colleagues ¹⁵ and Stafford and Cameron ²¹ successfully utilized this scoring method in their worksite audits. Scoring is dichotomous, with “observed – safe,” indicating that each observation of the item met the criteria for safety. If at least one observation does not meet the safety criteria, it is scored “observed – not safe”. “Not observed” is marked if an item is not observed during the audit due to the phase of construction or work activities occurring at the time of the audit. In addition to these items, worksite demographics are recorded including the type and stage of construction, type of dwelling, cycle time, and number in crew. The auditor notes whether work being performed at the time of the audit is appropriate for observed weather conditions, including mud, wind, snow, rain, ice, and heat. For example, if weather conditions are warm and very windy, and workers are setting trusses from ladders, the auditor would mark that the work was inappropriate for wind and appropriate for heat. A place for the auditors’ overall assessment and comments is provided. The complete SAFR instrument and administrator’s manual has been sent to the Electronic Library of Construction Safety and Health website (http://www.cdc.gov/elcosh/), developed by the Center for Construction Research and Training, for posting. ²³

Pilot Audit Results

All of the contractors that we contacted agreed to participate in the pilot testing. Sixteen new home construction worksites from four different contracting companies were audited in this pilot study. All of the homes except one were single-family dwellings. The average cycle time to complete the framing of the homes was 3 weeks (range 1-8 weeks), and the mean number of carpenters observed at each site was 4 (range 2 to 7). Most phases of the construction process were represented in the pilot, including framing of the first floor (7 sites), exterior siding installation (3 sites), and one each for framing second or third floor, foundation preparation, truss installation, window and door installation, roof sheathing, and drywall installation. Since only the work processes, worker behaviors, and construction phases occurring at the time of the audit were rated, the number of observations for individual audit items varied, ranging from 0% to 100% (see Table 1). Items measuring use of hard hats and safety glasses were rated at all 16 sites. Step ladders were visible to the auditors and rated at 75% of the sites; however they were set up for climbing at only 56% of the sites and workers were observed climbing ladders at only 31% of the sites. Items in the walking surfaces and edges domain were observed between zero and 56% of the time. We were able to observe truss setting, roof sheathing, floor joists and subfloor installation, scaffold use, and personal fall arrest domains at very few worksites. We were unable to observe the following audit items at any of the 16 pilot sites: truss installation on walls up to 8’ above lower level, access ladders to ladder jack scaffolds with walk board set up outside of the ladders, and use of brake on pump jack scaffold.

Table 1 also demonstrates the frequency that the observations met the SAFR performance criteria. These also ranged from 0% to 100%. When roof sheathing and operations requiring use of scaffolds and personal fall arrest were observed, the worksites consistently met audit safety criteria. Performance on other domains of the audit varied among pilot sites. Items that met the safety criteria most consistently include roof slide guards (100%), ladder condition (100%), the angle which extension ladders were set (100%), use of safety glasses (81%), guardrails present at unprotected openings (86%), and avoidance of climbing and working on the top three rungs of ladders (88%). Items that met the audit criteria less than 50% of the time include use of three-point contact while climbing ladders (40%), correctly designating and monitoring control access zones (25%), securing extension ladders at both the top and bottom (17%), setting first two trusses from ladder or scaffold (0%), and workers remove chain/webbing from truss while standing on ladder or secure truss (0%).

Journeymen auditor ratings were compared for all items of the SAFR scored by both journeymen auditors. Of these 892 items, there were 60 missing data points (6.7%), which were excluded from the level of agreement analysis. Of the 832 variable rated at all 16 pilot worksites visited, the auditors rated 788 of the items the same (94.7%), with a kappa statistic for agreement on all items on the audit of 0.87 (95% CI=0.83-0.91). Since the audit included items relevant to different phases of construction, and the auditors could only score the work processes they actually observed, there were many items on each audit rated ‘not observed’ by the auditors. Therefore, we also measured the rate of agreement between items that were actually observed and rated by both auditors. Of these 193 ratings, the journeymen auditors agreed on 188 of the ratings (97.4%). The level of agreement for these items measured with the kappa statistic was 0.93 (95% CI=0.88-0.99). Levels of agreement on the two versions of the audit used during pilot testing were very similar to these reported, as were levels of agreement between each of the journeyman auditors and the audit developer. Thus, inter-observer agreement was excellent between the two auditors, and between the auditors and the audit developer.

The mean time to complete the audit was 29 minutes (range 10-50 minutes). This included several minutes for discussion with the foreman and crew to orient them to the purpose and procedures of the audit. The multi-family site took longer due to the number of workers and work processes that were occurring simultaneously.

5. Discussion

We have described the development and pilot-testing of a residential construction fall safety audit, the SAFR, that is safe and simple to use, and reliably measures the presence or absence of recognized fall risks on residential construction sites when administered by trained auditors. The observational nature of the instrument ensures safety of the auditor and the construction workers during audit administration. Procedures for audit administration are outlined in a detailed administration manual, and our two trained journeymen carpenter auditors demonstrated excellent inter-rater reliability at the 16 pilot worksites using the standardized protocol.

Many steps in the development process helped to establish face, content, and construct validity ²⁴ of the SAFR. Review of surveillance data helped us to understand the types of surfaces and working conditions frequently associated with construction worker falls from heights. Measuring these specific conditions in our audit helped to establish content validity. The review of audit instruments used by other researchers helped us to choose audit methods that were most appropriate for measurement at residential construction work sites. The use of governmental safety and health standards to guide the safety criteria for most of the variables measured ensured that audit criteria were comparable to the only “gold standard” for construction safety in the U.S.A. The focus groups of apprentice carpenters helped us to understand the types of work tasks novice carpenters were performing on the job and how prepared they felt to perform these duties. Many of the tasks that these apprentices perceived as risky were included on the SAFR. By using seasoned carpenter trainers with safety expertise as the subject matter experts to guide audit development, we insured that our audit addressed key areas and that our safety criteria were appropriate. The resulting 52-item audit focuses on working conditions and equipment that are recognized fall risk factors for workers at residential construction sites; including general safety climate and housekeeping, floor joist and sub-floor installation, walking surfaces and edges, wall openings, truss setting, roof sheathing, ladders, scaffolds, and personal fall arrest equipment. Our audit instrument measures roof sheathing, but not roof shingling, as most carpenters in our area do not perform roof shingling. Since our audit focuses on the framing process, we did not incorporate items related to fall risk during the interior finishing of a home. Future investigators may want to expand the instrument to include these other construction phases of the home building process.

The observational nature of the instrument is both a strength and weakness. It allows the workers to be observed in their natural work environment performing their normal work duties; however, in order to observe all audit items the auditor would need to be present during most of the build cycle. We used the SAFR to measures workers’ fall risks during the stage of construction occurring at the time of the audit. Since workers’ behaviors and work practices during one phase of construction may not be predictive of those during other phases, assumptions about the overall fall risks at the construction site should not be made without further evaluation.

In order to be used as a population-level measure to assess fall safety risks for a contractor or geographical location, a larger sample of worksites at various stages of the construction process must be audited. Future administration of the SAFR at 200 residential construction sites in the St. Louis area will provide a better description of the fall prevention environment, work practices, and worker behaviors common in residential construction in our region. This will allow us to more accurately define exposures to fall risks as well as guide interventions to improve carpenter fall safety.

We assessed the reliability of the SAFR with a convenience sample of unionized carpenters working for large sized contracting firms in the St. Louis metropolitan area. Although this may limit the generalizability of the findings related to fall risk that we observed, it should not limit the appropriateness of items or the reliability of the tool on residential sites in other locations.

The development of the audit instrument for administration by carpenters is a clear strength. We found the SAFR to be practical in terms of time required to administer the assessment and ease of administration. Our carpenter auditors required relatively little training and found the detailed manual that accompanies the audit easy to follow, providing answers to most of their questions.

While we designed the SAFR to be used as an integral part of an ongoing research project, it could have useful applications in the field as an intervention tool as well. For example, the audit could be used by foremen, supervisors, or construction work teams on residential sites for assessment of fall risks and planning for hazard management. If used on a regular basis for this purpose, the amount of time to administer the audit would decrease with increasing auditor experience and familiarity of personnel at the worksite to the audit process. In order to decrease worker injuries and fatalities, we must identify and employ various methods of measurement and evaluation to control hazards and unsafe work practices to protect construction workers from falls. The SAFR has utility for fall safety assessment and intervention at residential construction sites.

Clinical Significance:

Understanding the fall risks inherent at residential construction sites is the first step to designing relevant fall protection interventions to decrease worker morbidity and mortality. A standardized observational audit will allow researchers to understand the impact of fall prevention programs on the workers’ behaviors in the actual work environment.