Use of the bootstrap method to develop a physical fitness test for public safety officers who serve as both police officers and firefighters

Abstract

Physical fitness testing is a common tool for motivating employees with strenuous occupations to reach and maintain a minimum level of fitness. Nevertheless, the use of such tests can be hampered by several factors, including required compliance with US antidiscrimination laws. The Highland Park (Texas) Department of Public Safety implemented testing in 1991, but no single test adequately evaluated its sworn employees, who are cross-trained and serve as police officers and firefighters. In 2010, the department's fitness experts worked with exercise physiologists from Baylor Heart and Vascular Hospital to develop and evaluate a single test that would be equitable regardless of race/ethnicity, disability, sex, or age >50 years. The new test comprised a series of exercises to assess overall fitness, followed by two sequences of job-specific tasks related to firefighting and police work, respectively. The study group of 50 public safety officers took the test; raw data (e.g., the number of repetitions performed or the time required to complete a task) were collected during three quarterly testing sessions. The statistical bootstrap method was then used to determine the levels of performance that would correlate with 0, 1, 2, or 3 points for each task. A sensitivity analysis was done to determine the overall minimum passing score of 17 points. The new physical fitness test and scoring system have been incorporated into the department's policies and procedures as part of the town's overall employee fitness program.

The physical demands of police work and firefighting are well documented (1, 2), as are the low fitness levels (3, 4) and high rates of morbidity and mortality from cardiovascular disease that are often associated with these occupations (1, 5). In theory, physical fitness testing is a straightforward and logical way of motivating public safety personnel to reach and maintain a minimum level of physical conditioning so they can perform arduous job tasks. In practice, however, choosing and implementing a physical fitness test at an individual work site can be highly problematic: Such tests must comply with US laws that prohibit discrimination on the basis of race, color, religion, national origin, sex, age, or disability (6), characteristics that define protected classes. The Highland Park Department of Public Safety (HPDPS), in the town of Highland Park, Texas, is unusual because it employs public safety officers who are cross-trained as firefighters and police officers. For over a decade, the department used two kinds of fitness tests simultaneously (a general test consisting of exercises such as push-ups, sit-ups, and running and a task-based test consisting of firefighting activities), but providing both was costly, time consuming, and cumbersome. In 2010, exercise physiologists from Baylor Heart and Vascular Hospital who are familiar with the functional capacity requirements for firefighters and police officers (7, 8) began working with fitness experts from HPDPS in a study to develop and evaluate a single, equitable test that would meet the department's unique requirements. This article describes that collaboration and the results it yielded.

METHODS

Baylor Research Institute's institutional review board approved the study, and all 50 subjects (all of whom were sworn employees of HPDPS) gave informed consent and provided demographic information. Data were collected on the 45 men and 5 women (aged 23 to 61 years; mean, 41 years) during a series of three quarterly testing sessions. The timed obstacle course comprised 10 activities and was divided into three sections. The first section, general fitness, evaluated agility with the Illinois test, flexibility with the sit and reach test, and muscular endurance with push-ups and sit-ups. The remaining sections evaluated muscular strength and cardiovascular fitness through a series of job-specific tasks related to firefighting (stair climbing, ceiling breach, and forcible entry) and police work (chasing, fighting, and handcuffing a perpetrator) (Figure 1). While the test was being developed, the subjects did two trial runs, enabling the study staff to fine-tune various aspects of administering the test before collecting data. The assumption was made that the employees' job performance was above average at the time of testing; therefore, only their data would be used to derive the grading criteria and minimum passing score.

Figure 1.

Examples of the job-specific tasks included in the physical fitness test: (a) stair climbing, (b) forcible entry, (c) punching, and (d) dummy roll ending with a simulated handcuff.

Testing took place at HPDPS headquarters and was monitored by the study staff and by HPDPS peer fitness coordinators. Paramedics and a mobile intensive care unit were available in case of an emergency. The subjects wore athletic apparel (shorts, t-shirt, and athletic shoes) and were given detailed instructions before and during the test. They were asked to complete the following tasks, in order, as efficiently as possible:

• Illinois agility test. From a prone position on the floor with outstretched hands placed behind the starting line, the subjects rose and sprinted (straight and serpentine) through a series of cones as fast as possible. (Recorded value: total time.)

• Push-ups. For 60 seconds, the subjects performed as many consecutive full-body push-ups as possible, touching their chest to a commercial push-up counting device when in the down position to ensure consistency. (Recorded value: number of repetitions.)

• Sit and reach test. The subjects, without shoes, sat on the floor with their legs extended and the soles of their feet against the edge of the testing box, which had a scale and a metal slider on top. With their arms evenly stretched, hands parallel, and palms down, they slowly reached forward as far as possible, pushing the slider along the scale. (Recorded value: longest reach during three attempts.)

• Sit-ups. With bent knees, fingers cupping the ears, and ankles held firmly by a partner for support, the subjects performed as many sit-ups as possible in 60 seconds. (Recorded value: number of repetitions.) The subjects were given a 2-minute rest before continuing to the firefighter tasks.

• Stair climbing. While wearing a 50-pound weighted vest with an additional 12.5-pound weight on each shoulder, the subjects performed a 20-second warm-up on a stepping treadmill at a rate of 50 steps per minute. They were then timed while stepping at a rate of 60 steps per minute for up to 3 minutes. (Recorded value: actual time at 60 steps per minute.)

• Ceiling breach. The subjects removed a 5-pound pike pole from a bracket and placed the tip on the 60- to 80-pound push/pull pike pole simulator. They attempted to complete 3 sets, each set consisting of 5 pulls and 3 pushes. (Recorded value: number of completed sets.)

• Forcible entry. While standing on steel planks alongside a 160-pound beam, the subjects slammed a 9-pound sledgehammer into the end of the beam until either the beam moved 5 feet or the 60-second time limit ended. (Recorded value: movement of the beam in thirds.) The subjects were given a 2-minute rest before continuing to the police tasks.

• Perpetrator takedown. The subjects a) sprinted 300 yards; b) participated in a mock fight by delivering a series of 24 punches to a target dummy; and then c) dropped to their knees on a mat, rolled a 150-pound dummy four times one way and four times back, and simulated a behind-the-back handcuff by bringing the dummy's hands together behind its back. (Recorded value: total time.)

The raw values collected during the three quarterly testing sessions were analyzed to determine the appropriate ranges for a scoring system that would assign 0, 1, 2, or 3 points for each task, according to level of performance. The bootstrap method was used; it was introduced in 1979 as a novel statistical technique that falls under the broad heading of resampling (9). When a statistic such as the sample mean is computed on a data set, that single statistic is known but its variability is not. Through resampling, the bootstrap method creates a larger number of data sets and computes the statistic on each of them. Thus, a distribution of the statistic is obtained, permitting an estimate of the statistic afterwards. The bootstrap technique involves many repetitions of a relatively simple procedure and is heavily dependent upon computer calculations.

In the present study, bootstrapping was used to first obtain a mean distribution of each of the test categories. The derived bootstrap distribution was then used to find the values at 2.5% (1 point) and 97.5% (3 points). The values between those for 1 point and 3 points defined the range for 2 points. The values for 1 point defined those for 0 points (e.g., if 10 to 31 sit-ups = = 1 point, then <10 sit-ups = 0 points). Figure 2 includes the histogram of the raw Illinois agility test values and the histogram of the mean distribution of such values via bootstrapping. Although the recorded values are right skewed, the derived mean distribution is quite normal after resampling 10,000 times.

Figure 2.

Histograms of (a) the raw values from the Illinois agility test and (b) the mean distribution of the raw values through bootstrapping.

After all the subscores for each item were added, a sensitivity analysis was done to determine the appropriate minimum passing score. Ranges of scores were chosen to ensure that 80% or more of the protected class (defined for this study as racial or ethnic minority, disabled, female, or age >50 years) would be able to pass the test (10). Overall scores for the physical ability test were compiled by summing all points achieved during each test component, and the minimum passing score was 17 points. All analyses were done with R 2.14 software (R Project for Statistical Computing, http://www.r-project.org).

Finally, HPDPS leadership worked with the town council, town administrator, human resources staff, and legal counsel to develop a long-range plan for implementing the new testing procedure.

RESULTS

The new physical fitness test and scoring system, summarized in the Table, have been incorporated into the policies and procedures of HPDPS as part of the town's overall employee fitness program. All sworn employees are required to exercise regularly while on duty (as emergency calls and other responsibilities permit) and to undergo quarterly physical fitness testing. HPDPS offers a score-based financial incentive for those who pass the test on the first attempt; however, the requirements for passing are being phased in over a 4-year period (June 2013, pass one quarter per year; June 2016, pass every quarter).

Table.

Components and scoring criteria of the new physical fitness test developed for the Highland Park (Texas) Department of Public Safety

	Scoring criteria
Test component	0 Points	1 Point	2 Points	3 Points
Illinois agility (seconds)	>40.00	20.54–40.00	19.51–20.53	<19.51
Push-ups (total)	<10	10–40	41–48	>48
Sit and reach (inches)	<10.00	10.00–18.84	18.85–19.99	>19.99
Sit-ups (total)	<10	10–31	32–36	>36
Stair climbing (minutes)	<1:00	1:00–2:19	2:20–3:19	>3:19
Ceiling breach (thirds)	<1/3	1/3	2/3	3/3
Forcible entry (thirds)	<1/3	1/3	2/3	3/3
Total firefighter time (minutes)	>7:00	5:54–7:00	5:42–5:53	<5:42
Perpetrator takedown: Total police time (minutes)	>6:00	2:17–6:00	2:04–2:16	<2:04
Total test time (minutes)	>20:00	16:41–20:00	16:21–16:40	<16:21

The department helps employees prepare for testing. When necessary, it provides additional support such as customized training with a peer facilitator or referrals to outside programs (e.g., diet and nutrition counseling or occupational rehabilitation). In addition, HPDPS policies include provisions for retesting those who fail to pass and for testing those who miss a quarterly test due to illness or injury.

DISCUSSION

Reaching and maintaining an adequate level of fitness is a key component of maintaining job readiness, and physical fitness testing can play an important role. The goal of achieving fairness in physical fitness testing remains elusive (11), however, and employers' use of such tests has been hampered by many factors, including fear of legal action, complaints by employee groups, and lack of funds (10). Various tests have been used in the past and are still used, but they have been challenged repeatedly. For example, it can be difficult to show that the exercises long used for general fitness testing are essential to a specific job (11). Task-based tests, such as climbing a fence or dragging a dummy, simulate job activities but often discriminate against women, minorities, and people who are older or disabled.

Despite these problems, physical fitness testing is a necessary tool in the public safety workplace. Fire departments seeking to implement a task-based test can choose from protocols that are commercially available but often very expensive to license or administer. Police departments, which have fewer testing options to choose from, tend to use activities from the general fitness category. Because no existing test adequately addressed the combined police officer/firefighter job description at HPDPS, the leaders chose to develop a new test to meet the department's nontraditional needs.

To determine a scoring system that would be fair, the bootstrap method was used to compare raw data from all employees who took the test, including those who were in the protected class. The resulting scoring criteria, determined solely from this group of subjects, allow for differences in skill levels while encouraging improvement. In the past, a task such as dragging the dummy was graded as pass or fail. In the new test, dragging the dummy one third of the total distance earns 1 point, which could motivate the employee to perform better in the future instead of giving up on the task completely.

Because this study involved a specific cohort of sworn officers in a single public safety department, the results (particularly the scoring system) are not generalizable to other police or fire departments. However, the structure and organization of the HPDPS test, along with the bootstrap method described herein, could benefit other departments that seek to develop a customized physical fitness test.

For more than 3 decades, disputes over the fairness of fitness standards in the US workplace have led to hundreds of legal actions, costing taxpayers millions of dollars. Even the fear of being sued has had a paralyzing effect on some employers, causing them to abandon physical fitness tests and standards entirely (10). These concerns, however, have not deterred HPDPS from its current goal of motivating sworn employees to improve and maintain their level of fitness and overall wellness. The present study produced the new testing procedure and scoring criteria that are being used to support that goal.