An Impact Evaluation of a Federal Mine Safety Training Regulation on Injury Rates Among US Stone, Sand, and Gravel Mine Workers: An Interrupted Time-Series Analysis

Abstract

Objectives. We evaluated the impact of a safety training regulation, implemented by the US Department of Labor's Mine Safety and Health Administration (MSHA) in 1999, on injury rates at stone, sand, and gravel mining operations.

Methods. We applied a time-series design and analyses with quarterly counts of nonfatal injuries and employment hours from 7998 surface aggregate mines from 1995 through 2006. Covariates included standard industrial classification codes, ownership, and injury severity.

Results. Overall crude rates of injuries declined over the 12-year period. Reductions in incident rates for medical treatment only, restricted duty, and lost-time injuries were consistent with temporal trends and provided no evidence of an intervention effect attributable to the MSHA regulation. Rates of permanently disabling injuries (PDIs) declined markedly. Regression analyses documented a statistically significant reduction in the risk rate in the postintervention time period (risk rate = 0.591; 95% confidence interval = 0.529, 0.661).

Conclusions. Although a causal relationship between the regulatory intervention and the decline in the rate of PDIs is plausible, inconsistency in the results with the other injury-severity categories preclude attributing the observed outcome to the MSHA regulation. Further analyses of these data are needed.

When the public thinks of dangerous occupations, mining is typically first on the list because of high-profile news reports of entrapments involving underground coal miners. Workers employed in other types of mining operations, however, face similar and sometimes higher risks of injury and disability than do their counterparts who work in coal mining. At surface dimension stone quarries, for example, workers use diamond-tipped cutting tools, high-velocity flame, and other specialized techniques to size and shape huge slabs of granite and other durable stone for use as countertops, street curbs, and building and bridge facing. The occupational injury incidence rate for workers employed at dimension stone operations was 5.0 per 100 full-time workers in 2006, more than twice the rate for surface bituminous coal miners.¹

Every year, thousands of US miners across the mining industry receive medical treatment of work-related injuries, more than one third of which require missing at least 1 day from work.^1–6 In addition to injury rates, fatalities in the mining industry are also higher than in most other industries, with a rate of 25.6 fatalities per 100 000 workers, compared with rates of 17.6 and 11.0 per 100 000 in the transportation and warehousing industry and the construction industry, respectively.⁷

To address hazards and prevent injuries and illnesses among US mine workers, the Mine Safety and Health Administration (MSHA) was established by the Federal Mine Safety and Health Act of 1977, which amended the Federal Coal Mine Health and Safety Act of 1969.⁸ In 1979, the agency issued mandatory safety and health training for any worker employed at a mine,⁹ including provisions requiring all new miners to receive at least 24 hours of safety training and all experienced miners to receive at least 8 hours of refresher training each year. The rule covered workers employed at underground and surface coal mines, underground metallic mines, surface pits and stone quarries, and any operations where workers were employed in extracting nonliquid minerals.

Representatives of the surface aggregate mining industry immediately appealed to members of Congress for an exemption from the MSHA regulation, asserting it was irrelevant and impractical for their non–coal-mining worksites. Their lobbying succeeded. In fiscal year 1980 and for 19 consecutive years, the following was included in the MSHA's annual appropriation:

MSHA is specifically prohibited from using appropriated funds to carry out Section 115 of the Federal Mine Safety and Health Act of 1977 or to carry out that portion of Section 104(g)(1) of such Act relating to the enforcement of any training requirements with respect to shell dredging, or with respect to any sand, gravel, surface stone, surface clay, colloidal phosphate, or surface limestone mine. [Pub L No. 105-78 (1998)]

The estimated 10 000 surface mines affected by this congressional rider came to be known as the “exempt” mines. The MSHA's existing training regulations did not apply to these operations, and federal mine inspectors were prohibited from inquiring about the health and safety training provided to workers employed at them. From 1993 to 1997, 200 miners died from fatal injuries while employed at noncoal surface mining operations, and 82% occurred at “exempt” operations.¹⁰

This persistent spike in fatalities and injuries at US sand and gravel mines, and a front-page story in USA Today about deaths at these operations,¹¹ created an opportunity for the MSHA's assistant secretary, congressional appropriators, and representatives of the “exempt” mines to negotiate a plan to remove the long-standing rider. Congressional leaders endorsed a plan to retire the rider if the MSHA worked closely with the affected mine operators to develop a new training regulation more appropriate for this sector of the mining industry. By congressional directive, the new training rules had to be issued by September 30, 1999.¹² Mine operators were given 1 year to be in compliance. These new mandatory safety and health training regulations, known as Part 46 (i.e., Part 46 of title 30 of the Code of Federal Regulations), took effect on October 1, 2000.¹³ The components of the training program are described in the box on this page.

Required Components of the Mine Safety and Health Administration's Part 46 Training Plan

• (a) You must develop and implement a written plan … that contains effective programs for training new miners and newly hired experienced miners, training miners for new tasks, annual refresher training, and site-specific hazard awareness training.

• (b) A training plan is considered approved by us if it contains, at a minimum, the following information:

  • (1) The name of the production-operator or independent contractor, mine name(s), and MSHA mine identification number(s) or independent contractor number(s).
  • (2) The name and position of the person designated by you who is responsible for the health and safety training at the mine. This person may be the production-operator or independent contractor.
  • (3) A general description of the teaching methods and the course materials that are to be used in the training program, including the subject areas to be covered and the approximate time or range of time to be spent on each subject area.
  • (4) A list of the persons and/or organizations who will provide the training, and the subject areas in which each person and/or organization is competent to instruct; and
  • (5) The evaluation procedures used to determine the effectiveness of the training.

Although the increased risk and dangers of surface mining are well documented, the effectiveness of occupational health and safety regulations to prevent work-related injuries, illnesses, and fatalities among US workers continues to be debated. Injury rates have gradually declined over the decades since the MSHA was established, but there are very few well-designed evaluations measuring the effectiveness of interventions of safety training and policy interventions.^14–16 As noted by the National Institute for Occupational Safety and Health in its National Occupational Research Agenda (NORA), many interventions, including hazard-specific safety training programs, are “undertaken based on faith and expert judgment without convincing evidence that these approaches are effective.”¹⁷ The primary aim of this evaluation was to assess the impact of the mandatory worker safety policy and training regulations issued in September 1999 by the US Department of Labor's MSHA on injury rates at US surface aggregate mining operations.

METHODS

MSHA regulations require surface aggregate and mineral mine operators to file a quarterly report, including employment hours.¹⁸ If an accident, injury, or illness occurs at the mine, the operator is required to report the event to the MSHA within 10 working days.¹⁸ The data used in this study are maintained by the MSHA's Office of Injury and Employment Information and are contained in 3 unique databases. Data for each of the 12 years of interest in this study, 1995 through 2006, were imported into SAS version 8.2 (SAS Institute, Cary, NC). The variables used in the analysis included mine identification number, the Bureau of Labor Statistic's standard industrial classification (SIC) code, employee hours, incident identification number, month and year of accident, and injury-severity category (e.g., lost time.) The preintervention evaluation and analysis period was defined as January 1, 1995, to September 30, 2000, the day on which the new regulation took effect. The postintervention evaluation and analysis period was October 1, 2000, to December 31, 2006.

All incidence reports involving a permanently disabling injury, a lost-time injury, a restricted-duty injury, or a medical-treatment-only injury suffered by a mine employee were used in the analysis. Injuries affecting contract workers were excluded from the analysis, as were reports of occupational illnesses. Injury rates were calculated as follows: rate = (∑ incidents / ∑ employment hours) × 200 000 hours. Quarterly injury rates were calculated for each severity category, and a serious-injury rate was calculated for each quarter by combining the count of permanently disabling, lost-time, and restricted-duty injuries.

Population at Risk

The MSHA's Part 46 regulations applied to about 10 000 mines (greater than 73% of all US mining operations) and affected an estimated 110 000 US workers. The mine sites used in this evaluation met the following inclusion criteria: they were (1) covered by the 19-year-old congressional appropriations rider, (2) operated between January 1, 1995, and December 31, 2006, and (3) had employees working at the mine for at least 8 quarters in both the preintervention and postintervention periods.

A total of 7998 mine sites met the inclusion criteria, with more than 85% reporting employee hours in at least 16 quarters in both the pre- and postintervention periods, and 42.5% reporting employee hours in all 48 quarters. Fifty-nine percent of the eligible mines were sand and gravel quarries, and about 32% were crushed and broken stone operations. Nearly 55% of the eligible mines were intermittent operations, defined as mines with at least 1 quarter per year with no reported employee hours.

Evaluation Design and Analysis

An interrupted time-series design and analyses, based on the models developed by Box and Jenkins¹⁹ for observations made sequentially through time, was used to evaluate the impact of the MSHA policy. Time-series analysis is a form of forecasting in which a collection of observations from the past are modeled to make predictions about the future. With time-series data, the observations are close together in time and analyses must take into account the correlation between proximal observations.²⁰ There is an inherent assumption in forecasting that the behavior of the observations will continue in the future unless external factors, such as the mandated worker training program, influence the events.²¹

Injury rates were analyzed by time-series techniques to determine their autocorrelation structure. Model identification techniques were used to distinguish which time-series model adequately characterized the autocorrelations among the observations. The goal was to identify a model that adequately explained the behavior of the observations and their interdependence. Time-series models were classified as autoregressive, moving average, or autoregressive integrated moving average (ARIMA). When the plots of quarterly injury rates and the time-series models suggested an intervention effect, we used regression models to assess the possible effect of antecedent trends in the preintervention period.

RESULTS

Quarterly serious injury rates were calculated for the 7998 mines for 1995 through 2006. The data were stratified by year-round vs intermittent mining operations and by ownership (i.e., whether or not the mine was controlled by 1 of the 10 largest US aggregate producers, which were responsible for about 50% of annual US aggregate production from surface mines^22,23). Neither the data series plots nor the model identification techniques provided evidence of change in serious injury rates following the MSHA's Part 46 rule. (Descriptive data and plots of injury rates are available as a supplement to the online version of this article at http://www.ajph.org.) Figure 1 presents the data stratified by SIC codes. The period in which the MSHA Part 46 training regulation was issued is indicated by the first vertical line. The effective date of the rule, September 30, 2000, when it was enforceable by federal mine inspectors, is indicated by the second vertical line. During the 12-year period, the overall rate of serious injuries declined by 52.6% at crushed stone operations, 46.2% at sand and gravel operations, and 38% at other surface mineral mines. The serious injury rate at dimension stone operations (not shown in Figure 1), representing 4.2% of the study population, was consistently 1.5 to 2.0 higher than the rates among the other 3 aggregate types.

FIGURE 1.

Quarterly rate of serious mining injuries by aggregate type, 1995–2006.

The data series for serious injury rates stratified by the 3 aggregate types fit the ARIMA models, with the variation left unexplained by the model best characterized as white noise. Like the data series stratified by production schedule and ownership, there was no evidence of an intervention effect on serious injury rates following the MSHA's Part 46 rule.

Injury Severity

Nearly 96 000 injuries of varying severity were reported to the MSHA over the 12-year period among the study population of mines. Lost-time, medical-treatment-only, and restricted-duty injuries accounted for 38.7%, 36.4%, and 23.7%, respectively, of all reported injuries. Figure 2 presents the quarterly incidence rates by the injury-severity categories. Lost-time and medical-treatment-only injuries declined by 56.8% and 60.3%, respectively, over the 12-year period.

FIGURE 2.

Quarterly rate of mining injuries by severity, 1995–2006.

The data series fit an ARIMA model, with the variation left unexplained best characterized as white noise. We noted the convergence in the rate trend for lost-time and restricted-duty injuries, particularly from 1995 to 2003, an occupational injury reporting trend identified by others.^24–28 Time-series data of quarterly lost-time, restricted-duty, and medical-treatment-only injury rates provided no evidence that the observed decline could be attributed to implementation of the MSHA's Part 46 training regulation.

Permanently Disabling Injuries

Among the study population of mines, 1131 permanently disabling injuries were reported to the MSHA. These injuries, which can be partially or totally disabling, are those that incapacitate the individual from “following any gainful occupation,” such as losing one or both eyes or limbs.²⁹ Figure 3 is a plot of the quarterly rates of permanently disabling injuries, which declined by 53.8% from the pre- to postintervention periods. The plot reveals a pronounced shift in the injury rate trend following the MSHA's Part 46 regulation.

FIGURE 3.

Quarterly rate of permanently disabling mining injuries, 1995–2006.

A Poisson regression model revealed no significant interaction between the intervention and time with respect to the injury rates. The analysis of main effects revealed a statistically significant difference in the time-series trend in the pre– and post–Part 46 periods (χ² = 10.17, P = .001). A risk ratio estimate was calculated to quantify the change in the population risk of experiencing a permanently disabling injury in the 2 periods (risk ratio = 0.591; 95% confidence interval = 0.529, 0.661). This risk ratio estimate suggests that there was 41% less risk of a permanently disabling injury after the MSHA's Part 46 rule was issued.

DISCUSSION

Our analyses offer inconsistent results on the impact of the MSHA's Part 46 regulation on injury rates. We identified a statistically significant decline in the rate of permanently disabling injuries, but rate reductions were not observed for injuries of lesser severity. The discordant findings suggest that (1) there are rival explanations, other than the MSHA's Part 46 rule, for the decline in permanently disabling injury rates and (2) further analysis of the data are needed to identify factors related to the reported permanently disabling injuries that may explain the observed effect.

We assessed specific biases that might provide rival explanations that threaten the internal validity of results, including the following: maturation (aging, development of new skills, etc.) or other changes to the study subjects; history or extraneous events; definitional or recordkeeping changes; and instrumentation changes in data collection tools.^20,30,31 We examined the ownership profile of our study population to determine whether consolidation occurred in the surface aggregate industry. We did not, however, identify unusual trends in local or regional mergers or acquisitions, nor did we find independent evidence of substantial corporate consolidation in the industry.^32,33 We also reviewed changes in the proportion of union to nonunion mines during 1995 through 2006. The majority of workers employed in the surface aggregate mining sector have not typically been members of labor organizations.^34,35 For the entire mining industry, an estimated 8% of workers are union members, and only 1% of these are employed outside of the coal and metal extraction sectors.³⁶ We have no evidence to suggest that unionization in the industry changed in any notable way during the 12-year study period.

We explored whether the tools used to collect the MSHA's injury and employment data were modified during 1995 through 2006 and could attribute for the observed effect. For, example, were changes made to the definition of reportable injuries or severity categories? We examined the MSHA's regulations for employer reporting of injuries, emergency events, and employee work hours, as well as the content and design of the forms used by employers to report this information. The regulations underlying these reporting requirements and the forms have not changed since 1980.^29,37,38 We recognize that underreporting of injuries by employers has been corroborated in the literature,^39–44 including underreporting by mine operators.^45–47 In our analysis, we assumed some underreporting of injuries by employers, and that the degree would be comparable in the pre– and post–Part 46 periods. We also determined that the MSHA did not issue any regulations during the intervention period applicable to the surface aggregate industry, besides the Part 46 training rule, that might have addressed specific injury-causing hazards.

We identified historical events, which occurred about the same time as the observed drop in the rate of permanently disabling injuries, to determine whether they provided logical rival explanations. One was the election of George W. Bush as president, but we considered it implausible that this influenced the results because the most profound shift in the permanently disabling injury rate occurred in the third quarter of 2000 and continued to decline for 2 additional quarters. These periods preceded President Bush's administration. The second was the Transportation Equity Act for the 21st Century of 1998 (TEA-21), which authorized $162 billion for road building and other transportation projects.⁴⁸ TEA-21’s proponents included aggregate producers; more than 80% of crushed stone produced in the United States is used for highway and road construction.⁴⁹ Mine safety experts hypothesized that increased demand for aggregates because of TEA-21 would increase the risk of injury for mine workers. Thus, instituting new safety training regulations to protect aggregate industry workers from injury was imperative.^50,51 We observed an increase in injury rates in our study population in the second half of 1998, when the Federal Highway Administration had already made 2 apportionments to the states.⁵² Our analysis revealed, however, that these trends were consistent with historical trends in injury rates. Moreover, a trade association representing 90% of US aggregate producers asserts that most states did not direct TEA-21 funds to projects that would have increased demand for their product.⁵³

We were unable to identify a specific threat to internal validity that might provide a rival explanation for the statistically significant decline in the rate of permanently disabling injuries, but not in injury rates for the other severity categories. We plan to scrutinize the data further to examine specific factors related to each of the 1131 permanently disabling injuries. The data set includes a variable providing 44 different “accident classifications” and another with 39 “nature of injury codes,” such as crushing injuries, burns, or amputations. One hypothesis to explore is whether certain types of injury-contributing hazards (e.g., energized equipment leading to amputations) were more prevalent in the pre–Part 46 period, and further, whether particular topics related to specifically grave hazards were addressed in mine operators’ training plans. One might also hypothesize that the demographic characteristics of the workers who suffered permanently disabling injuries differed in the pre– and post–Part 46 periods. Further analysis of the data could focus on injured workers’ dates of birth and their years of mining experience or years in a current job, which are factors available in the MSHA's injury incidents database.

Study Limitations

A significant limitation in this study's quasi-experimental design is the assumption that the regulatory requirement for a written safety training program for mine workers translated into actual implementation of the program. Our study did not attempt to document, for example, the extent to which the program components listed in mine operators’ written training plans were implemented with fidelity (i.e., identifying a type III error).^31,54,55 The MSHA is required to inspect surface mines at least twice a year, and scrutiny of training records is part of the inspection. Data obtained from the agency indicate that mine operators largely complied with the Part 46 requirements, with compliance determined by having a training plan and showing employee attendance records. Of the 7998 mines in the study population, 64.7% did not receive a single citation for the period October 1, 2000, through December 31, 2006, for failing to comply with any provisions of the rule. Fewer than 10% of the mines received more than 1 violation of Part 46. (Descriptive data are available as a supplement to the online version of this article at http://www.ajph.org.) Furthermore, we did not assess whether the model training plan prepared by the MSHA to ease employers’ compliance with Part 46⁵⁶ was capable of yielding injury prevention. Critical features of an effective program include assessing training needs and specifying well-designed measurable training objectives.^57–61 Neither components are required in Part 46. The historical and political context in which the regulation was developed provides insight into why the rule may not have measurably affected injury rates.

Beginning in 1998, members of Congress, the aggregate industry, and the Clinton administration engaged in negotiations for a plan to remove the appropriations rider that made surface aggregate mines “exempt” from MSHA enforcement of training regulations. The MSHA was directed specifically to develop a proposed training rule based on draft regulatory text prepared by a coalition of aggregate industry officials and labor representatives, the Coalition for Effective Miner Training (CEMT).^13,62 The CEMT, which was chaired by the senior vice president of the National Stone Association, comprised 19 organizations, 16 of which represented aggregate producers and employers. Their draft was inspired by the structure, format, and content of the MSHA's existing Part 48 training regulations (which was adopted in 1979 for all underground mining operations), and an overarching goal was regulatory flexibility and compliance simplicity. Mine operators wanted a regulation with minimal “paperwork,” one that would allow them to conduct training informally, did not interfere with production demands, and explicitly stated what was required to be “in compliance.”^13,63,64

The demand for flexibility and compliance simplicity, coupled with the congressional directive to use the CEMT's draft regulation as a starting point,⁶³ significantly influenced the structure and content of the employers’ training programs. The key MSHA staff assigned to negotiate with the CEMT and write the rule were primarily attorneys, economists, and regulatory specialists, not experts in training, adult learning, or program evaluations. Its content was predisposed to be short on learning theory and was not evidence based; it was grounded instead in the mandate for consensus. It is plausible that the MSHA's Part 46 regulation failed to result in a marked reduction in injury rates because the political and administrative goals of compliance simplicity and flexibility eclipsed goals related to learning objectives and measurable outcomes for injury prevention.

Another conceivable explanation for the lack of a Part 46 intervention effect is that surface aggregate employers may already have been providing safety training to their employees, despite the congressionally provided exemption from the MSHA's training regulations. It is plausible, therefore, that the nonprescriptive mandates of Part 46 did not compel employers to substantially modify their existing training practices. Although safety and health training is a necessary component of an effective occupational injury and illness prevention program, it is just one element. Management commitment, worker involvement, worksite analysis, and hazard recognition and control are equally important. An effective occupational injury and illness prevention program requires active participation by workers in its planning, implementation, and evaluation.^65–68 A body of empirical research suggests that safety and health training may be effective only if it is consistent with organizational factors, such as management's policies and practices.^69–73 Thus, even if a Part 46 training plan was conducted with a high degree of fidelity because of the MSHA rule, it may not have had a direct effect on injury rates if concomitant factors related to workplace organization, including a positive “safety climate,” were not present.^74–77 As one researcher noted, “Any disillusionment over our ability to ‘train away’ workplace injuries and illnesses may be more an indication of our unrealistic expectations than an indictment of the training process.”⁷⁸

Topics for Future Research

A logical step for further research would be examining the quality, time, frequency, and intensity of the training sessions instituted by mine operators in response to the Part 46 requirements. A process evaluation documenting these characteristics and confirming the degree of fidelity of program implementation would provide empirical evidence to explore hypotheses about why and how interventions worked or failed to work.^31,79–81 Assessing the actual implementation of a written program would serve several purposes, including (1) understanding obstacles in the “real world” to program execution, (2) forcing accountability for program implementation, (3) enhancing the validity of program evaluation, (4) learning which program components could be modified to improve their effectiveness, and (5) documenting the cost, cost effectiveness, and benefit of the program.^57,69

On the 10th anniversary of the NORA, officials noted that intervention effectiveness research was “an underutilized tool when NORA began,” and indicated that over the subsequent decade, few contributions were made to the science base.¹⁵ This study's findings on the impact of the MSHA's Part 46 regulation illuminate the need for policymakers to plan for effectiveness evaluations during the early stages of developing occupational-hazard policies. Because US occupational health and safety agencies do not receive funding commensurate with their statutory responsibilities,^82,83 it is particularly important for them to validate the effectiveness of injury-prevention intervention efforts. Data demonstrating the social benefits of occupational health and safety policies can help build support for agencies’ prevention efforts.