Abstract
Measuring the ultimate impact of research on health and economic well-being has presented challenges that have rarely been surmounted, and research on preventing occupational injuries and illnesses is no exception. Nevertheless, there is an increasing need to demonstrate the value of publicly funded research. The National Institute for Occupational Safety and Health (NIOSH) recently contracted with the RAND Corporation to conduct six in-depth case studies that aimed to quantify the benefits of key NIOSH research efforts using economic metrics. These case studies focused on silica exposure in asphalt pavement milling, firefighter cancer risks, a multi-industry matching-grant program for purchase of safety equipment, personal coal dust monitors for coal miners, redesign of ambulance patient compartments for safety, and workplace amputation surveillance. In this article, we summarize what we learned about how measurement of research benefits may be pursued. We summarize the benefit measurement methods that were used and the results of these research efforts in terms of costs saved, injuries and illnesses prevented, and the statistical value of reductions in risk of death or illness. We then distill some observations about the characteristics of research efforts that make measurement of research benefits feasible and suggest steps that could make it feasible to apply the same methods more widely. We also outline key NIOSH activities that appear not to be amenable to benefit measurement but suggest potentials for progress toward at least partial or qualitative benefit assessment. Finally, we discuss implications of the benefit measurement case studies for strategic research planning.
Keywords: cost–benefit analysis, NIOSH, occupational safety and health research, prevention effectiveness, strategic research planning
1 |. INTRODUCTION
It is notoriously difficult to measure the benefits of research for several reasons. The knowledge that new research produces enters into and combines with a larger pool of existing knowledge. And if the research produces new, actionable information, that information is only part of what decision-makers consider as they respond to complex incentives and organizational and market environments. The new information from research may also be put into practice in varied ways by numerous, mostly private parties whose actions are often undocumented. Sometimes as well, it is uncertain what share of the products of research should be credited to research versus the learning from practical experience it built upon.
Nevertheless, there is an increasing need to demonstrate the value of publicly funded research that promises to improve the safety, health, and well-being of workers. Conducting this type of applied research is the mission of the National Institute for Occupational Safety and Health (NIOSH). To help address the need to demonstrate the value of its research, NIOSH recently contracted with RAND to conduct six, in-depth, case studies that aimed to quantify the benefits of key NIOSH research efforts using economic metrics. We summarize what we learned from these case studies and how this can help expand and improve future assessments of NIOSH research, as well as contribute to strategic research planning and translation of research into practice. Some of these lessons should apply to assessment and planning of prevention research by other organizations as well.
We begin with a brief overview of some notable prior efforts to measure the economic benefits of research, identifying those that are most applicable to NIOSH research. We then provide an overview of the subjects and results of the RAND case studies. Following this, we summarize and provide insights on the methods used to conduct the case studies, both on a general and on a more detailed level. This provides a guide to conducting future case studies. Because one of the steps in conducting the case studies is verifying and assessing the role of NIOSH research in reducing injury and illness outcomes, we next highlight the indispensable contributions to these reductions made by other stakeholders. Next, we address the central issue of feasibility of measuring research benefits by identifying the main features of the six case study subjects that made them feasible. We also discuss ways to address the barriers to feasibility for other, potential subjects of case studies. Finally, we outline additional ways to apply what we learned from the case studies to project and strategic planning.
2 |. PREVIOUS STUDIES MEASURING BENEFITS OF RESEARCH
NIOSH has previously undertaken many program evaluations that identify research impacts. A major advance in these efforts was the commissioning of a series of monograph reports from National Academies of Sciences expert review panels, in 2005–2008.1 Each report summarized a program’s achievements and evaluated the relevance and apparent impact of research in a qualitative way, yielding recommendations for improvement. Later, NIOSH conducted a series of program evaluations using a framework called contribution analysis.2 These evaluations documented how NIOSH research outputs were received, adopted, or translated into practice by intermediaries (e.g., professional and trade associations or equipment manufacturers) and end users (e.g., employers and workers). The logic of how this use of information should result in fewer illnesses and injuries was then articulated. However, the evaluators did not attempt to quantify impacts due to lack of sufficient data. NIOSH’s counterpart in Ontario Canada, the Institute for Work and Health, has similarly focused on documenting how its research has been used by other parties in a series of very brief case studies, but has not generally been able to document and quantify impacts on worker injury and illness.3,4
We must therefore look outside NIOSH and the field of occupational safety and health to find relevant, previous efforts to quantify the benefits of research with economic metrics. We begin with discussion of a landmark study that pointed to the paucity of such efforts. Following this, we describe several sets of studies that provide partially applicable models for measuring benefits of NIOSH research and highlight the distinguishing characteristics of the NIOSH-sponsored RAND studies.
In 2011, Congress directed the National Research Council (NRC) to produce a study to develop improved impact-on-society metrics for research, with special emphasis on federally supported research.5 A notable part of its statement of task was to “review and synthesize the broad variety of efforts to assess research output and impacts in industry and government,” and to “explore various methodological approaches to measurement of research quality, productivity, and impact, including those that help to quantify the return on investment of research.” Thus, the NRC cast its net broadly. In so doing, it cited most of the notable, previous efforts to quantify benefits of research with economic metrics that are most relevant to NIOSH’s recent efforts.
The NRC report observed that government-wide tools for measuring research performance, such as the Government Performance and Results Act (GPRA) and the Program Assessment Rating Tool focus on measuring short-term outputs rather than long-term outcomes. Furthermore, they cited the findings of another major report that surveyed 20 existing and proposed frameworks and tools for evaluating research, but concluded that these approaches do not address the fundamental question that NRC was concerned with: What would have happened without the research funding program? Without the answer, the true impact remains unknown.
The NRC report also discusses evaluations of programs at the National Science Foundation (NSF) and the National Institutes of Health (NIH). The NRC states that the descriptive statistics and case studies in the NSF program evaluations “rarely yield insights valuable for measuring impact.”
The NRC report did not summarize the character of the NIH program evaluations in a similar way, but descriptions of them are available on NIH evaluation websites.6–9 The evaluations encompass a rich range of methodologies and metrics adapted to specific programs, but appear to be aimed mainly at providing guidance for managing the programs, rather than producing quantitative estimates of ultimate program impact. A report on the website of the NIH Office for Evaluation, Performance and Reporting9 provides an ambitious set of guidelines for future development of capacity to “systematically, comprehensively, dynamically, and strategically” assess the value of biomedical research.10 However, the report also explains the great challenges to realizing this vision by detailing the difficulties involved in tracing the impacts of biomedical research ranging from basic science, through many stages of development, to widespread application, and normally involving contributions by a host of other parties. It is these challenges that appear to motivate the overall thrust of the NRC report itself, which steers away from a central focus on summary metrics to instead emphasize the importance of further developing understanding of the complexity of the scientific research and innovation system to improve its performance.
We should briefly note another government-wide initiative that will promote and require more evaluation of government programs that was undertaken since the appearance of the NRC report: the Foundations for Evidence-Based Policymaking Act of 2018 (“Evidence Act” Pub. L. No. 115–435, 132 Stat. 5529), signed into law on January 14, 2019. While agencies may pursue any type of evaluation that is appropriate to their activities, there is explicit focus on the plans and processes for gathering data to support evaluations, and on selection and development of evaluation methods, suggesting that there will be more emphasis on longer term outcomes.11
Some of the best-known quantitative studies of the impacts of research have been conducted on a broad societal or industrywide level. This contrasts with the NIOSH-sponsored case studies, which required evidence on the connections between specific research efforts and outcomes. However, these aggregate-level studies are worth taking note of, because they have been highly influential in making the general case for the possibility of measurement of research benefits with economic metrics, and because some of their features provide partial models for evaluating NIOSH research. A seminal paper by Murphy and Topel (2003) compared the value of the overall increase in longevity in the United States in 1970–1998, net of increases in health care costs, to the total cost of public and private medical R&D.12 Because the R&D costs were far smaller than the net value of increased longevity (less than 3%) and since the decline in mortality rates appears to be due in significant part to research-based innovations in health care, they concluded that benefits of medical R&D were likely much larger than the costs. Another paper in the same volume by Cutler and Kadiyala (2003) examined the causes of recent declines in cardiovascular mortality and divided them into three, similarly sized groups (behavioral change, pharmaceuticals, and invasive procedures).13 Based on attributing all three of these sources to research (and including research dissemination costs in research costs), they estimated that research on behavioral risks had about a 30 to 1 benefit-cost ratio and that research related to technology improvement had a 4 to 1 benefit-cost ratio.
Another body of literature examines the relationship between the dollar amount of R&D investment and total sales or stock market value in a variety of industries to estimate a financial rate of return to R&D (ROI).5,14,15 In contrast to increased length of life and reduced mortality, total sales and stock market value measure benefits in ways that relate to gains of private firms and only indirectly and imperfectly represent benefits to society. But similar to the society-level health research studies, this literature examines research impacts on an aggregate level, avoiding attempts to determine the connections between specific research efforts and outcomes. In studies of the relationship between R&D and economic outcomes at the firm level, it has been assumed that the firm undertakes R&D to benefit its own bottom line, enabling the statistical relationship between R&D investment and economic outcomes to be plausibly interpreted as causal. Since much of the knowledge produced by firm-level R&D eventually becomes useful to other competitors in an industry, studies have also examined the relationship between R&D and economic outcomes on the industry level. Hall and colleagues (2014) have observed that early studies in this line of research, mostly done in the early 1980s, estimated rates of return to R&D of 15% to 100% and have provided a prominent support for funding of private and public research well into the 21st century.15
ROI research has proved particularly useful for support of public agricultural research. Analyses have related agricultural sector productivity to the total dollars spent on private and public agricultural research. More recent analyses have focused on the specific roles of private versus public research, generally finding that they are complementary in promoting productivity.16 Some studies have taken account of factors facilitating research transfer to practice, including the role of agricultural extension services and education, as well as road infrastructure, to estimate the impact of public R&D spending on productivity growth in the sector. The literature has found rates of return of 20%–60% and benefit-cost ratios of between 10 and 20 to 1.16 As in the domain of health, it is important to note that these studies have been enabled by the evident fact that research supported by the public sector has accounted for a large, if uncertain, share of increases in agricultural productivity. While large agricultural input firms and others have made important contributions, individual farms have few incentives to drive productivity growth in the industry with their own research.
Studies that relate dollar amounts of R&D funding to aggregate, economic outcomes on the societal, industry, or firm level can provide compelling evidence of the overall value of research, but this avenue is not open to us in the field of occupational safety and health research for at least three main reasons. First, even though we could substitute safety and health outcomes for economic outcomes to perform an aggregate analysis, we lack data needed to estimate changes over time in occupational safety and health outcomes and research investments. We have reasonable data on injury rates showing a long-term decline that has been dramatic, but we lack similar data on change in rates of occupational illnesses. Individual occupational illness cases are not usually documented, and our estimates of their incidence are based on the estimated work-attributable fraction of the overall incidence of these illnesses, the changes in which have been driven mainly by factors outside the workplace.17 Data might possibly be assembled on the history of occupational safety and health investments, but this is a challenging task that has not been undertaken.
The second reason that aggregate R&D investment benefit studies are not feasible in occupational safety and health is that we do not have a basis for starting with a clear presumption that new research-based knowledge has had a major impact on safety and health, even if this is so. It is not evident how much research-based guidance has influenced practice. There are many channels through which such influence takes place, including unions, trade groups, business periodicals, conferences, Occupational Safety and Health Administration (OSHA) consultation service providers, safety and health consultants, and safety and health professionals working at employers and in workers compensation insurers, but the general level of effectiveness of this array of parties in research translation is unknown. Furthermore, it has been very difficult to independently track guidance implementation, since much of the guidance can be implemented by alteration of everyday practice, without being marked by installation of new technology. Thus, it is not surprising that there are no mechanisms to gather implementation information in a consistent way over time in a large and enormously varied population of workplaces.
A third reason why aggregate analysis is not a viable option is that any observed statistical association between R&D and safety and health outcomes would be affected by other major factors, especially general trends in awareness and value of safety, technology, and management changes unrelated to safety, and changes in incentives to avert costs of injury and illness, and perhaps others. These factors are very difficult to measure, especially on an aggregate level, with changes differing by sector. Without a pre-existing presumption that research has had major impacts on safety and health, and without a reasonable way to measure other known factors, we may neither detect nor confidently interpret any aggregate statistical association between R&D investment and safety and health outcomes.
Given the lack of viability of measuring impacts of occupational safety and health research on the aggregate societal or industry level, we must look to examples of previous, research-benefit research that rely on tracing the specific connections between research, practice, and ultimate outcomes. Two of the best examples are sets of analyses sponsored by the Department of Energy and the National Institute of Standards and Technology (NIST) of the Commerce Department. The DOE’s Energy Efficiency and Renewable Energy (EERE) program has produced six economic benefit studies of its research.18 An example is an evaluation of the HVAC, Water Heating, and Appliance program.19 Three research efforts within this program were selected, presumably based on the expected size of their contribution to benefits. The total benefits for these three efforts were compared to the total costs of the entire research portfolio in this program. Benefit estimates were based on a set of 91 interviews with experts who were asked to estimate what trends in energy efficiency would have been in the absence of DOE EERE research. Fuel and energy savings were translated into cost savings, and some additional health benefits from reduced air pollution were calculated using an EPA model. Benefit-cost ratios of 20–66 were estimated.
NIST produced a series of 17 economic impact analyses focused on specific research programs involving calibration, testing, and standards. Link and Scott (2012) provide summaries of these analyses produced between 1997 and 2009.20 One example focuses on a thermocouple program. Thermocouples are devices for measuring temperature and are used in processes that require fine temperature control. Ten industrial users of thermocouples and seven-wire suppliers to thermocouple manufacturers were surveyed. The wire suppliers, similar to the DOE HVAC and appliance study, were asked to estimate the likely future time path of costs if NIST were to cease production of its “reference tables and functions, test methods, and calibration services.” These counterfactual costs were expected to arise from use of foreign laboratories, the gradual formation of an alternative private consortium to provide substitute standards and services, and the need to hire additional personnel or consultants to replace telephone consultations with NIST. Benefits to the users of more accurate thermocouples were judged to be even more important, but could not be estimated since they are not reducible to a set of cost savings, but instead take the form of fewer quality problems in production and higher customer satisfaction with end products.
Another NIST example is an evaluation of research on combinatorial methods for rapid analysis of large numbers of related chemical samples. The evaluation focused on the period 1998–2007 and relied on semi-structured interviews and surveys, reaching over 70 scientists from companies representing over 70% of total R&D spending among those who use combinatorial methods in the key, relevant industries.21 Total costs included not just NIST research, but membership and project costs of firms who joined a public-private research consortium set up to develop and disseminate the new methods. Costs also included purchases of NIST technology by private firms. The estimated benefits were due to improved efficiency of research in private firms. As with the previous example, benefits to end users of products based on the research could not be quantified.
The most important feature of the DOE and NIST examples is that they provide models for estimating impacts of research activities through consultation with experts who can estimate what would have happened in the absence of specific research results. This was also an important part of the methods of the RAND case studies (see Section 4.4 below on counterfactual scenarios).
The CDC has set forth methods for estimating effectiveness of public health prevention measures that also provide a partial precedent and model for the NIOSH-sponsored case studies.22,23 Like the DOE, NIST, and NIOSH-sponsored studies, CDC prevention effectiveness studies trace the impact of specific technologies and actions on outcomes, but more similar to the NIOSH-sponsored studies, they focus on population health outcomes. On the other hand, unlike the DOE, NIST, and NIOSH-sponsored studies, they focus on the implementation and impact of prevention measures without concern for whether research was responsible. Many CDC studies of prevention effectiveness are included in lists of publications by participants in the CDC Prevention Effectiveness Fellowship program.24 These studies frequently focus on effectiveness of screening programs, vaccines, treatments, and education for self-care and risk reduction. Some recent notable examples include studies of costs and benefits of outreach to install smoke alarms in high-risk communities,25 the cost-effectiveness of education and contact-tracing programs to prevent sexually transmitted disease,26 the health and healthcare cost impacts of referring patients with poorly control hypertension to team care involving pharmacists,27 and a review of the cost-effectiveness studies of adult vaccines.28 These kinds of studies usually focus on prevention measures applied directly to individuals within broadly defined populations by healthcare providers or community public health programs. This is likely to make it easier to gather information on the number of the individuals affected by the implementation of prevention measures than in the occupational setting where prevention measures are usually implemented privately by employers and applied only to individuals engaged in very specific kinds of tasks.
The CDC and other agencies that focus on improvement of health have developed the economic metrics needed for translation of injury and illness reductions into dollar amounts. Both CDC prevention effectiveness studies and the NIOSH-sponsored case studies measure benefits in terms of (1) saved costs of illnesses and injuries averted in terms of medical costs and lost productivity, and (2) the statistical value of lives saved or illnesses averted based on values from willingness-to-pay (WTP) studies. The first metric, known as “cost-of-illness,” is well developed, having been demonstrated by Rice (1985)29 and later employed by Leigh (2011)30 to construct estimates of the economic costs of occupational injury and illness at the national level. WTP metrics are also well-developed and are routinely used in the federal government to estimate the value of regulations designed to prevent illness and death. Both types of metric are summarized in Haddix et al. (2003).22 When these methods are applied to CDC interventions and occupational injury and illness research, they require data specific to particular injury and illness types. This contrasts with the simpler data requirements of the aggregate, society, and industry-level studies discussed above that measured impact in terms of reduced mortality and increased length of life.
In sum, the NIOSH-sponsored RAND case studies represent unusual efforts to measure the benefits of research with economic metrics. The DOE, NIST, and CDC prevention effectiveness studies provide the best precedents and models for these NIOSH studies. However, the NIOSH case studies employ a unique combination of methods and challenges.
3 |. THE RAND STUDIES AND THEIR RESULTS
Responding to the increasing need for government agencies to demonstrate value added by their activities, NIOSH contracted with the RAND Corporation in 2017 to conduct an initial set of three case studies of the economic benefits of NIOSH research.31 NIOSH has been tracking the impacts of its research, mostly through the use of “intermediate outcomes.” Intermediate outcomes is a term used in logic models to signify “actions by stakeholders in response to NIOSH products or efforts.”2 This can take a wide variety of forms, including citations of NIOSH research by other researchers, further dissemination and adaptation of NIOSH guidance, or most importantly, implementation of prevention measures in the workplace. However, NIOSH has infrequently been able to estimate the number of illnesses and injuries prevented by even its best-known, successful efforts. Clearly, there are data and analytical obstacles to obtaining such estimates. With assistance from RAND, we were able to devote concentrated attention to this problem and develop estimation methods that could overcome these obstacles.
RAND worked cooperatively with NIOSH economists to identify the subjects of the first three case studies. Because NIOSH had not conducted such economic-benefit case studies before, we needed to determine and demonstrate the best methods to use. We also recognized that we needed to start with case studies that appeared to have substantial impacts, at least in terms of having achieved intermediate outcomes. In addition, we needed to have access to data with which we could quantitatively assess both intermediate and injury and illness outcomes. NIOSH economists reviewed NIOSH reports and presentations and queried colleagues to identify such potential case studies and provided a menu of options which RAND researchers evaluated according to their own criteria.
The first of the three selected case studies focused on the design of dust controls for large asphalt milling machines, which are used to remove damaged pavement layers in preparation for resurfacing. Milling machine manufacturers ultimately incorporated these controls in all new large milling machines, thus reducing exposure to silica-containing milling dust and promising to reduce silicosis, lung cancer, and other lung diseases. The second case study focused on a major epidemiological research study that provided the first, widely compelling evidence of elevated rates of cancer among firefighters in the United States, spurring a cultural shift in the industry and leading to widespread adoption of new guidelines for reducing firefighter exposure to carcinogens found in smoke, fumes, and soot. NIOSH actively disseminated the findings and participated in formulation of the new guidelines, which focused mainly on new practices and use of existing technologies, particularly use of self-contained, breathing apparatuses during additional phases of the work, and various cleaning, containment, and positioning practices to avoid respiratory and dermal exposures. The third case study focused on an assessment of the effectiveness of an Ohio Bureau of Workers’ Compensation (OHBWC) program that provided matching grants to employers investing in a wide variety of safety equipment. The NIOSH research showing resulting declines in injury rates led to a major expansion of the program, which inspired similar programs in two other states.
Because the first three case studies were successful in using economic metrics to assess the benefits of NIOSH research, NIOSH contracted with RAND again, late in 2018, for another set of three case studies.32 To select these case studies, NIOSH developed a menu of ten potential case studies. RAND ranked these ten by several criteria, and RAND and NIOSH then agreed on the three final selections. The leading criteria were the presence of an apparent, substantial impact and the feasibility of quantitatively measuring that impact. Another major criterion was the diversity of case study characteristics such as type of research and industry, because this would help to further develop insight into the challenges of, and methods for quantitatively assessing economic benefits.
The first case study of this second set focused on the development of a personal dust monitor (PDM) for use by underground coal miners. The use of these monitors was enforced throughout the industry as part of a new Mine Safety and Health Administration (MSHA) coal mine dust rule.33 The monitors were designed to provide continuous, real-time readings that were useful in alerting miners to the need to fix ventilation systems problems and to reposition themselves to avoid dust. The second case study focused on the development of safer patient compartments in the rear of ambulances that provide new, more practical passenger restraints and reduce the likelihood of impacts with flying and fixed interior objects during collisions, sudden stops, starts, and swerves. Ambulance purchasing standards in almost all states incorporated the new designs. The third case study focused on the development of a system in Michigan for surveillance of work incidents resulting in an amputation (usually finger amputation). The system developers have been providing this information to the Michigan Occupational Safety and Health Administration (MIOSHA), which has been using it to target inspections. These inspections revealed that many of the workplaces where amputations had taken place had not controlled the hazards associated with these amputations.
Table 1 provides a summary of basic results for all six case studies. The principal monetary results are in the third column, which reports the averted burden on society due to the prevention of illness and injury. RAND chose to focus on annualized benefits measured in dollars to facilitate the comparison of the magnitude of benefits among case studies. The case studies varied widely in the number of years over which benefits were estimated, as reported in the fourth column. A key reason for this was that changes in exposures often take many years to have their full effect on worker health, while in other case studies involving injury, prevention measures are effective as soon as they are fully implemented. Note that due to the lack of adequate information, RAND was unable to estimate some categories or types of benefits. The last column of the table lists some of these important but omitted benefits.
TABLE 1.
Results of RAND case studies on benefits of NIOSH research
| Case study | Illness and injury cases averted (annual average) | Monetary results (annualized) | Time horizon | Examples of omitted benefits |
|---|---|---|---|---|
| Development of silica dust controls in asphalt pavement milling (asphalt milling) |
|
|
|
|
| Building and disseminating evidence on firefighters’ cancer risk (firefighter cancer) |
|
|
|
|
| Assessing impacts of Ohio safety intervention grants for safety equipment (safety grants) |
|
|
|
|
| Developing continuous personal dust monitors (PDMs) for coal miners (coal miner PDM) |
|
|
|
- |
| Re-design of ambulance patient compartments for safety (ambulance redesign) |
|
|
|
|
| Work-related amputation surveillance in Michigan (amputations) |
|
|
|
|
Note: Annualized figures represent conversion of a varying stream of estimated dollar values over the time horizon of the analysis into an equivalent, constant stream of unchanging annual values with the same total present value, using a discount rate to represent the lower value of benefits received further in the future. Ranges for estimated benefits are based on alternative assumptions and discount rates (3% and 7%). WTP values are willingness-to-pay values, as explained in the text. Dollar figures for asphalt milling and firefighter case studies in 2016 dollars, for safety grant case study in 2013–2017 dollars, and for PDM and ambulance redesign case studies in 2018 dollars.
Abbreviations: NIOSH, National Institute for Occupational Safety and Health; NMRD, nonmalignant respiratory disease; OSHA, Occupational Safety and Health Administration.
What do these results suggest, and what conclusions might we draw? The first, most basic observation is that these case studies successfully constructed estimates of benefits expressed in terms of economic metrics. Thus, we demonstrated that, for at least some selected research efforts, economic benefits can indeed be estimated. Second, most of the case studies generated estimates of lives saved and nonfatal injuries or illnesses prevented, which are additional and usually unavailable basic metrics of impact. This lends concreteness to the results of NIOSH research efforts that have already been widely recognized as successful. Third, the estimates come with substantial uncertainties as represented by the wide ranges that were based on calculations with alternative assumptions. Fourth, despite uncertainty in the estimates, they still serve to differentiate the scale of benefits achieved by different research efforts. This could be helpful in setting research priorities.
Fifth, by design, the monetary economic benefit estimates are expressed with two different and widely accepted metrics.22 The first method assesses reduction in medical costs and productivity losses resulting from worker injury and illness. This method uses directly observable medical costs and the number of workdays missed due to absenteeism, disability, and death. It is the most commonly used method and less dependent on complex statistical analysis and assumptions than the second method. The second method uses assessments of “willingness-to-pay (WTP)” to reduce risk of fatalities and nonfatal injuries and illnesses, based on a body of literature on how much individuals are willing to pay for reductions of risk. This method is required for cost–benefit analyses included in government regulations. Its strength is that it assesses the value of avoiding the experience of suffering and of preventing death. The difference between the emphasis on observable costs in the first method and the emphasis on a more holistic approach in the second method results in a large difference in the scale of benefits derived by each method.
Our sixth observation is that, as previously mentioned and shown in Table 1, not all the benefits of each case study were included in the estimates. Such omissions are generally due to lack of data, which is a common problem that clearly leads to some downward bias in the benefit estimates.
Our seventh observation is that some of the economic benefits demonstrated by the case studies are larger than the annual NIOSH research budget and the research costs associated with each case study. In 2019, for example, the NIOSH budget was 336 million dollars. While NIOSH research costs were not fully estimated for all of these case studies and were not part of RAND’s estimates, they generally ranged from a total of 100,000 or 200,000 up to 2 or 3 million dollars over multiple years. This immediately suggests that NIOSH research is well worth the investment. It also leads us to the eighth observation, which is that the NIOSH cost does not reflect the full cost of achieving these benefits. The full cost would include costs of other organizations who disseminated NIOSH research results and the organizations and employers who purchased or used internal resources to implement preventive measures. The case studies did not include prevention costs, largely because information on costs of organizations outside NIOSH periods that may span several years is difficult to collect. This information might possibly be collected in the future to create a more complete picture of costs and benefits.
4 |. SUMMARY OF RAND CASE STUDY METHODS
In general, RAND took the following steps to estimate the benefits of NIOSH research:
Step 1. Establish a timeline of NIOSH activities and outputs, and significant stakeholder activities and actions.
Step 2. Estimate the reduction in illness or injury due to prevention actions associated with the research.
Step 3. Estimate the monetary value of averted injuries or illnesses.
Step 4. Estimate the probable net effect of NIOSH research by estimating the reductions in injuries, illnesses, and their costs that might have occurred in the absence of NIOSH research.
The first step, establishing a timeline of NIOSH activities and stakeholder actions, defines the scope of activities the case study includes. Further, the timeline provides an integrated view of NIOSH and stakeholder actions. This helps us to identify the significant stakeholder actions that preceded and followed NIOSH actions, and to understand the potential relationships between research outputs and the actions they may have informed or inspired. The second step, estimating the reduction in injuries and illness, is the heart of the challenge in conducting a case study, sometimes requiring creative use of available data, as well as careful use of informed assumptions. This step requires estimating the extent to which prevention measures are implemented and estimating their effectiveness. The third step, assigning monetary value to averted injuries and illnesses, follows more standard methods that are not as specific to a case study but do require information specific to certain injuries or conditions that is sometimes difficult to obtain. The final step, estimating what would have occurred in the absence of NIOSH, is based on expert judgment and is necessary for arriving at a reasonable estimate of benefits that can be attributed to NIOSH research. We should also note that an additional step was taken in one of the injury case studies (safety grants). In this step, productivity changes stemming directly from the prevention measures themselves, rather than their health and safety impacts were added to the tally of benefits.
Out of the six case studies, three focused on illnesses and three focused on injuries. After all case studies were completed, we were able to recognize that illness and injury case studies generally required somewhat different approaches for Step 2, estimating the reduction in illness or injury. Next, we separately review in more detail how the reductions in illnesses and injuries were estimated. Following that, we discuss the details of Steps 3 and 4.
4.1 |. Illness case studies: Estimating the reduction in illnesses
The three case studies that focused on illnesses focused on asphalt milling, firefighter cancer, and PDMs. Table 2 summarizes the steps taken to estimate the reduction in illnesses. Steps of the estimation method are listed across the top—one column for each step. While this method consists of three general steps, there were significant variations among the three case studies in the strategies required to carry out these steps, as the discussion below will explain.
TABLE 2.
Illness case studies: Steps taken to estimate reduction in illnesses (information sources and estimation methods)
| Case study | Step 2a: Population of workers to which prevention is applied (estimation challenges and facilitating factors) | Step 2b: Prevention effectiveness in reducing exposure (information source) | Step 2c: Reduction in number of illnesses (information source and method) |
|---|---|---|---|
| Asphalt milling |
|
Surveillance and field engineering studies | Risk assessment used in economic analysis of OSHA silica rule |
| Firefighter cancer |
|
Expert judgment about percent reduction in overall exposures due to multiple prevention measures | NIOSH epidemiological study of firefighter excess risk, plus assumption that percentage decrease in exposure = percentage decrease in excess risk |
| Coal miner PDM |
|
Coal mine inspection data, pre- and post-implementation | Risk assessment used in 2014 MSHA coal mine dust rule |
Abbreviations: MSHA, Mine Safety and Health Administration; NIOSH, National Institute for Occupational Safety and Health; OSHA, Occupational Safety and Health Administration; PDM, personal dust monitor.
In Step 2a, RAND estimated the size of the population to which the prevention was applied. This estimation can be viewed as breaking down into two parts: estimating the size of the target population of the relevant prevention measure, and then estimating the percentage of this population for which the prevention measure was actually implemented.
The asphalt milling prevention measure targeted those who operate large milling machines. This occupational group is too small and specific to be captured in standard government surveys. OSHA had obtained an estimate of the population of workers who use pavement milling machines of any size for the analysis supporting its silica rule.34,35 The share of these workers who used large machines was estimated based on OSHA’s estimate of the percentage of milling machines in each size class, combined with expert judgment about average crew sizes for each machine size. Because the new prevention measure was a technology built into all new, large milling machines, RAND could assume that 100% of large milling machine operators would benefit from exposure reduction after a certain number of years.
The firefighter population was large and of known size, but RAND had to estimate the proportion of that population working in departments that implemented the new recommended prevention measures, and how fully these departments actually implemented them. In the absence of actual data, RAND used expert judgment to estimate these parameters and sensitivity analysis to cope with the uncertainty. Expert judgment was based on the fact that a large proportion of fire departments participate in events and activities of national firefighting organizations which serve as a major source of authoritative firefighting information. Another uncertainty was the extent to which the new prevention measures would benefit volunteer firefighters. RAND had limited information on their cancer risks and their fire departments’ rates of adoption. Thus, they did not include these firefighters in the analysis.
The underground coal miner population was similarly large and of known size. RAND also knew that implementation rates were close to 100% because use of the new PDM technology was required by MSHA’s recent coal dust rule implemented in 2016.33
In Step 2b, RAND estimated the reduction in exposure due to the prevention measures. This is an exposure to a substance or a set of substances that can be summarized by a single number, and which, in a later step, can be linked to a rate of illness. This is a step that only applied to the illness case studies, since in the injury case studies, there was no such summary exposure measurement available, and impact of prevention measures on injury outcomes had to be estimated directly, as explained below. While all three illness case studies estimated a reduction in exposure, each of them required a different strategy to accomplish this.
This step was easiest in the asphalt milling case study, because we could compare results from engineering tests of exposures with the new controls with previous surveillance study measurements of silica exposure without the new controls. The previous surveillance studies had been part of NIOSH-supported efforts to document previously under-recognized exposures to silica in a range of construction tasks.
In the firefighter cancer case study, we had to rely on expert judgment. This was not simple. First, research industrial hygienists estimated the share of several routes of exposure to combustion byproducts. Then, these experts estimated the percentage by which exposure via each route was reduced by the newly applied prevention recommendations. The experts used insights and data from their field research to develop these estimates. The detailed rationale the experts followed to develop these estimates are included as an appendix in the RAND report.
In the PDM case study, the biggest challenge was using MSHA coal mine inspection data to determine the effectiveness of using the PDM. By luck, MSHA required the use of PDMs 6 months before other requirements of the new coal dust rule came into effect. Thus, this 6-month period promised to reveal the decrease in exposure due only to the new PDM. However, inspection samples were collected for a mix of occupations and work locations that was continually shifting over time, and exposures exhibited strong seasonality. Therefore, much statistical work had to be done to detect the PDM impact on exposure.
After completing Steps 2a and 2b, the basic approach for estimating the resulting decrease in illnesses in Step 2c was straightforward in all three case studies. This involved applying existing quantitative analyses of the relationship of exposure level to disease risk. This is also a step that was unique to the illness case studies, since there were no studies relating an exposure measure to rates of injury outcome. In the asphalt milling and PDM case studies, these analyses had already been identified and synthesized as part of the benefit-cost assessments that are required for implementing new OSHA or MSHA rules. In the firefighter cancer case study, the estimate of the risk of cancer due to firefighter exposures came from the NIOSH research that estimated this risk and was the central focus of the case study. The percent reduction in excess risk was assumed equal to percent reduction in exposure.
4.2 |. Injury case studies: Estimating the reduction in injuries
The three injury case studies focused on safety grants, ambulance redesign, and amputations. As with the illness case studies, the three injury case studies represented a significant degree of diversity, leading in turn to significant diversity in methods used, even though they follow the general steps listed above for injury case studies. Table 3 summarizes the steps taken to estimate the reduction in injuries.
TABLE 3.
Injury case studies: Steps taken to estimate reduction in injuries (information sources and estimation methods)
| Step 2a: Injury or injury cost rate in the population to which prevention is applied | Step 2b: Prevention effectiveness in reducing injuries or injury costs | Step 2c: Reduction in number of injuries or injury costs | |
|---|---|---|---|
| Safety grants | Assumption: Share of grant beneficiary population in state total workers’ compensation claim cost = share of grant beneficiary population in insured population | NIOSH study based on longitudinal claims data for workers affected by grants | Multiplication of baseline workers’ compensation claim cost amount by cost reduction percentage found in NIOSH study |
| Ambulance redesign | National police report data on crash injuries by vehicle type and occupant location in vehicle used to extract data on ambulance patient compartment injuries | National police report data on crash injuries for vehicle occupants for whom similar engineering standards already met. Also, general seat belt effectiveness literature | Difference in injury rates obtained in first two steps Also, percentage reduction associated with seat belts applied to baseline injury rate |
| Amputations | Data unavailable: Neither OSHA inspection data nor surveillance data tracks injury rates and costs for individual employers | Data unavailable to estimate reduction in injuries or injury costs, but effectiveness of amputation inspections in increasing number of violations estimated by comparing results of amputation and other inspections | Not estimated |
Abbreviation: NIOSH, National Institute for Occupational Safety and Health; OSHA, Occupational Safety and Health Administration.
Step 2a involved estimating the injury rate or injury cost rate in the population to which the prevention was applied. Note that the injury case studies did not focus on measuring exposure as the illness case studies did. The illness case studies focused on exposure to a single substance or to a single source of harmful substances that is considered to be the main illness risk, and this exposure could be measured quantitatively by a single, continuous variable. In the injury case studies, the source of injury risk is not reducible to a single exposure, and usually includes many factors related to equipment, work environment, workload, materials, management, worker behavior, and a host of specific circumstances. As a result, the risk of injury is usually defined by the rate of injury outcomes for workers in a particular task or occupation—that is, the rate of injury or the rate of cost for injury, per worker and per unit of time. In two of the three injury case studies, it was possible to estimate the rate of injury or injury cost in the relevant population, although using different strategies to do so.
In the safety grants case study, the cost of injuries was estimated for the population of workers affected by the grants. While the workers’ compensation insurer could estimate this number using grant program records, it would be available with a time lag. Therefore, RAND used a simple approximation instead. First, they calculated the share of the safety grant worker population in the state’s total insured population. Then, they applied this share to the state’s total workers’ compensation claim costs and derived an estimate of the workers’ compensation claim costs attributable to the workers benefiting from the safety grants.
In the ambulance redesign case study, RAND extracted the number of injuries and the distribution of their severity levels from a national database based on police reports. This was straightforward, but the data was quite unstable from year to year for unknown reasons. Another drawback to this data was that it did not include injuries caused by hard braking or sudden acceleration events that were not associated with crashes. Those injuries were also a target of the design improvements. In addition, urban and rural rates were tabulated separately, because the rate of adoption of the design improvements was very different in these areas.
The amputations case study was the most difficult, and it did not yield economic benefit estimates. One reason was that data on injury rates and injury costs were not available for the inspected workplaces where amputations had occurred.
Step2b required an estimate of prevention effectiveness in reducing injuries or injury costs. This step was unique to the injury case studies, but it is analogous to the step in the illness case studies in which prevention effectiveness in reducing exposures was estimated. As for other steps, there were important differences among injury case studies in the strategies needed to complete this step.
In the safety grants case study, the NIOSH study of prevention measure effectiveness that was the subject of the case study had already provided such estimates.
In the ambulance redesign case study, constructing credible prevention effectiveness estimates was the biggest challenge. There was no data demonstrating the effectiveness of the redesigned ambulances in the field, and there were many engineering improvements made. Each of the improvements was tested in controlled conditions, but those tests did not yield any overall measure of effectiveness in preventing injuries. However, the tests used to evaluate the new designs used equipment and standards that were meant to be equal to or surpass those that had already been used to upgrade safety in other vehicles and occupant positions. Based on this, the new injury rates and severity levels were assumed equal to those for passengers and drivers who had already benefited from analogous improvements. Thus, the injury rates for these other passengers and drivers were extracted from the police report data. Two other groups were taken to represent the new improved safety levels achieved for ambulance patient compartment occupants: back-seat passengers of automobiles, and drivers of ambulances. In addition, the safety belt effectiveness literature based on observed results of seat belt use in other settings was used for a third estimate of projected percentage declines in injury rates, because passenger restraints were central to the ambulance improvements.
As mentioned, the amputations case study did not produce benefit estimates. Again, data were not available on pre- or post-inspection injury rates, and even if they were, RAND did not have a valid comparison group. Such a group could theoretically be constructed through randomizing the choice of amputation workplaces to inspect and comparing subsequent injury rates, but any such experiment would be problematic in the context of an enforcement program.
The amputations case study did demonstrate that the inspections targeted by information on amputations yielded a greater number of violations and greater total fines. However, RAND could not assume that inspection effectiveness was proportional to violations or fines. There is some literature on effectiveness of OSHA inspections on reducing injuries, but these studies have not provided quantitative estimates of the effectiveness of finding more violations per inspection.36–38
In Step 2c, the last step in Table 3, calculation of the reduction in injuries and injury costs, was straightforward, requiring only the estimates made in the previous two steps.
4.3 |. Assigning values to prevented injuries and illnesses (Step 3)
In the previous section, we saw that there was a general difference in the methods used in the illness and injury case studies, but variation within these two groups of cases in the strategies used to follow these methods. When it comes to assigning values to prevented injuries and illnesses, there is more commonality between illness and injury cases, but we will note some differences in data availability and requirements.
As mentioned above, where possible, we used two methods for assigning dollar values to prevented injuries and illnesses in each case study: WTP and the sum of medical costs and productivity losses. For fatal injuries and illnesses, the WTP method uses relatively consistent, value of a statistical life (VSL) estimates that are usually applied to all deaths equally, regardless of age or other circumstance. This makes these estimates simple to use. RAND used a VSL value from Viscusi and Aldy (2003)39 expressed in 2016 dollars (9.5 million) or 2018 dollars (10.0 million) for most case studies, but a value of 10.7 million in 2018 dollars taken from the National Safety Council40 for the ambulance safety case study.
4.3.1 |. WTP for avoiding nonfatal illnesses and injuries
Unlike values for VSL, WTP values for averting nonfatal cases of illness and injury are different for each illness or for each injury severity level, and there is no single set of widely accepted values. In the asphalt milling case study, RAND followed OSHA in using a low-end estimate of $68,000, a high-end estimate of $5.5 million, and a midpoint estimate of $2.8 million for the value of averting a nonfatal illness. In the firefighter cancer case study, RAND did not report separate values for nonfatal and fatal cases, but noted that their high and low estimates of total WTP benefits (nonfatal and fatal combined) varied by a factor of 2.3 depending on the assumption about WTP values for morbidity. In the PDM case study, a variety of sources were used for WTP estimates to avert nonfatal illness. The least severe form of coal workers’ pneumoconiosis (CWP 1+) was assigned a value of $71,000 for lost workday injuries; the next most severe form (CWP 2+) was assigned a value of $490,000 for chronic bronchitis from an Environmental Protection Agency publication,41 and values for progressive massive fibrosis (PMF) and COPD were assigned values using estimates assembled for the most recent coal dust rule, which were derived from two cited publications.42,43 These values averaged 36% of the value of averting death, so that figure was used to arrive at an estimate of $3.6 million for PMF and COPD. In the ambulance redesign case study, estimates of WTP values for nonfatal injuries of different severity levels were taken from a publication of the National Safety Council (2017).44
In the safety grants case study, no attempt was made to estimate WTP to avert nonfatal injuries, due in part to the fact that injury rate reductions were not estimated in that case study. Instead, cost reductions (medical costs and productivity losses) resulting from the prevention measure were estimated directly.
4.3.2 |. Medical costs and productivity losses of averted illnesses
Success in estimating medical costs and productivity losses of illnesses was mixed. Medical costs of cancer cases are available in more detail than other conditions. Only for cancer are there readily available estimates for how annual healthcare costs change over time from year of diagnosis to death, which allows proper discounting of future cost savings. For other illnesses, typically only average annual costs across all illness stages are available. In the asphalt milling case study, medical costs for fatal cancer cases but not for fatal cases of PMF or nonmalignant respiratory disease (NMRD) were estimated. In the firefighter cancer case study, different forms of cancer were the only illnesses, so medical cost data was available. In the PDM case study, average annual medical costs for CWP and COPD were each available from specialized published studies or tabulations, and these were combined with estimates of years of treatment based on assumptions about average age at diagnosis and lifespan. Medical costs associated with the broad category of deaths due to NMRD were calculated from results of a well-known study by Leigh (2011) of total US costs of occupational injury and illness.30
Productivity losses due to illness can be divided into those that occur due to early death, and those that occur before death due to loss of employment, absenteeism, and loss of ability to do household work. Those that are due to early death are most easily estimated, based on published tables giving average earnings and household work values for men and women by age.45 While these are not specific to people working in different occupations, they can be adjusted for the average income of the relevant occupational group, as they were in the asphalt milling and PDM case studies. Productivity costs incurred before death are more difficult to estimate. In the PDM case study, RAND followed Leigh (2011)30 by multiplying productivity costs due to early death due to NMRD by a ratio of pre- to post-death productivity costs for fatal cases estimated in an early, foundational study of costs of occupational injury and illness by Rice et al. (1985)29 Productivity losses associated with nonfatal cases of CWP, PMF, and COPD were based directly on estimates in Leigh (2011).30 Similar methods were used for productivity costs associated with fatal lung cancer in the asphalt milling and firefighter cancer case studies.
4.3.3 |. Medical costs and productivity losses of averted injuries
When injuries result in fatalities, their productivity costs can be estimated based on age at death, just as they are for deaths due to illness. Their medical costs can be estimated using workers’ compensation records. However, the share of fatal injuries in total injuries is small, so estimating the cost of nonfatal injuries is particularly important. The safety grants case study only involved nonfatal injuries. The medical costs and productivity losses associated with these injuries were estimated using data on costs of workers’ compensation claims, whose benefits are intended to pay for all medical care for work injuries and replace a certain percentage of lost earnings (indemnity benefits). However, available claim cost information does not typically include administrative costs for delivering these benefits, which are substantial. Only a few estimates of these costs are available, contributing to uncertainty in the RAND estimates. More important is the fact that indemnity benefits may not be a good indicator of loss of earnings if these losses go beyond the loss of wages recorded for the claim. RAND provided one estimate of total lost earnings based on the formulas for determining percent of wages replaced by benefits and another much higher estimate based on an estimate of income losses over 10 years following work injury claims that was published for New Mexico.46 In the ambulance redesign case study, medical costs and productivity losses associated with crashes were derived from National Safety Council (2017)44 estimates of total crash costs, inclusive of property damage, medical costs, productivity losses, and administrative costs. Estimates were published both with and without injuries, with the difference being attributed by RAND to injury costs. Note that property damages are not usually included in cost of injury estimates. Though they might be relevant to other injury prevention measures, they do not apply in this situation, because the prevention measures were designed to reduce the injury consequences of crashes, but not crash events.
4.4 |. Estimating illness and injury reductions in the absence of NIOSH (Step 4)
The previous steps constructed an estimate of the value of observed or projected illness and injury reductions. But the degree to which NIOSH is responsible for these reductions needs to be assessed. In each of the six case studies, this assessment was done based on whether there were any other known research efforts that could have produced similar technology or information, and whether it was reasonable to assume that the prevention activities that led to the reductions would have happened in the absence of NIOSH. From a short-run perspective, the answers were generally clear. In the asphalt milling, PDM, and ambulance redesign case studies, no one else other than NIOSH and its partners was doing similar research, and the illness reductions depended upon the new technologies created.
In the firefighter cancer case study, when NIOSH began its work, there was already concern in the firefighting community that firefighting exposures may be leading to higher cancer rates. In addition, there were a number of small US studies that supported this idea, as well as three notable large-scale studies in other countries emerging in the same general time period. But there was no consensus about whether firefighter exposures were causing higher cancer rates. Industry experts indicated that it was clearly the NIOSH cancer study, using rigorous methods and a large sample of US firefighters, that crystallized a new consensus about the dangers of firefighter exposures and led very quickly to a nationwide shift in attitudes that drove efforts to implement a common set of prevention measures.
In the safety grants case study, the expansion of the grant program occurred very quickly after the preliminary results became known and was directly driven by those results, according to the program’s administrator. The Michigan amputation surveillance system created with NIOSH support was clearly dependent on NIOSH funding and led directly to targeting some OSHA inspections with information that was not otherwise available.
Because NIOSH research was indispensable for realizing illness and injury reductions, all of these reductions could be counted as benefits of NIOSH research. But over the long run, it is less likely that at least some reductions would not have occurred by other means. To take account of this possibility, longer time-range counterfactual scenarios were constructed and used for two of the case studies—asphalt milling and safety grants case studies. A counterfactual scenario projects what might have happened, or might happen in the future, if NIOSH had not acted. This step can be critical to both illness and injury case studies.
Both the asphalt milling and safety grants case studies used a simple form of counterfactual—an assumption that the same outcomes would have been achieved without NIOSH, but with a delay. This type of counterfactual essentially modifies the baseline injury or illness trend to which the post-NIOSH trend is compared. For asphalt milling, the counterfactual was that engineering controls would have been developed in Europe and then introduced widely in the United States 15 years later. There were some known research efforts in Europe, although they had not translated into practice. This counterfactual results in somewhat less than half of the estimated benefits without the counterfactual. In the safety grants case study, the program’s administrator estimated that its success might have otherwise become known 5 years later and then expanded, just as it was following the NIOSH research.
Other case studies did not include a long time-range counter-factual, although they could have. For example, just before NIOSH began its study of cancer among firefighters, the National League of Cities stated in 200947 that there was a lack of evidence to confirm or deny the link between firefighting and cancer, but there was already widespread concern in the firefighting community about potential cancer risks. There were also several related important studies in other countries and there could have been additional studies in the absence of NIOSH. Therefore, it could be straightforward to apply a delayed firefighter protection counterfactual to the RAND study and derive lower, alternative benefit estimates, as was done in the asphalt milling case study. We note, however, that while possible future impacts of foreign studies in the United States were not taken into account, neither were the positive impacts of NIOSH studies on the rest of the world.
Although counterfactuals are speculative, they can have a large impact on estimated benefits. Thus, they need to be constructed based on known facts, and be perceived as reasonable. Knowledge of international developments is sometimes critical to project the future, and the asphalt milling and firefighter cancer cases are good examples.
Scenarios that reach decades into the future are subject to large uncertainties, including the difficulty of foreseeing technological change. Automation may greatly reduce the need for some current jobs. Changes in materials, sources of energy, or means of delivering products and services could substantially reduce or increase the hazards to which workers are exposed, without any change in level of attention to safety and health. For example, it is not hard to imagine that the need for asphalt pavement milling and coal mining could change in unexpected ways in the future, or that the means of emergency medical transport or the techniques of firefighting might be subject to large changes. While detailed speculation about the distant future might not be productive, it could be worth noting signs of the general direction of future trends.
4.5 |. The contributions of other stakeholders to the outcomes
In all six case studies, the NIOSH contribution was not the only one that was necessary to achieve the outcome. Some of the major contributions from other stakeholders are summarized in Table 4. We note that counterfactual analyses of the contributions of these other stakeholders could reasonably conclude that they also were responsible for the outcome.
TABLE 4.
Contributions of nonemployer stakeholders
| Asphalt milling | Coal miner PDM |
|---|---|
| Critical roles of milling machine manufacturers in designing and implementing control, and OSHA in motivating control development and use through silica rule | Some of original impetus for development came from outside NIOSH; and industry, labor, and MSHA contributed to development and testing. Implementation required in new MSHA rule |
| Firefighter cancer | Ambulance redesign |
| Critical roles of firefighter industry and worker representative organizations in developing and promoting recommendations based on NIOSH findings | Large number of public and private partners contributed to design development, and standard setting organizations adopted standards |
| Safety grants | Amputations |
| Ohio Bureau of Workers’ Compensation proactive in seeking program evaluation and supported NIOSH evaluation in kind | OSHA used the amputation information to target workplace inspections |
Abbreviations: MSHA, Mine Safety and Health Administration; NIOSH, National Institute for Occupational Safety and Health; OSHA, Occupational Safety and Health Administration; PDM, personal dust monitor.
A full cost–benefit analysis would include all parties who incurred costs to bring about the result. In such a framework, total costs could be compared to total benefits, and there would be no need for the benefits to be allocated to various contributors. Obtaining information on all of the costs of other parties involved in research, dissemination, and implementation would present challenges that were not addressed in the RAND case studies, especially because these costs are incurred by numerous parties and may not all be recorded as separately identifiable expenditures of time and funds.
5 |. HOW CAN ECONOMIC ESTIMATION OF NIOSH RESEARCH BENEFITS BE MORE WIDELY APPLIED?
5.1 |. Factors contributing to feasibility of RAND case study benefit estimates
To assess the extent to which we can more widely apply the approach RAND used to estimate economic benefits of NIOSH research, we need to understand how several factors were critical to the feasibility of the six case studies described. Some of these factors were present to some degree in all case studies, and some were important in only a subset.
The first factor is the presence of a clear link between NIOSH research and subsequent prevention implementation in the workplace. This link was key to the selection of the case studies, although RAND had to clarify and confirm it. For example, in some instances, NIOSH created new technology that was not available elsewhere, and in others, the expert informants stated that NIOSH research had initiated prevention activities, and the timing of events supported this.
The second factor was the presence of government regulation, which typically requires universal implementation of certain prevention measures in a well-defined population. This enabled specification of the timing and extent of prevention implementation. The establishment of regulations based on exposure limits also requires a formal risk assessment, which provides needed information on the relationship between reduction in exposure and reduction in number of illnesses. In the ambulance redesign case study, regulation took the form of establishment of purchasing specification standards by the states, so the risk assessments that are part of the traditional federal regulatory process were not available.
The third factor was the presence of a strong network of industry and labor organizations. The firefighter cancer case study did not involve regulation, but a fairly standard set of recommendations was disseminated very widely in the firefighting community, with most departments voluntarily deciding to implement them. This appears to be the result of a very strong network of active organizations that shares and improves techniques and knowledge of a wide variety of firefighting subjects, as well as a national firefighter community that has been organized to address health issues.
A fourth factor was the focus on prevention measures embodied in technologies that, once installed, either automatically or typically lead to risk reduction. The simplest case study was asphalt milling, where the technology was built into the milling machines. The ambulance redesign case study also centered on a technology improvement, but was more complicated, with multiple engineering control improvements and a reliance on workers to adopt a new habit of using restraints. The firefighter cancer case study was centered more on changed practices than new technology, but the effectiveness of the techniques and equipment that were recommended could still be tested and measured, and this informed expert judgments upon which the analysis relied. In the PDM case study, the effectiveness of prevention depended upon miners responding to the dust levels the PDM displayed. Still, this single technology was a clear marker of implementation. Using small numbers of miners, additional NIOSH research verified that miner actions to reduce exposure did indeed flow from use of the technology.48
A fifth factor in some of the case studies was implementation of the prevention measure in a well-defined group. This factor was sometimes present due to regulatory requirements as mentioned above, either directly or indirectly. Without a well-defined population, it is harder to quantify the number of workers targeted and the extent to which the prevention measure was implemented. For example, it was harder to define the number of those working with large asphalt milling machines than the number of miners using PDMs. In the firefighter case study, the extent of prevention implementation in the relevant population was uncertain, but professional firefighters were a well-defined group of known size. In the safety grant case study, the group benefiting from prevention was not defined by occupation or task, but it was clearly identified, because the prevention measures were funded and documented on the individual employer level by a government program. The previously mentioned small case studies from the Institute for Work and Health of Ontario also provide an illustration of how frequently government program administration can be a key to defining target worker groups. Only five of their 42 published case studies were designated as documenting benefits (“societal outcomes”) such as cost reductions, shortened time away from work, or injury and illness reductions, and all of these focused on outcomes for populations served by a public workers’ compensation insurance agency.4
A sixth factor in most of the case studies was that the prevention measures assessed were not implemented together with other prevention measures targeting the same risks. This made it easier to measure their impact. For example, the PDMs were just one part of a new rule that enforced lower exposure standards and made inspection protocols more rigorous. Only because use of PDMs was required 6 months earlier than the other parts of the rule was the data able to show the separate effect of the PDMs.
Seventh and finally, case studies were facilitated by the availability of data on exposures in the post-prevention period, as in the PDM case study, or of data on post-prevention injury rates, as in the safety grant case study, which used state workers’ compensation claims data. In the ambulance redesign case study, data was not available for the post-prevention period. However, data was available from police reports on the rates of crash injuries for vehicles and vehicle occupant positions that had previously met the engineering test standards used by the ambulance redesign effort.
5.2 |. Increasing the feasibility of economic measurement of benefits
The case studies conducted by RAND were selected largely because of their feasibility, but the challenges encountered suggest some potential avenues for increasing the feasibility of economic benefit assessment for a wider range of research efforts.
One way to increase capacity to measure the economic benefits of research would be to establish more formal support or processes for gathering information on the long-term impacts of research. This might not only support additional quantitative estimates of impacts, but also offer insights into how these impacts could be increased in the future. Currently, it remains hard to track the transfer of occupational safety and health research to practice by others, including regulatory agencies and the private sector. The transfer often takes place many years after the research is completed and researchers have shifted their attention to other projects. While researchers may learn, through their networks, about uses of their research by others, more focused efforts may be needed to gather information through literature searches, communications with key government and private employer contacts, or surveys of workers and employers.
Another way to increase capacity to measure benefits with economic metrics would be to improve baseline estimates of excess risk or exposure. This information provides a foundation for measuring the benefits that added prevention may bring. Establishing such baseline estimates in some of the RAND case studies was challenging, so other potential case studies may pose similar challenges. For example, baseline measurements of exposures to silica in asphalt milling dust were available, but they were few, and only from some regions of the country. In the ambulance redesign case study, there were no statistics readily available on the excess number of injuries occurring in the rear patient compartment, or their severity distribution, so estimating this baseline risk of injury became a central challenge of that case study. While estimating baseline excess risk and exposure is one of the basic functions of occupational safety and health surveillance, given limited resources, surveillance often focuses on numbers and rates of injury and illness outcomes rather than the hazards and exposures that may cause them. Illness and injury outcome information is often reasonably viewed as the highest priority and is generally easier and less expensive to collect. In contrast, exposure assessment efforts typically require labor intensive involvement of highly trained personnel or specialized surveys. Developing methods of prevention and providing implementation guidance is a higher priority than gathering representative baseline risk and exposure information. And, unlike in some of the RAND case studies, there are typically already several prevention methods in place to varying degrees among many employers, so determining the overall baseline requires more information about both risks and mitigation measures in a wider range of workplaces.
One of the ways that NIOSH research may have an impact is by producing information that can lead to better targeted OSHA inspections. We saw this in the amputation surveillance case study. This was also true in a NIOSH research effort that was considered, but not selected as a subject of a RAND case study. NIOSH researched the rates and causes of carpal tunnel syndrome among poultry processing workers. The main identified impact of that activity was establishment of regional special emphasis programs at OSHA for poultry processors. In both of these examples, measuring the impact of NIOSH research requires knowledge of the impact of OSHA inspections and assistance to employers. Some studies of the impact of OSHA inspections compare employers who received inspections to those who did not36,38 but since improving the targeting of inspections generally has no effect on the total number of inspections completed, these studies cannot be directly used. There are other studies providing strong evidence that inspections with penalties reduce injuries more than those without,37 but we really need quantitative estimates of the impact of shifting inspection resources from employers with lower hazard levels (fewer violations) to those with higher hazard levels (more violations). Research to generate such estimates could thus provide a necessary support for benefit estimates of some NIOSH research.
Identifying and further developing a set of standard values of averting nonfatal injuries and illnesses would also facilitate the measurement of economic benefits. Valuation of fatality reductions is relatively straightforward, because it employs a single standard value applicable to all types of fatalities called the VSL. While the standard value used varies somewhat across federal agencies, there is a single value mandated for use by the Department of Health and Human Services.49 It would help, however, to have standard information on distribution of age at death for various conditions to compute lost productivity associated with death, as well as standard estimates of medical costs by cause of death. The larger challenge is with respect to nonfatal cases, where WTP estimates vary quite widely, as noted by RAND. Medical costs are well-known for major types of cancer since the National Cancer Institute compiles information on costs of medical care over time from year of diagnosis to year of death.50 This time path of costs enables proper calculation of the present value of medical costs based on discounting. However, this information is not usually available for other illnesses. Information on average annual medical costs per illness case have been obtained from sources such as the Medical Expenditure Panel Survey (MEPS), or might be obtained somewhat less comprehensively from commercial medical claims databases. However, this information is usually developed based on research efforts focused on one or a small number of conditions and is not available for many conditions. The same may be said of information on the productivity costs of medical conditions for which MEPS is again the leading source of data. We note that the need for this information is not specific to the occupational safety and health community, so it would make sense to support a broader effort to further develop and keep updated a basic set of cost estimates.
Another way to support expanded use of benefit measurement is to use the RAND case study experience to guide smaller scale, internal benefit assessments by NIOSH personnel. RAND assisted NIOSH by using complex quantitative analysis skills, and an independent perspective on evaluation standards and methods. In addition, the completion of the six case studies has also made it much clearer what the general steps in conducting such an analysis need to be. However, in the process of developing a menu of potential case studies and identifying the basic methods and data available to carry them out, NIOSH also found that the general scale of benefits could be estimated in rough but useful form with a relatively small investment of time and budget. To be sure, any smaller scale, less fully developed analysis may reach conclusions that are less exact and certain. But if they were constructed with full transparency, they might be used as a reasonable basis of assessment of research results, and would still represent a valuable addition to current measures of impact. Part of the economy achieved by this approach would be due to the fact that, even with RAND’s expertise and extensive efforts, a lot of the assembly of data and preliminary search for potential estimation strategies had to be carried out by NIOSH personnel.
While the RAND case studies did not attempt to estimate the total cost of the prevention measures that lead to the measured benefits, it would be desirable to pursue this in the future, to assemble complete, cost and benefit assessments. One means of obtaining cost information would be to build into NIOSH research projects, where appropriate, an objective of collecting information from research subjects and partners on their expenditures of time, resources, and funds to implement recommended prevention measures. This information might be collected in the course of collecting other information, and in the context of working relationships established for the research process. Assistance from personnel with specialized experience in cost analysis may be needed. While the information collected would not necessarily be representative of all workplaces for relevant occupations and industries, it would be likely to yield information on the general scale of costs per workplace and protected worker, so that a range of values for national costs could be constructed.
5.3 |. Barriers to feasibility of economic benefit studies of NIOSH activities
NIOSH engages in a wide variety of activities, many of which present much greater challenges to estimating economic benefits than did the RAND case studies. Examples include many epidemiological studies, occupational health surveillance, training for workers and managers, prevention recommendations, and laboratory and field methods for measuring exposure.
Most epidemiological studies do not immediately lead to an industrywide campaign to implement a new set of prevention recommendations as they did in the firefighter cancer case study. More typically, epidemiological findings tend to accumulate over time and gradually lead to actions in workplaces, many of which are voluntary and unrecorded in publicly available data.
Much the same can be said about surveillance information, which may point in a general way to industries and occupations or other broad groups of workers with elevated rates of certain injuries or illnesses. Such data do not often represent a wholly new understanding of risks and needed prevention measures, or the only impetus for progress in prevention.
Training for workers and managers on how to identify certain risks and mitigate them is also difficult to evaluate. Typically, post-training surveys are used to obtain participant reactions to the training and retention of its main messages. But it is difficult to measure training’s impact on subsequent prevention practices and results, especially over the long term. The outcomes of training can be enormously varied, and it is difficult to isolate the impact of a specific training on subsequent actions when there are other sources of information and safety initiatives that are also important.
A major responsibility of NIOSH is to provide prevention recommendations that are made freely available for voluntary use. But it is often difficult or impossible to know who is accessing and acting on those recommendations. The number of downloads and requests can often be recorded, but there is usually no practical system for determining their use or even gathering reports about their usefulness.
Developing and disseminating standard methods of exposure measurement is one of the infrastructural science responsibilities of NIOSH, but tracing out their influence on subsequent investigations and research, on subsequent prevention recommendations and on workplace prevention appears daunting. It may be better to regard such activities as overhead costs of other activities that are closer to practical application. In other words, rather than attempting to isolate their benefits, perhaps it should be acknowledged that their benefits will be part of the benefits attributed to other applied research efforts that use the measurement standards, and that the cost of developing the measurement standards should therefore be added to the total cost of those other efforts.
So what may be done in situations where economic measurement of benefits appears daunting? One response is to say that “partial assessment” of benefits can be useful, even if it cannot deliver bottom-line benefit estimates. This might particularly apply to prevention recommendations. As the RAND case studies demonstrate, the greatest benefits are achieved when the population addressed is large, when the baseline risk is high, and when the relevant prevention measures are effective. If these parameters are available, and if the lower level of risk with the recommended prevention measures in place can be estimated, then the potential number of illnesses or injuries averted can be estimated. Such estimates, along with attention to available information on the proportion of relevant employers that may have been reached through dissemination efforts, could help to measure relative potential for impact among NIOSH efforts devoted to providing guidance. Measurement of risk levels subsequent to dissemination of NIOSH research results could also help retrospectively to define the upper limit of the value of safety and health improvements for which NIOSH may have been responsible.
When safety and health improvements are observed following dissemination of NIOSH information, it is usually difficult to determine the degree to which the information was responsible. NIOSH has conducted surveys to determine the overall frequency with which professional, trade, and labor organizations rely upon and further disseminate information from NIOSH.51,52 In contrast, instead of focusing on rates of dissemination and use of information, a case study approach could focus on a specific set of workplace safety and health improvements for an industry and work backward to gain insight into how those improvements were made by interviewing a sample of the employers responsible for them. Researchers could attempt to determine the motivating factors behind the specific safety and health improvements and then to determine the sources of information that may have been critical for supporting the improvements. These critical information sources could then be examined to assess the extent to which they may have been directly or indirectly based on NIOSH research. This could require some tracing of information as it gets picked up and repackaged by disseminating parties in government, industry, and worker organization networks.
6 |. IMPLICATIONS FOR STRATEGIC AND PROJECT PLANNING
The RAND case study results are a validation of the basic approach NIOSH uses to guide and select research proposals for funding within and outside NIOSH: the Burden, Need, and Impact (BNI) framework.53 In this framework, project proposals are assessed on (1) magnitude of burden, as indicated by the size of the worker population they address, and the rate and severity of outcomes associated with the hazards they address, (2) the need for the particular research approach and resulting information as indicated by the demand of stakeholders for the information to be produced and their willingness to support the research, as well as by identified gaps in the safety and health literature, and (3) the potential for impact in terms of the effectiveness of relevant prevention measures and the prospect for wide dissemination and adoption of those measures. The factors that lead to success and relatively large benefits in the RAND case studies are well represented in these BNI categories.
If the RAND case studies represent some of NIOSH’s most successful research efforts, then they also point to the effectiveness of not only having stakeholder support but having stakeholders in the industry who are both research collaborators and direct users of the resulting information. And they highlight the value of stakeholders who can enforce or have an unusual capacity for promoting industrywide use of the results. This provides some insight into the factors that contribute to successful research translation, which is the focus of a recently established program at NIOSH. They also illustrate that the research knowledge being translated into prevention does not always consist of new prevention measures or guidance. It may also be new knowledge of risk levels as in the firefighter cancer case study, or about the effectiveness of already existing prevention measures as in the safety grants case study, or about the location of known risk types as in the amputation surveillance case study.
Another perspective on strategic and project planning can be gained by noting that some of the basic parameters of BNI were not necessarily available when the research that was the subject of the RAND case studies was begun. For example, in the asphalt milling case studies, there was little knowledge of the size of the population addressed by the research. RAND had to estimate it and find relevant information within the cost–benefit analyses done for OSHA. In the firefighter cancer case study, it was unknown whether any excess risk of cancer would be found, and it if had not been, it could not have led to the major results observed. In the ambulance redesign case study, there was no quantitative knowledge of the existing burden of injuries, though there were reasons to believe it was significant. In the safety grants and amputation surveillance case studies, the effectiveness of the resulting prevention measures was unknown until quantified by the NIOSH research that was the subject of the case studies. This points up the fact that research often needs to be undertaken without basic knowledge that allows prediction of its effects. However, a quantitative range of potential benefits might still have been possible to develop and might have strengthened the original research proposals.
7 |. CONCLUSION
We noted that there have been few attempts to date to measure the societal benefits of research with economic metrics, and none that require the same combination of methods as the NIOSH-sponsored RAND case studies. The NIOSH case studies required linking of specific research efforts to specific actions by others (employers), quantitatively measuring the extent of those actions and their impact on specific health outcomes, assessing the extent to which those actions might have occurred without research, and assigning dollar values to specific health outcomes. The experience gained through this combination of challenges may help point the way forward to realizing the widely shared goal of measuring research benefits, particularly health-related research supported by the public sector. The feasibility of quantitatively measuring benefits of NIOSH research with economic metrics has been demonstrated in several instances in which the impact of NIOSH research was already recognized in a qualitative sense and data availability was particularly favorable. Some additional NIOSH research efforts of this type can be identified, and there are ways that application of the same methods might be extended to other NIOSH research efforts that appear more difficult to assess. Such applications could be useful for making benefits more concrete to NIOSH’s funders and stakeholders, and to workers, employers, and its own staff, and for helping to assess the value of continuing investment in NIOSH research. The experience of these six case studies provides a set of general methods to follow but demonstrates that additional applications are likely also to require creativity in seeking out new data sources and expert informants, as well as means of analysis that overcome data limitations. In addition, we should recognize that a number of NIOSH responsibilities, such as providing prevention guidance, training occupational safety and health specialists, conducting surveillance of injury and illness patterns, producing epidemiological evidence for occupational risks, and establishing exposure measurement standards will often lead to benefits that are quite difficult to measure. For these critical activities, only partial steps toward quantification of benefits might still yield ranges of potential impact magnitudes and assist in judging results and setting priorities. Further, attempting quantification of benefits can help to document how impacts were achieved, even when it does not reach a bottom-line benefit estimate. The difficulties of applying these methods to situations in which there is no available mechanism to track, compel, or strongly incentivize prevention implementation also suggests that some effort might be directed to focused investigation of the sources of information that have played major roles in instances of notable improvements in safety and health and the part that NIOSH research may have played, either directly or indirectly.
ACKNOWLEDGMENT
The authors report that there was no funding source for the work that resulted in the article or the preparation of the article.
Footnotes
CONFLICTS OF INTEREST
The authors declare no conflicts of interest.
DISCLOSURE BY AJIM EDITOR OF RECORD
John Meyer declares that he has no conflict of interest in the review and publication decision regarding this article.
ETHICS APPROVAL AND INFORMED CONSENT
No ethics review nor informed consent was required as no human subjects research was involved.
DISCLAIMER
The findings and conclusions in this report of the authors do not necessarily represent the views of the National Institute for Occupational Safety and Health, the Centers for Disease Control and Prevention, or the US Department of Health and Human Services.
DATA AVAILABILITY STATEMENT
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
REFERENCES
- 1.Centers for Disease Control and Prevention (CDC), National Institute for Occupational Safety and Health (NIOSH). National Academies evaluation of NIOSH programs. Accessed August 17, 2021. https://www.cdc.gov/niosh/nas/default.html
- 2.Downes A, Novicki E, Howard J. Using the contribution analysis approach to evaluate science impact: a case study of NIOSH. Am J Eval. 2019;40(2):177–189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Van Eerd D, Moser C, Saunders R. A research impact model for work and health. Am J Ind Med. 2021;64(1):3–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Institute for Work and Health (IWH). Impact case studies. Accessed August 17, 2021. https://www.iwh.on.ca/impact-case-studies
- 5.National Research Council. Furthering America’s Research Enterprise. The National Academies Press; 2014. doi: 10.17226/18804 [DOI] [PubMed] [Google Scholar]
- 6.National Institutes of Health (NIH), National Institute for Allergy and Infectious Diseases (NIAID). Program evaluation at NIAID. Accessed August 17, 2021. https://www.niaid.nih.gov/about/evaluation
- 7.National Institutes of Health (NIH), National Institute of Neurological Disorders and Stroke (NINDS). Program evaluation. Accessed August 17, 2021. https://www.ninds.nih.gov/About-NINDS/Strategic-Plans-Evaluations/Program-Evaluations
- 8.National Institutes of Health (NIH), National Institute of General Medical Sciences (NIGMS). Evaluation reports. Accessed August 17, 2021. https://www.nigms.nih.gov/about/dima/Pages/reports.aspx
- 9.National Institutes of Health (NIH), Office of Evaluation, Performance, and Reporting (OERP). Impact analysis. Accessed August 17, 2021. https://dpcpsi.nih.gov/oepr/impact-analysis
- 10.National Institutes of Health (NIH). Report on approaches to assess the value of biomedical research supported by NIH. Scientific Management Review Board (SMRB). Accessed August 17, 2021. https://smrb.od.nih.gov/documents/reports/VOBR%20SMRB__Report_2014.pdf [Google Scholar]
- 11.Office of Management and Budget (OMB). Memorandum for Heads of Executive Departments and Agencies. Subject: phase 1 implementation of the foundations for evidence-based policymaking act of 2018: learning agendas, personnel, and planning guidance. July 2019. Accessed August 17, 2021. www.whitehouse.gov/wp-content/uploads/2019/07/M-19-23.pdf
- 12.Murphy K, Topel R. The economic value of medical research. In: Murphy K, Topel R, eds. Measuring the Gains from Medical Research. University of Chicago Press; 2003:41–73. [Google Scholar]
- 13.Cutler D, Kadiyala S. The return to biomedical research: treatment and behavioral effects. In: Murphy K, Topel R, eds. Measuring the Gains from Medical Research. University of Chicago Press; 2003:110–162. [Google Scholar]
- 14.Hall BH, Mairesse J, Mohnen P. Measuring the returns to R&D. In: Hall BH, Rosenberg N, eds. Handbook of the Economics of Innovation. Elsevier; 2010:1034–1076. [Google Scholar]
- 15.Hall MJ, Layson SK, Link AN. The returns to R&D: Division of Policy Research and Analysis at the National Science Foundation. Sci Public Policy. 2014;41(4):458–463. [Google Scholar]
- 16.Wang SL, Heisey P, Schimmelpfennig D, Ball E. Agricultural productivity growth in the United States: measurement, trends, and drivers, ERR-189. U.S. Department of Agriculture, Economic Research Service. July 2015. Accessed August 17, 2021. https://www.ers.usda.gov/publications/pub-details/?pubid=45390 [Google Scholar]
- 17.Groenewold M, Brown L, Smith E, Sweeney MH, Pana-Cryan R, Schnorr T. Burden of occupational morbidity from selected causes in the United States overall and by NORA industry sector, 2012: a conservative estimate. Am J Ind Med. 2019;62(12):1117–1134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.U.S. Department of Energy (DOE). Aggregate economic return on investment in the U.S. DOE Office of Energy Efficiency and Renewable Energy. October 2016. Accessed August 17, 2021. https://www.energy.gov/eere/analysis/downloads/aggregate-economic-return-investment-us-doe-office-energy-efficiency-and
- 19.Galleher M, Scott T, Oliver Z, Clark-Sutton K, Anderson B. Benefit-cost evaluation of U.S. Department of Energy Investment in HVAC, Water Heating, and Appliance Technologies. Final Report prepared for the Building Technologies Office, Office of Energy Efficiency and Renewable Energy September. U.S. Department of Energy by RTI International, with contribution from Rosalie Ruegg (TIA Consultants). September 2017. Accessed August 17, 2021. https://www.energy.gov/sites/prod/files/2017/09/f36/DOE-EERE-BTO-HVAC_Water%20Heating_Appliances%202017%20Impact%20Evaluation%20Final.pdf [Google Scholar]
- 20.Link A, Scott J. The theory and practice of public-sector R&D economic impact analysis. Planning Report 11-1 prepared for the National Institute of Standards and Technology, U.S. Department of Commerce. January 2012. Accessed August 17, 2021. https://www.nist.gov/system/files/documents/2017/04/28/report11-1.pdf [Google Scholar]
- 21.RTI International. Retrospective economic impact assessment of the NIST combinatorial methods center. Planning Report 09-1 prepared for the National Institute of Standards and Technology, U.S. Department. of Commerce. April 2009. Accessed August 17, 2021. https://www.nist.gov/system/files/documents/2017/05/09/report09-1.pdf [Google Scholar]
- 22.Haddix A, Teutsch S, Corso P. Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation. 2nd ed. Oxford University Press; 2003. [Google Scholar]
- 23.Centers for Disease Control and Prevention (CDC). Assessing the effectiveness of disease and injury programs: costs and consequences. MMWR Morb Mortal Wkly Rep. 1995;44(o. RR-10): 1–10. https://www.cdc.gov/mmwr/preview/mmwrhtml/00038592.htm7799912 [Google Scholar]
- 24.Centers for Disease Control and Prevention (CDC), Steven M. Teutsch Prevention Effectiveness (PE) Fellowship Program, Key publications from PE fellows. Accessed August 17, 2021. https://www.cdc.gov/pef/accomplishments/key-publications.html
- 25.Yellman MA, Peterson C, McCoy MA, et al. Preventing deaths and injuries from house fires: a cost-benefit analysis of a community-based smoke alarm installation programme. Inj Prev. 2018;24(1): 12–18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Chesson HW, Ludovic JA, Berruti AA, Gift TL. Methods for sexually transmitted disease prevention programs to estimate the health and medical cost impact of changes in their budget. Sex Transm Dis. 2018;45(1):2–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Overwyk KJ, Dehmer SP, Roy K, et al. Modeling the health and budgetary impacts of a team-based hypertension care intervention that includes pharmacists. Med Care. 2019;57(11):882–889. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Leidner AJ, Murthy N, Chesson HW, et al. Cost-effectiveness of adult vaccinations: a systematic review. Vaccine. 2019;37(2): 226–234. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Rice D, Hodgson T, Kopstein A. The economic costs of illness: a replication and update. Health Care Financ R. 1985;7(1):61–80. [PMC free article] [PubMed] [Google Scholar]
- 30.Leigh JP. Economic burden of occupational injury and illness in the United States. Milbank Q. 2011;89(4):728–772. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Miller B, Metz D, Smith T, Lastunen J, Landree E, Nelson C. Understanding the economic benefit associated with research and services at the National Institute for Occupational Safety and Health. Research report prepared for NIOSH. RAND Corporation. 2017. Accessed August 17, 2012. https://www.rand.org/pubs/research_reports/RR2256.html [PMC free article] [PubMed] [Google Scholar]
- 32.Miller B, Metz D, Smith T, Lastunen J. Selecting and evaluating case studies of the economic benefits of research and services at the National Institute for Occupational Safety and Health RAND Corporation. Research report prepared for NIOSH. 2020. Accessed August 17, 2021. https://www.rand.org/pubs/research_reports/RR4201.html [Google Scholar]
- 33.Mine Safety and Health Administration (MSHA). Lowering miners’ exposure to respirable coal mine dust, including continuous personal dust monitors. Fed Regist. 2014;79(84):24814–24994. https://www.govinfo.gov/content/pkg/FR-2014-05-01/pdf/2014-09084.pdf [Google Scholar]
- 34.Occupation Safety and Health Administration (OSHA). Final economic analysis and final regulatory flexibility analysis: supporting document for the final rule for occupational exposure to respirable crystalline silica in Occupational Safety and Health Administration, occupational exposure to respirable crystalline silica: final rule. OSHA-2010-0034-4247. 2016: 16399–16690. Accessed March 10, 2022. https://www.regulations.gov/document/OSHA-2010-0034-4247
- 35.Eastern Research Group (ERG). Rulemaking support for supplemental economic feasibility data for a preliminary economic impact analysis of a proposed crystalline silica standard; updated cost and impact analysis of the draft crystalline silica standard for construction. Task report. Submitted to Occupational Safety and Health Administration, Directorate of Evaluation and Analysis, Office of Regulatory Analysis under Task Order 11, Contract No. DOLJ049F10022. April 20, 2007. [Google Scholar]
- 36.Levine DI, Toffel MW, Johnson MS. Randomized government safety inspections reduce worker injuries with no detectable job loss. Science. 2012;336(6083):907–911. [DOI] [PubMed] [Google Scholar]
- 37.Tompa E, Kalcevich C, Foley M, et al. A systematic literature review of the effectiveness of occupational health and safety regulatory enforcement. Am J Ind Med. 2016;59(11):919–933. [DOI] [PubMed] [Google Scholar]
- 38.Li L, Singleton P. The effect of workplace inspections on worker safety. ILR Review. 2019;72(3):718–748. [Google Scholar]
- 39.Viscusi WK, Aldy JE. The value of a statistical life: a critical review of market estimates throughout the world. J Risk Uncertainty. 2003; 27(1):5–76. [Google Scholar]
- 40.National Safety Council (NSC). Injury facts, workers’ compensation costs. Accessed August 17, 2021. https://injuryfacts.nsc.org/work/costs/workers-compensation-costs/
- 41.Environmental Protection Agency (EPA). Regulatory impact analysis for the federal implementation plans to reduce interstate transport of fine particulate matter and ozone in 27 states. Correction of SIP approvals for 22 states. June 2011. Accessed August 17, 2021. https://www3.epa.gov/ttn/ecas/docs/ria/transport_ria_final-csapr_2011-06.pdf [Google Scholar]
- 42.Viscusi WK, Magat WA, Huber J. Pricing environmental health risks: survey assessments of risk-risk and risk-dollar trade-offs for chronic bronchitis. J Environ Econ Manag. 1991;21(1):32–51. [Google Scholar]
- 43.Magat WA, Viscusi WK, Huber J. A reference lottery metric for valuing health. Manage Sci. 1996;42(8):1118–1129. [Google Scholar]
- 44.National Safety Council (NSC). Injury facts, guide to calculating costs 2017. Assessed August 17, 2021. https://injuryfacts.nsc.org/all-injuries/costs/guide-to-calculating-costs/data-details/
- 45.Grosse SD, Krueger KV, Mvundura M. Economic productivity by age and sex: 2007 estimates for the United States. Med Care. 2009; 47(7):S94–S103. [DOI] [PubMed] [Google Scholar]
- 46.Seabury SA, Scherer E, O’Leary P, Ozonoff A, Boden LI. Using linked federal and state data to study the adequacy of workers’ compensation benefits. Am J Ind Med. 2014;57(10):1165–1173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.National League of Cities (NLC). Assessing state firefighter cancer presumptions laws and current firefighter cancer research. 2009. Assessed August 17, 2021. http://www.effua.org/wp-content/uploads/2017/09/PresumptionReport2009ASSESSING-STATE-FIREFIGHTER-CANCER-PRESUMPTION-LAWS-AND-CURRENT-FIREFIGHTER-CANCER-RESEARCH.pdf
- 48.Haas E, Colinet J. Miners implement corrective actions in response to CPDM dust data. Coal Age. 2018;123(2):36–38. [Google Scholar]
- 49.Department of Health and Human Services (DHHS), Office of the Assistant Secretary for Planning and Evaluation. Guidelines for regulatory impact analysis: a primer. January 12, 2017. Accessed August 17, 2021. https://aspe.hhs.gov/reports/guidelines-regulatory-impact-analysis
- 50.National Cancer Institute (NCI). Annualized mean net costs of care: annualized mean net costs of care by age, gender and phase of care (per patient)—costs in 2010 US dollars. Accessed August 17, 2021. https://costprojections.cancer.gov/annual.costs.html
- 51.Anderson VP, Schulte PA. Customer Satisfaction Survey. NIOSH Publications and Information Services. U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, DHHS (NIOSH) Publication; 2013:1–48. [Google Scholar]
- 52.Okun AH, Watkins JP, Schulte PA. Trade associations and labor organizations as intermediaries for disseminating workplace safety and health information. Am J Ind Med. 2017;60(9): 766–775. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Felknor S, Schulte P, Schnorr T, Pana-Cryan R, Howard J. Burden, need, and impact: an evidence-based method to identify worker safety and health research priorities. Ann Work Expo Health. 2019; 63(4):375–385. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data sharing is not applicable to this article as no new data were created or analyzed in this study.