RESPONSE TO WEINSTOCK AND SLATIN’S (2012) CRITIQUE OF IWH-NIOSH SYSTEMATIC REVIEW OF THE EFFECTIVENESS OF OSH TRAINING

Abstract

This commentary responds to the recent critique by Weinstock and Slatin [1] of our systematic review on the effectiveness of occupational safety and health (OSH) training conducted jointly by the Institute for Work & Health (IWH) in Canada and the National Institute for Occupational Safety and Health (NIOSH) in the United States [2, 3]. We address misunderstandings of our perspectives on training, especially empowerment training, and the evaluation of such training.

Weinstock and Slatin [1] recently critiqued a systematic review on the effectiveness of occupational safety and health (OSH) training conducted jointly by the Institute for Work & Health (IWH) in Canada and the National Institute for Occupational Safety and Health (NIOSH) in the United States [2, 3]. We thank them for drawing attention to the review, which is a synthesis of high-quality evidence about the effectiveness of OSH training. We write to address what we believe are misunderstandings of our perspectives on training, especially empowerment training, and the evaluation of such training.

First, NIOSH has consistently identified worker empowerment as critical to occupational disease and injury prevention and continues to do so. It follows that OSH training enhancing the knowledge and skills workers need to make changes in their environment is critically important. Accordingly, the introduction to our review described the scope of OSH training as including training that aims to “empower workers and managers to become more active in making changes that enhance worksite protection” [3, p. 193].

Weinstock and Slatin were concerned that the review excluded empowerment training. Indeed, no training interventions included in the review had empowerment as their main focus. This resulted from a decision made at the outset of the review to focus on studies with high internal validity. With this in mind, the criteria used to decide whether a particular study should be included in the review stipulated, among other things, that it be a randomized controlled trial (RCT) and have measured a training-related outcome (e.g., knowledge, health) both before and after the training. This focus on internal validity was adopted because earlier literature reviews, including that of Cohen et al. [4], were hampered in their ability to draw conclusions about health-related outcomes. For example, Cohen et al. concluded that “reductions in work injuries and medical costs may also be noted in conjunction with these changes [knowledge gain and behavior change] but evidence to show actual linkage or dependency remains to be ascertained” [4]. A well conducted RCT can minimize such uncertainty in attribution. Unfortunately, when the review team opted for high internal validity, there was a trade-off with external validity: the full breadth of OSH training was not included and the number of studies was limited. We did not intend to exclude particular classes of training, but some, like empowerment training, ended up being omitted simply because they were not studied using a research design that met the inclusion criteria. Our hope is that the review will encourage a new generation of training studies, including empowerment training studies, using pre- and post-measurement and a field-based RCT design.

A second concern of Weinstock and Slatin was the choice of outcomes included in the review to assess training effectiveness. They disputed the inclusion of health outcomes, disliked the focus on behavioral outcomes, and advocated for the inclusion of hazard reduction as an outcome. We assert that health outcomes are appropriate for inclusion, as per well accepted workplace training evaluation frameworks such as Kirkpatrick’s [5], which include outcomes substantially distal to (i.e., downstream of) the immediate training impacts. We termed health outcomes as “ultimate” goals in the sense of being more distal to training. After all, we don’t just remove hazards as an end in itself; we remove them because they would otherwise result in adverse health outcomes. That said, it does not follow that training is the only thing needed in order for a workplace to minimize work-related injury and illness.

We turn now to the issue of behaviors versus hazards as an outcome category. The first thing to note is that hazards were not excluded from the review. The category of outcomes labeled as “behaviors” was in fact described more precisely in the methods as “behaviors, behavior-dependent hazards, behavior-dependent exposures” [3, p. 195]. There was actually some representation of hazards in the review: for example, the effects of office ergonomic training on workstation layout (ergonomic hazard). More importantly, if an empowerment training study had been selected into the review, we would have extracted for the review the training effects based on the empowerment behaviors targeted by the training (e.g., reporting, advocating) or hazards/exposures changed as a result of worker action if they had been measured by its authors. What data was actually extracted from primary studies and synthesized in the review was determined by the nature of the studies meeting the criteria for inclusion. This then brings us back to the nature of the review.

Any single research study has its limitations, and our systematic review was no exception. The type of limitation encountered in our review has been found in other systematic reviews. Some researchers have therefore grappled with the internal-versus-external validity dilemma (as well as the issue of complexity) by conducting realist reviews [6]. This approach typically allows more flexibility in the type of study designs and nature of evidence included. Realist review methods are still maturing, with questions of best practice still under discussion [7], but we applaud the direction. We also applaud any efforts to demonstrate— preferably unambiguously—the value of OSH training, including empowerment training.

Today, we are in an era in which society requires empirical evidence about the value of interventions. Occupational safety and health training is one such intervention, and it is part of the $100-billion-a-year investment in building skill competencies and awareness in workers in the United States. Therefore, it is appropriate to provide a review of the effectiveness of OSH training to determine its value and identify where it can be improved. The current standard for synthesizing study results and evaluating evidence is the systematic review, which prioritizes the best evidence and uses dispassionate analysis. Such reviews do not always bring results that stakeholder communities want or expect.

Our review found that the available body of RCT evidence was not consistent with the conclusion that OSH training (as a lone intervention) has a positive effect on health outcomes: observed effects were too small and were inconsistent in direction. We and others have been concerned that this finding could be misused by those seeking to justify reducing or eliminating OSH training for reasons unrelated to training effectiveness. We counter this stance by pointing to one of the review’s other findings: there was strong evidence for training’s impact on “behaviors,” which leads to a recommendation for training to be continued. Moreover, health and safety advocates can use the results of the review to their advantage when faced with the all-too-common situation where the solution to a hazardous situation is proposed to be simply training. As stated in the closing remarks of our review, “we strongly suggest that decision-makers consider more than just education and training when addressing a risk in the workplace. Large impacts of training alone cannot be expected, based on research evidence” [3, p. 205]. It follows that other actions must also be considered, foremost among them the elimination or reduction of hazards. Usually this requires the involvement of others besides front-line workers who receive training. After all, it is ultimately the responsibility of employers to provide safe and healthy workplaces.

Biographies

LYNDA ROBSON, Ph.D., is an Associate Scientist at the Institute for Work & Health (IWH). Her activities often involve the application of program evaluation methods to occupational health and safety (OHS). These include leading authorship of the IWH-NIOSH Guide to Evaluating the Effectiveness of Strategies for Preventing Work Injuries and conducting systematic reviews of the effectiveness of OHS interventions. Areas of OHS research include management systems and audits, organizational performance measurement, and organizational change. She is also member of the Institute’s joint health and safety committee, working toward continuous improvement in her own organization. Her email address is lrobson@iwh.on.ca.

PAUL A. SCHULTE, Ph.D. is the Director of the Education and Information Division, and Manager of the Nanotechnology Research Center and the Prevention through Design programs at NIOSH. Dr. Schulte is an epidemiologist with experience investigating occupational cancer, genetics, and risk communication. Write to him at pas4@cdc.gov.

CAROL M. STEPHENSON, Ph.D. is the Chief of the Training/Translational Research and Evaluation Branch in the Education and Information Division of the National Institute for Occupational Safety and Health. Her research focuses on optimizing health communication and knowledge transfer, diffusion of innovations, and workplace intervention effectiveness— particularly training impact and transfer. She currently directs a program of research investigating ways to instill foundational OSH skills in new and young workers, health disparities among immigrant workers, health and safety motivation for small business employers, and targeted occupational safety and health issues (such as noise and hearing loss) in the construction, manufacturing, and service sectors. Send email to cem3@cdc.gov.

EMMA L. IRVIN is the director of Research Operations for the Institute. As a member of the executive team, she oversees research planning and operational aspects of the Institute’s business. She also oversees the internal library, systems, and a key foundational program area, systematic reviews. Currently, Irvin is working with executive and research staff on the ongoing development of the strategic research plan and a knowledge management strategy that will guide the Institute’s operations over the coming years. This plan includes seeing the systematic review program through the next five-year review; continuing to develop research methods for conducting systematic reviews and maintaining the high quality of IWH’s research operations. Write her at eirvin@iwh.on.ca.

BENJAMIN C. AMICK III is Associate Scientific Director of the Institute for Work & Health and Professor of Behavioral Sciences and Epidemiology, University of Texas School of Public Health. His research focuses on the development and evaluation of organizational indicators of occupational health and safety performance, organizational and ergonomic interventions to improve health and productivity, and evaluation of occupational health and safety programs. Currently, he is conducting several large scale surveys to determine the most appropriate leading indicators of occupational safety and health performance in Ontario, the evaluation of high engagement office ergonomics training, the evaluation of a participatory change program in long-term care and the role of unions in affecting health and safety outcomes in construction. He is involved in several systematic reviews on worksite interventions to support return-to-work and prevent MSDs. His address is bamick@iwh.on.ca.