Research Methodologies for Total Worker Health^®

Abstract

Objective:

There is growing interest in the NIOSH Total Worker Health^® program, specifically in the process of designing and implementing safer, health-promoting work and workplaces. A Total Worker Health (TWH) Research Methodology Workshop was convened to discuss research methods and future needs.

Methods:

Twenty-six experts in occupational safety and health and related fields reviewed and discussed current methodological and measurement issues and those showing promise.

Results:

TWH intervention studies face the same challenges as other workplace intervention studies and some unique ones. Examples are provided of different approaches and their applications to TWH intervention studies, and desired developments in the TWH literature.

Conclusions:

This report discusses and outlines principles important to building the TWH intervention research base. Rigorous, valid methodologic, and measurement approaches are needed for TWH intervention as well as for basic/etiologic, translational, and surveillance research.

BACKGROUND

Defining Characteristics of Total Worker Health^®

The National Institute for Occupational Safety and Health (NIOSH) defines Total Worker Health (TWH) as policies, programs, and practices that integrate protection from work-related safety and health hazards with promotion of injury and illness prevention efforts to advance worker well-being. The paradigm expands upon the previous definition that emphasized integration of health protection and health promotion¹ to encourage integration across a wider set of workplace efforts that support safety, health, and well-being. Integration can occur through collaboration and coordinated programming around organizational leadership and commitment; supportive organizational policies and practices; accountability and training; management and employee engagement strategies; supportive benefits and incentives; and integrated real-time evaluation and surveillance that leads to corrective action where needed.^1,2 Several frameworks and models—such as those developed by Sorensen et al, Pronk, and Punnett et al—have been proposed to help explain what integration looks like in practice.^3–7 Differences among these models reflect the variety of ideas emerging about how to implement TWH approaches and the evolving nature of this relatively new field.

The TWH definition explicitly reflects the priority placed on changes to improve physical, organizational, and psychosocial factors in the work environment. This approach is distinctly different from interventions that only address or place sole responsibility on the worker to manage the risks. In other words, a workplace intervention must consider how to redesign the work environment to reduce the risk (rather than only providing personal protective equipment for the worker) and address how the work itself may be stress-inducing (rather than instituting a wellness program that only promotes stress management). Programs with a primary motivation of cost savings, rather than improved worker safety, health, and well-being, also do not fit into the TWH framework.

Although the TWH research base has grown,⁸ it is still an emerging field that can benefit from further examination of how current and new research methods can be used to solidify the evidence base and advance the field. A literature review of TWH intervention research identified only 17 articles on the evaluation of occupational safety and/or health plus wellness or well-being outcomes.⁸ Promisingly, most of these studies showed an improvement and demonstrated that the gain achieved was greater than when either the safety and/or health or wellness intervention was evaluated alone; this has also been noted in previous literature.^9,10 The review further highlighted current gaps in the TWH field, including a need to focus on methodological and measurement issues.

The TWH Research Methodology Workshop

On March 7 and 8, 2017, the University of Iowa's College of Public Health and Healthier Workforce Center of the Midwest hosted a TWH Research Methodology Workshop (“Workshop”). Planning and implementation support for the Workshop was provided by an executive planning committee consisting of several TWH experts and NIOSH personnel. The invited participants were leading researchers in the field of TWH and other pertinent areas.

The impetus for the Workshop was two-fold. The first was to respond to two of the eight recommendations put forth by the Independent Panel of the Pathways to Prevention 2015 Meeting (Total Worker Health^®: What's Work Got to Do With It?): (1) expand research and evaluation design options to include a range of rigorous methodologies and (2) develop a core set of measures and outcomes that are incorporated into all integrated intervention studies.¹¹ The second was to respond to one of the research intermediate goals: “to apply and develop rigorous, standardized methods for studying TWH interventions,” supported by several activity/output goals (Sections 1.3 and 1.3.2–1.3.6 in reference [12])¹² in NIOSH's National Total Worker Health^® Agenda (National TWH Agenda).

Although the TWH field can and should be advanced by means of other research methods (basic/etiologic and surveillance), the Workshop, and thus this paper, concentrate primarily on intervention and (to a lesser extent) translational research.^13,14 This is because intervention research, in particular, accounts for the majority of the TWH literature to date and was emphasized in the recommendations and goals by the Independent Panel of the Pathways to Prevention 2015 Meeting¹¹ and the National TWH Agenda.¹²

The Workshop planners selected a few TWH-related research methodological and measurement focus areas that were then reviewed by participants who discussed limitations to currently-used approaches and described others that show promise for advancing the TWH scientific evidence base. This report summarizes both the 2-day Workshop and subsequent working group discussions. It is intended to highlight the TWH methodological and measurement approaches currently in use, describe the range of possibilities, and suggest promising research practices with the potential to advance the field, through rigorous and repeatable TWH intervention research. The authors also refer to approaches from other research fields that may be applicable to the TWH field. Though some of the issues raised and points for consideration may not be new to occupational safety and health or wellness research, the Workshop participants agreed that it was useful to highlight these in order to strengthen and advance this rather new field's knowledge base. The Workshop topics are as follows:

• Study design considerations for measuring the effectiveness of TWH programs.

• Potential selection bias, attrition, and unanticipated changes to the work environment that could affect TWH research.

• Qualitative and mixed-methods approaches in TWH research.

• Case study design in TWH research.

• Measures and outcomes relevant to TWH research.

The expectation is that the summary of these Workshop topic discussions will be relevant not only to seasoned and new TWH and occupational safety and health researchers, but also to others in related disciplines, and to practitioners and stakeholders who read and interpret the TWH literature.

DISCUSSION

Study Design Considerations for Measuring the Effectiveness of TWH Programs

General Considerations for TWH Programs

The range of TWH research approaches allows for a research design that best fits the research question. Social epidemiological research is important for understanding how working conditions shape safety and health outcomes, whereas dissemination and implementation research helps reveal what and how such research is being translated and delivered in practice. Surveillance research can provide additional information and data on current TWH efforts, to further inform other research processes and practice.

Regardless of research study design type, the measurement and evaluation of interventions can pose several challenges. For instance, TWH interventions are, by definition, multi-factorial; therefore, it is difficult to isolate any given element and compare it to another element found at an organization or worksite not implementing TWH programs. In addition, workers and employers may view data collection as intrusive, burdensome, and/or not central to their work requirements. This difficulty is further complicated when there is a need to collect data from control group workers who will not benefit immediately from the intervention being evaluated; this also touches on the interrelationship between methodological possibilities or decisions made and Institutional Review Board considerations.

Workplace intervention implementation is largely controlled by the employer, in terms of content, duration, and intensity and manipulation of specific variables. Organizational changes, including those in leadership and the work environment, can introduce significant challenges. Fidelity of the intervention and consistency across worksites may prove difficult to control, especially when a TWH program is tailored to a specific workplace and delivered for different exposure periods. Furthermore, a long-time horizon may be needed to realize changes from TWH interventions. This is true especially when a change in safety or health culture is the foundation for other program features and when the outcomes of interest are relatively rare (such as work accidents, incidents of disease, or changes in health habits) and/or involve long latency from exposure to onset.

Another consideration is that potentially sensitive financial and personnel data may be needed to achieve buy-in for investing in organizational redesigns to improve worker safety, health, and well-being.¹⁵ The business case may address the cost consequences of not addressing the problem; the resources needed to support a TWH solution, for both the short and long term; the cost-effectiveness of interventions, or value-on-investment (VOI); and the potential cost-benefit of a TWH approach, and its return-on-investment (ROI). An organization may be reluctant to provide such information to investigators, even though employers themselves typically ask for such results before committing themselves to participate. Also, upon implementation, workers may not wish to provide information to employers, a possibility that should be considered in assessing the practicality of such an interaction to be useful.

Study Design Options

Studies for measuring TWH program impacts are of three basic designs, each of which has its strengths and limitations. The first is pre-experimental, relying primarily on post-hoc surveys, pre/post measurements, case studies, or time series studies where interventions are modulated over time. These measurements are applied to only the group receiving the TWH intervention; there is no comparison or control group. In these pre-experimental studies, the employee and sometimes the business serve as their own control, when there is a lag period of observation before implementing the intervention.

The second is a quasi-experimental design, meaning an observational study in which a comparison group is used to answer the question, “What would have happened were there no intervention?” Comparison groups can be drawn from within the organization—either as nonparticipant or unexposed populations—or from outside the organization in the form of another company or group of companies providing a normative database. Quasi-experimental studies may include evaluations of interventions initiated by employers, such as comparisons of change between intervention sites and comparison sites not randomly assigned to the intervention condition.¹⁶ Because of the nonrandom selection, sophisticated statistical techniques like multivariate analysis, propensity score matching/weighting, coarsened exact matching, instrumental variable analysis, and other adjustment methods are used to control for confounders.^17–19

The third option, a randomized controlled trial (RCT), is a true experimental design, which requires random assignment of workers to a TWH program or a control condition and then comparison of the experience of treatment versus control subjects. A useful variation is the cluster randomized trial which uses a multi-level design; usually there are two levels, the individual participant and the cluster (organization, department, or other group), where the cluster is the unit of randomization.²⁰ Although there can be challenges to conducting RCTs in workplaces, experimental designs for TWH research are encouraged when these conditions exist: organizational leaders allow it²¹; a long enough time horizon is available to record short- and long-term sustainable results; few inclusion or exclusion criteria are mandated, a circumstance which improves the generalizability of results; and there is minimal worry about spillover effects across sites, which would affect intervention fidelity—the degree to which the intervention is implemented as intended.

On the other hand, the RCT design was not intended for organization-level interventions and would be virtually impossible to apply, at the very least because so many organizations would be required that the research costs would be prohibitive. Further, an aversion by management to experimental study designs is understandable because, by definition, some interventions would be offered to a certain segment of workers while other workers would be denied access to potential safety and health improvements. However, this does not need to be the case if a delayed intervention control is used, such as in the stepped wedge design. For a discussion of ways to address common challenges to all workplace safety and health program evaluations, the authors refer readers to several publications that offer thoughtful consideration of the issues raised here.^19,22–26

Each of these designs for studying intervention effectiveness can be supplemented with in-depth examination of the intervention process using qualitative and/or quantitative data to document intervention feasibility and the reasons for success or failure, as well as long-term sustainability.²⁷

Selecting Comparison Groups in Non-Randomized Designs

A quasi-experimental design is generally preferred to pre-experimental because it introduces a reference group against which the experience of intervention subjects can be gauged. However, because participants (whether workers or workplaces) are not assigned randomly to the intervention and control conditions, selection bias may result. Therefore, quasi-experimental designs should draw comparison subjects from a group of individuals who are as alike as possible to the intervention subjects. Comparison subjects may be employees recruited from inside the organization or from other organizations that are similar in size, industry, region of the country, and workforce composition. In either case, treatment subjects can be matched or “twinned” with comparison subjects, or data from each group can be weighted so that, on average, the groups look the same on measurable factors. Matching or weighting variables may include socio-demographics, region, job type, industry, union/nonunion, hourly/salaried, health plan, plan generosity, medical costs, healthcare service use, adequacy and completeness of the program, and readiness to adopt safe and healthy habits. The variables chosen for matching or weighting would include forces likely to influence program participation, as well as the outcomes of interest aside from participation.

Quasi-experimental designs are sometimes achieved by leveraging large, integrated databases that facilitate detection of small differences among groups; however, those differences, while statistically significant, may not be meaningful in terms of program impact. That said, smaller datasets may prove equally problematic, as the magnitude of effect needed to show a positive result may then be less likely, given the intervention. This further demonstrates the need to consider effect size and the population sample size as part of the study design. Quasi-experimental designs may also be applied retrospectively to determine what occurred following a given TWH intervention and what the cumulative impacts might be. This approach may offer an advantage over prospective trials, which may require a long-time horizon for data collection and for the intervention to take root within an organization. Indeed, to document the benefits of a TWH program, years may be needed to fully realize the advantages of one approach over another. However, the authors recognize that this is not always feasible, and it may also introduce other challenges (such as attrition, a topic discussed later on). In some instances, assessing more intermediate outcomes may be an achievable compromise, as discussed in the measurement portion of this paper.

Participatory Approaches in TWH

A key distinguishing feature among intervention studies—whether in the workplace or in the community—is whether or not the individuals who are the subjects of the study actually participate in designing and carrying out the intervention and/or the evaluative data collection and analysis (ie, the research). Such methods were pioneered in community-based participatory research and have since been adopted by some workplace investigators.²⁸ For TWH programs, workers’ insights into the interactions between work and nonwork hazards are particularly valuable, because they can shape the selection of the intervention itself as well as how it is delivered and communicated. Participatory involvement is critical for all stakeholders and participants at all levels, including executives and senior leaders, middle management, labor representatives, and workers. A number of intervention research studies have relied on worker participation and emphasize that participation is vital at all stages of program design, development, and evaluation.^10,29

Prioritizing worker involvement is critical to ascertain the culture and climate of a worksite, including which type of intervention is most appropriate. Another continual part of the process is creating communication plans through a loop-back effect, whereby information and results are disseminated to labor representatives, workers, and employers.

Data Reporting

It is important to be transparent in describing methods and analysis and providing data summaries that are adequate to understand the study sample. This includes delineating the studied population in an attrition table and describing numerically how many individuals were dropped because they did not satisfy specified inclusion or exclusion criteria. Creating tables containing raw numbers is vital to the credibility of the study. Descriptive data tables should include the number of individuals studied (“n”) and values such as group means, medians, standard deviations, ranges, and outliers. Researchers should explain the steps taken to clean and normalize the data by providing cut-off values for nonvalid responses, explaining how negative values or missing data were handled, and describing any transformation techniques or other methods used to address skewed data. Finally, supplying formulas applied in multivariate analyses, along with the rationale for including or excluding key variables in the model, is key for meaningful interpretation as well as for replication and comparisons across studies.

Key Conclusions

There are many points to bear in mind when selecting study designs to measure the effectiveness of TWH programs. Each intervention is necessarily carried out in a rich natural environment and whose characteristics may be determinative of the outcome. Balancing feasibility and process insights with potential threats to validity and rigor is always a challenge, no less in TWH. Within the context of study design selection and planning, the determination of appropriate comparison groups and inclusion of involved stakeholders using participatory approaches are important considerations. Lastly, in order to understand multi-component interventions fully, transparent and detailed data-reporting processes are central.

Potential Selection Bias, Attrition, and Unanticipated Changes to the Work Environment That Could Affect TWH Research

Selection Bias

As in all observational research, sample type and size as well as participation rates may be influenced by selection bias, which may then impact program outcome estimates. These concerns are particularly relevant to TWH research, given that TWH interventions occur in the workplace rather than a laboratory. In addition, any intervention that leads to modification of the job and/or workplace might be more susceptible to subject refusal or withdrawal from participation, potentially leading to selection bias.

Selection bias is an inclination or partiality introduced by the selection of individuals, groups (such as workers or worksites), or data for analysis in such a manner that accurate randomization is not realized.³⁰ The effect is that the sample acquired may not fully represent the intended study population, because program participants and nonparticipants may not be similar enough to make valid comparisons of outcomes or adequately signify the eligible population. As a result, the representation of the sample may be compromised and conclusions may be limited or even wrong.

Selection bias might occur at the organizational level as well as the worker level. If a study includes multiple worksites, some may be willing to participate but others not, and willingness might be associated with size, readiness for change, innovation, or leadership's commitment to worker safety and health. Similarly, individual workers within those worksites who volunteer to participate may differ from their counterparts who decline to participate (as in demographics, baseline well-being, risk behaviors, or communication styles). Hence, it is best to use a control group or workplace that is equally willing to participate, as determined by the investigators in preliminary screening for eligibility and willingness. This reduces the likelihood of confounding by those distinguishing characteristics. Although not always possible, using a RCT design with all participants recruited via standard protocol may help alleviate potential selection bias and other biases.

Next, obtaining a large, diverse sample of workers will likely aid in achieving a more accurate representation of the population of interest, along with a more accurate analysis of how participating workers differ from those who do not. There are several econometric approaches and techniques that can help address and control for selection bias due even to unobservable determinants of the outcomes of interest.³¹ Terza et al³² describe an extension of these techniques to work in health-related program evaluations.

Nonetheless, despite best attempts, selection bias may be inevitable. Studies with substantial selection bias may still prove useful by highlighting the issue of effectiveness (in the “real world”), even if they do not address the issue of efficacy (in an ideal situation). If participation is less than expected, using qualitative methods to better understand the reasons for nonparticipation can add to the study findings and improve future interventions. Finally, another strategy to help counter this potential inevitability is a series of replication studies in different settings and populations that demonstrate that selection bias had minimal influence on study outcomes.

Attrition

Attrition occurs when individuals or participating worksites are lost from a sample over time³⁰ and may, in fact, produce selection bias if there is differential loss to follow-up. Although not exclusive to the TWH field, attrition can occur at multiple levels and thus has particular applicability here. At both the organizational and worker levels, potential factors include changes in priorities, mergers, downsizing, and turnover, as well as type of work environment and work itself. In fact, some work settings, such as contingent, seasonal or precarious work, are more prone to attrition than others. Moreover, support (or lack thereof) at the organizational and management levels is a vital factor that can influence workers’ ability to participate and remain in a study, thereby impacting both recruitment and attrition.

Although a longer study timeframe is often most ideal, strategies to address attrition might include a more compact study timeframe, especially in an industry prone to high turnover rates. While shorter-term studies limit the types of outcomes that can be assessed, they do tend to suffer less from attrition. Having solid leadership buy-in at the outset of any study is highly encouraged to address this issue, as is having strong champions, incentives, and continuous active and strategic communication between participants and those implementing the TWH program. Understanding the patterns of attrition, whenever possible, is an added but useful effort. For example, it would be valuable to know whether participants—at either the organizational or worker level—are missing at random or not. If an organization drops out of a study, investigators should attempt to conduct a qualitative “exit interview” to fully understand why. Although some researchers take this approach, such interviews are not widely performed and the information gleaned is not always shared in publications. In an emerging field such as TWH, publishing these findings and lessons learned can be particularly beneficial.

Unanticipated Changes to the Work Environment

In all work environments, unexpected changes during a research study can occur at multiple levels and with different implications. These changes can include external ones, such as political, legal, social, and economic, as well as internal ones within the work environment (structure, leadership, management, and workforce). Although the impact of some changes may be easier to ascertain and quantify than that of others, researchers should consider using an array of assessment tools to establish the baseline status of the organization and to track any changes in relation to potential unexpected external and internal issues that might impact the organization, its workers, and existent TWH endeavors. For instance, an external change impacted by the state of the economy might influence an organization's hiring, turnover rates, layoffs, morale, and working conditions. All of these factors might be applicable internally, to workers’ safety, health, and well-being outcomes. Although some changes can positively or negatively affect an organization's current TWH efforts, it is wise to try to salvage and continue efforts and to report changes accordingly. Even if an unexpected change may affect the nature of integration, the participating parties may still benefit and valuable reportable information for the scientific community and other stakeholders can be obtained.²

Key Conclusions

Selection bias, attrition, and unanticipated changes to the work environment are not unique to the field of TWH. Nonetheless, we highlight them here given that they are ubiquitous and potentially problematic at both the organizational and worker level. When the “real-world” setting creates such limitations for the chosen method and plan, researchers are encouraged to describe those in their publications and to consider all available statistical approaches to handle these challenges.

Qualitative and Mixed-Methods Approaches in TWH Research

Why Use Qualitative or Mixed-Methods Approaches?

As noted earlier, study design and research methods used in an investigation should always align with the research question(s) at hand. Increasingly, researchers have been gathering qualitative data—particularly useful when studying new and complex areas or questions, hard-to-measure constructs, and experimental settings—to help describe, understand, and explain their quantitative findings. Despite this usefulness and recommendations for using a mixed-methods (quantitative and qualitative) approach in intervention research, few TWH studies have done so, especially in RCTs. In 2014, O’Cathain et al³³ reviewed all of the health research proposals funded in the United Kingdom from 2001 to 2010 that involved RCTs. They found that only 2% of the funded proposals also involved qualitative methods, and those provided only brief descriptions of the qualitative methods used.

The evidence base would be more actionable if researchers used qualitative or mixed-methods research approaches alongside or independent of quantitative methods. For instance, case studies (see next section), focus groups, Photo Voice,³⁴ and peer interviews can empower workers to share their insights and experiences and learn from one another.³⁵ Online social media forums with images and photography sharing, toolboxes, and active online community discussions are other approaches. These strategies may be used throughout the research process and combined with quantitative assessments to confirm scientific accuracy and provide metrics around participation. These metrics and other measures of dissemination and implementation^13,14 would offer supplementary information on the process and strategies used, why workers or employers participate or not, and the makeup of the intended audience.

Whereas using only qualitative research methods can provide the researcher with useful information about the effectiveness of a TWH intervention, some circumstances may call for a mixed-methods approach to more comprehensively answer the what, why, and how of the research issue being examined.³⁶ Mixed-methods allow for the inclusion of mental schemes into the same area of analysis, to enable a more collective understanding of a phenomenon.³⁷

Additional Qualitative or Mixed-Methods Research

The Experience Sampling Method (ESM) is a qualitative research method that provides insight into workers’ experiences by frequently sampling their behaviors or perceptions before, during, or even after the study period.³⁸ Often the sampling is conducted at randomly selected times throughout 1 or more days.³⁹ The data gathered in this “real life” monitoring activity is then reviewed with the workers through in-depth interviews to better understand the factors that might have influenced the study-related behaviors or work patterns. Ecological Momentary Assessment (EMA) is another qualitative approach that provides real-time and repeated qualitative and quantitative data.⁴⁰ EMA gathers qualitative data by using study participants’ mobile phones to ascertain their emotions, cognition, and behaviors, producing data on safety and health outcomes within the work and community environment at different times in a day. Regardless of the qualitative or mixed-methods approach used, assessments must be rigorous and tailored for the particular work organization and workforce. In this context, Table 1 provides some examples of how qualitative and mixed-methods approaches can be used at various stages of a TWH intervention.

TABLE 1.

Selected Examples of Mixed-Methods Approaches for Total Worker Health Studies

Activities	Quantitative	Qualitative
Formulate the research question	Review and analyze data from injury and other worksite reports, workers’ compensation claims, observations in worksite walkthroughs (checklists, eg), and needs-assessment surveys	Conduct case studies, focus groups, and/or interviews with workers, managers, and CEOs; use semi-structured interviews
Select measures	Pilot test measurement scales with target audience, using existing or new survey instruments to assess their reliability and validity; use direct observation methods, checklists	Conduct focus groups with workers; ensure cultural appropriateness of data collection instrument; conduct a pile sort activity to identify task frequency associated with a specific job
Develop an intervention	Review and analyze checklists and other tools to determine what type of intervention will address the safety and health risks identified in earlier stages	Use Photo Voice, focus groups, or subject diaries to determine what aspects of an intervention are likely to be successful; use key informant interviews
Implement and evaluate an intervention	Use RCT, pre-post design; assess biometric indicators, injury rates, surveys to assess effectiveness at reducing risk or adverse outcomes	Examine attrition rates to determine why participants dropped out; use interviews or open-ended questions to assess participant reaction to intervention
Translate research to practice	Use dissemination and implementation science models, gap analysis of target behavior; consider cost of implementing through administrative data, reduction in injury rates	Conduct interviews with workers and thought leaders on barriers, relative advantages, and other measures of dissemination and implementation; assess readiness for intervention via observational interviews and community forums to determine scalability

RCT, randomized controlled trial; TWH, Total Worker Health^®.

Reporting of TWH Results From Mixed-Methods Approaches

Publishing studies that use qualitative or mixed-methods approaches can be challenging, especially when the approaches vary in types of results and research questions, or they produce conflicting results.⁴¹ The authors of this paper strongly encourage journal editors to consider enlisting more reviewers who are well-versed in mixed-methods and then in publishing such findings to help advance the field of TWH more rapidly. Additional dissemination channels that researchers might consider include webinars, social media, and other web and communication outlets. These channels can improve dissemination, enhance visibility to more diverse audiences, and draw attention to mixed-methods research in TWH studies. This dissemination reach can be measured with different tools, such as altmetrics.⁴² Also known as “alternative metrics,” they measure influence and reach of academic research and other outputs by following traffic to and conversations about them across online and social media outlets.⁴³ Such dissemination channels and metrics hold the potential to improve adoption of research via targeted communication to practitioners and occupational safety and health professionals who might not have access to paid peer-review journal articles and/or also use other channels to glean the latest research findings.

Key Conclusions

Relevant stakeholders should make a greater case for—and support—studies using qualitative and mixed-methods approaches, using both more commonplace and innovative techniques. Studies that are more inductive enable researchers to discover new patterns and relationships among variables. They can build on prior findings and existing frameworks as well as raise questions that lead to new hypotheses tests. Researchers have noted that the big advances in science come from induction and not from deduction, which merely confirms the correctness of prior explanations but adds little to new knowledge.⁴⁴ Finally, in addition to traditional publication formats, authors might also consider using alternative ones such as online forums and social media to draw more attention to such methods (eg, case studies) that are not as widely reported and, as a result, may be overlooked and/or under-appreciated.

Case Study Design in TWH Research

Why Case Studies?

Case studies are a promising research method in the developing field of TWH, as they may be applied to clarify⁹ highly complex issues and concepts such as integration, leadership, worker engagement, culture, confidentiality and privacy of workers, and worker well-being. One of the greatest strengths of a case study here is the opportunity to gather information on the context and rationales for TWH efforts, their reach and depth vis-à-vis the extent of participation among eligible members of the target audience, their specific details, and their perceived impact. Indeed, the ability to query in detail the why and how lies at the heart of the appeal of case studies.⁴⁵ There is a need to document interventions in terms of how they are delivered as well as how they are received, to gain a fuller understanding of not only the many ways that the concept of TWH can support worker safety, health, and well-being but also the obstacles that might arise in implementing TWH programs.

Case studies are in-depth investigations of a single person, group, event, or community.⁴⁵ Typically, data are gathered from a variety of sources and by using several different methods (such as direct observations, key informant interviews, focus groups, worker surveys, quantitative data collection, or interviews) to understand complex social phenomena. Case studies describe and investigate contemporary real-life phenomena through detailed contextual analysis of a limited number of events or conditions and their interactions and relationships. They are routinely used in social and business research⁴⁶ and may be retrospective, focus on current activities, or be conducted longitudinally.

An example of a case study representative of TWH efforts is the LifeWorks@TURCK program (Minneapolis, MN).⁴⁷ This case study presents the 10-year experience of a medium-sized manufacturing facility that intentionally addressed the safety, health, and well-being of its workers, their families, and the community and in the process fundamentally changed its corporate culture. The case study provides context, is organized according to a set of best practice program design principles, and includes both quantitative assessment of program performance as well as qualitative perspectives and viewpoints.

Approach to the Case Study Design

Ideally, a case study follows a set of principles and characteristics that generate a well-rounded understanding of the cultural systems of action at play (ie, the interrelated activities operative in the TWH program at hand).⁴⁸ Case study designs need to be appropriate to the research question, record systematically a “web of evidence” using qualitative, quantitative, or mixed-methods, and be linked to a theoretical framework.

Case study designs might be single-case or multiple-case. A single-case design is adopted when no other cases are available for replication; that is, it pertains to a single unique event. A multiple-case design must follow a replication rather than a sampling logic, using a rigorous and standardized research framework.⁴⁹ A case study may also be categorized as exploratory, descriptive, or explanatory.^50–52 These three categories of case studies may be seen to provide different levels of depth that satisfy the three tenets of the qualitative method to describe, understand, and explain.

More specifically, exploratory case studies involve efforts prior to defining research questions or hypotheses and may be considered a prelude to social research. It is assumed that a conceptual study framework created a priori guides this effort, and pilot projects are useful in determining the final protocols to be used. Case selection for exploratory studies should maximize the opportunity to learn about the situation.

Descriptive case studies cover the depth and scope of the case being studied, since the overall aim is to form hypotheses of cause–effect relationships. This may include study and in-depth description of multiple conditions or situations that may be compared with each other and to an idealized hypothetical or theoretical construct or pattern. Pattern-matching techniques (comparisons of two patterns to determine whether they are the same or different) may be used in analyses of complex and multivariate cases and are defined as the linkage of data to propositions as well as criteria for interpreting the study findings.⁵³

Explanatory case studies are designed to address causality. They describe and examine data closely at both a surface level and a deep level to explain phenomena observed in the data. Often, explanatory cases will involve complex and multivariate situations. Here, just as in descriptive cases, pattern-matching may be used in the analysis.⁵³

Advantages and Disadvantages of Case Studies

Advantages of case studies include the fact that examination of the data is conducted within the context of its use. This is in direct opposition to what happens in experimentation, where there is an attempt to deliberately isolate a phenomenon from its context and focus on a limited number of variables. Case studies also allow for both quantitative and qualitative analyses of data, along with detailed qualitative accounts that help to explain complexities of real-life situations, a promising consideration for TWH research.

One of the key uses of case studies is to develop an understanding of the ways in which employers and organizations select, adopt, and sustain certain TWH approaches. This use is particularly important given that a complete realization of the TWH approach must be built through stages. It is a learning process that requires development of managerial and worker perspectives and practices. Different employers are likely to take diverse paths in their response to the particular nature of safety, health, and well-being issues they confront and in their relationship to other organizational goals. Sometimes TWH interventions come as a packaged program prepared by others, and sometimes they may be more homegrown. In either case, it is useful to know who the key leaders and facilitators are, the resources they rely upon, the origins of their thinking, and the problems and priorities they are addressing. This is useful not only for devising ways to promote the TWH approach but also for gaining insight into variations in paths to TWH, for developing guidance concerning a pathway, and even for supporting the case for TWH by documenting that many of its components are logically compelling responses to immediate problems and opportunities. Any management innovation may be adopted with attention to its specific fit to problems at hand or, instead, based on a perception that it is universally effective, regardless of context. Case studies may help researchers better understand the different pathways to adoption and how these might translate into various contexts. Although TWH approaches may be broadly considered as ideal practice everywhere, documenting instances of the situation-specific and well-considered adoption of its elements can help make the case for wider adoption and adaptation. Finally, case studies in mixed and/or triangulation methods similarly use more than one approach to explain and investigate a research question more comprehensively by exploring it from different angles. The use of either or both of these may confirm validity of the overall process, findings, or conclusions, and recommendations offered by a TWH program.⁵⁴

Disadvantages of case studies include the perception that they lack rigor. They may be labeled as being too long or difficult to conduct and are thought to produce massive amounts of documentation requiring extensive time and resources. At times, they are perceived as providing little basis for scientific generalization because they use only one or a few subjects. A contrary perspective is that in any single application in a work setting, there typically is little or no emphasis on generalization. The localized approach is meant to improve the situation at that particular organization at that particular time, regardless of the generalizability of the TWH intervention, how it occurred, and its results.

Context, in terms of the research questions that the TWH study is designed to address, may be most important for determining whether a case study approach is preferable. For example, although a case study may be less effective in assessing efficacy where causality and rigor are central, the use of it may be more suitable for generating hypotheses or examining possible mechanisms of how an approach might work within certain types of settings.

Key Conclusions

Though there are both advantages and disadvantages to using a case study approach, it is a valid method to further the TWH evidence base and complements more quantitative comparisons. The techniques delineated above can be used to support case studies individually and also to pool their results. Using case studies may even prove to be a more attractive learning approach and buy-in option for organizations that place attention on what similar competitor organizations are doing to assure a safer and healthier workforce.

Measures and Outcomes Relevant to TWH

TWH Measures and Outcomes

TWH approaches have holistic applicability for wide-ranging worker safety, health, and well-being issues across occupations and industries. This is exhibited in the numerous and varied outcomes and measures reported across TWH studies.^6,8,9 Although a diversity of applicable measures and outcomes can be beneficial for new learning, it is challenging to assess and compare findings across studies that include different measures and outcomes. Indeed, Feltner et al¹⁰ reviewed 15 studies on the effectiveness of TWH interventions and were unable to determine the strength of evidence across the studies because of the diversity of measures as well as methodological differences and limitations.

As is the case with all emerging areas, the definition and scope of the TWH field continue to evolve on the basis of scientific research findings and input from stakeholders.¹² For example, the revision of NIOSH's definition of TWH in 2015 better reflects TWH priorities and helps to differentiate the TWH paradigm from traditional workplace wellness programs that do not integrate worker protection fundamentals.¹² This revision had implications for the TWH-related measures and outcomes selected and studied subsequently. That said, studies based on the earlier definition of TWH were pivotal in building the evidence base and foundational in shaping the eventual refining and refocusing of the TWH paradigm.^{2,5,6,8,10,55–57}

The Pathways to Prevention 2015 Meeting recommended that common TWH measures were needed to compare results across research studies.¹¹ TWH measures include both independent and dependent variables. The independent variables are the baseline characteristics of workers and workplaces, as well as the interventions themselves, meaning the workplace policies, practices, and programs that are intended to ensure safety and grow health. For instance, relevant measures for small businesses or contingent workforces may differ from those addressing issues faced by workers in large organizations or more stable workforces. The dependent variables, or outcomes, similarly include organizational changes in these same workplaces, along with health, behavioral, attitudinal, and other characteristics of the workers that might demonstrate the impact of the intervention.⁵⁸ Measures may be obtained at the organizational level from a worksite representative or at the worker level through an employee survey. Outcome measures focused on the organizational level may include new worksite programs, policies, and practices. At the worker level, they may include perceived working conditions and supervisor support, for example.

The authors recognize and endorse that the wide variety of possible TWH programs will necessarily require different independent and outcome variables, in line with the types of interventions being evaluated and their hypothesized or desired changes from baseline. Therefore, each set of investigators will inevitably select specific measures that are tailored to their research questions. It is true that comparability of findings would be facilitated by the adaptation of core or standard TWH measures. For the outcomes, this might entail assessment of broad, multi-dimensional well-being in addition to the specific health conditions targeted by the intervention. While the authors concur with prior work that steps toward achieving a core set of relevant measures and outcomes is a useful endeavor, in this report, the authors outline areas of measures and provide examples to illustrate them instead of listing specific measures that should only be used. The authors contend that doing otherwise at this juncture may not only constrain the TWH field but also confirm such measures and outcomes without the lengthy base of research evidence needed to validate them.

Table 2 provides a broad overview of examples of TWH measures and outcomes that fall into several categories, span multiple levels, and have short-term and/or long-term implications, as described in the TWH literature and referenced throughout this document. All structural and process measures are assessed at the organization level, but outcome measures can also be assessed at the worker level. In this sample table we first delineate program structure and process measures. Structural measures assesses the programs, policies, and environmental supports and degree of integration and can be guided as well as evaluated by established and validated tools such as the CDC Worksite Health ScoreCard,⁵⁹ Leading by Example,⁶⁰ the CPH-NEW Program Evaluation Rating Sheet,⁶¹ and similar organizational assessment tools.⁶² Here, also, the plurality and evolving nature of available measures has led to inconsistency among investigations; standardization may become possible in the future. Process measures consist of these factors: (1) the degree to which the program (as a whole or specific elements) is implemented as planned (its fidelity), (2) the extent to which all aspects of the interventions are put in place with appropriate intensity, frequency, and duration (its dose), and (3) the degree of awareness, participation, and satisfaction with various program components and the program overall (the dose received and program reach).

TABLE 2.

Examples of Total Worker Health Measures and Outcomes

Structural Measures
Program elements
Degree of integration by design

Process Measures
Recruitment methods and statistics
Fidelity of intervention implementation (degree of intervention achieved)
Dose (adoption, participation)
Social validity/satisfaction from management and worker perspectives

Outcome Measures
Organization Level Outcome Categories	Sample Constructs
Leadership	• Organizational commitment • Support • Meaningful work and engagement
Policy	• Annual, family, and medical leave • Compensation and benefits • Employer-sponsored health insurance
Occupational injury and illness rates	• OSHA logs • Workers’ compensation claims
Working conditions and environment	• Physical hazards, exposures, and reductions (such as chemicals, physical and biological agents) • Psychosocial and human factors • Risk assessment and management • Job design • Work/life integration • Built environment • Engineering controls • Ergonomics
Safety culture	• Bullying, harassment, violence • Discrimination
Productivity	• Absenteeism • Presenteeism • Turnover rates

Worker Level Outcome Categories	Sample Constructs
Safety practices	• Personal protective equipment use • Safe patient handling
Health (short- and long-term)	• Musculoskeletal disorders • Injuries and near misses • Adverse mental and emotional health • Blood pressure • Cholesterol • Body mass index • Cardiovascular disease • Cancer
Job satisfaction	• Flexibility • Motivation • Engagement/participation • Morale
Well-being	• Health-related quality of life • Functional status • Stress or distress
Healthcare utilization	• Hospitalizations • Emergency room visits • Outpatient clinic visits
Personal health behaviors	• Sleep practices • Tobacco use • Alcohol use/misuse • Drug use • Physical activity • Sedentariness • Nutrition

TWH, Total Worker Health^®.

Next, the table illustrates six key categories of outcomes for consideration at the organization level: leadership, policy, occupational injury and illness rates, working conditions and environment, safety culture, and productivity. At the worker level, we highlight six relevant categories: safety practices, health (both short- and long-term), job satisfaction, well-being, health care utilization, and personal health behaviors. It may also be useful to ascertain the same particular measure(s) at both organization and worker levels to compare perceptions of the employers versus employees. Table 2 is not meant to provide an exhaustive list of TWH measures and outcomes; rather, it offers several examples for consideration that have been and continue to be used in TWH studies (both intervention and nonintervention).

Emerging TWH Measures

Many of the measures listed above are not specific to the evaluation of TWH interventions (although the integrated use of them is), but some of the most relevant measures may only just be emerging in the literature. These assess the degree to which the intervention integrates protection from work-related safety and health hazards with promotion of injury and illness prevention efforts. As reflected in the table, structural measures include program elements—addressing both work and nonwork risks and opportunities, and coordination of activities—and the degree of integration by design.^2,3,29,63 Process measures that may be considered include recruitment methods and statistics (such as eligible organizations/workers, those deemed ineligible and why, those enrolled, number completing measurement waves, and reasons for dropout), fidelity of intervention implementation and integrity, and dose (adoption and participation rates).

The social validity of TWH interventions is another valuable category of process measurement and includes participant ratings of the acceptability of the goals, procedures, and outcomes of an intervention. Finally, investigators should also consider assessing transferability—the successful application of the TWH intervention to diverse organizations, occupations, industries, and workers.

What to Consider When Selecting TWH Measures and Outcomes

As noted above, the unique worksite and workforce will determine the research question(s) and study design, which in turn will guide the measure and outcome selection process. Each measure should be considered in terms of whether it requires single, repeated, short-term, and/or long-term assessment. The conceptual causal model that underlies the study should also specify whether each outcome is considered intermediate or lagging. Examples of intermediate outcomes include eliminated or reduced hazards, workplace safety practices, and personal health behaviors. Lagging outcomes may include incidence of musculoskeletal disorders; disability related to injuries, illness, or chronic disease; and overall mortality rates.

Well-being, in particular, is a broad concept that extends beyond traditional conceptions of occupational safety and health and is an outcome that can be impacted by TWH interventions.⁶⁴ Defining and measuring well-being can be complex, but many validated measures are available that address such facets as functional status, quality of life, and stress or distress. Recently, NIOSH and RAND published an article on the development of a worker well-being framework to expand upon current understanding of well-being in the occupational safety and health literature.⁶⁵

Outcomes measured objectively, especially if sequential or longer-term, or those obtained from data collection efforts that are other than self-reported, are highly desirable. Examples of objective outcomes include organizational records of recordable injuries, absenteeism, turnover, and production. It should be noted, though, that some investigators do not view these as truly objective because they rely fundamentally on the behavior of a worker and/or employer representative. Organizations may already be collecting such data for other purposes (for instance, occupational injury and illness cases recorded on OSHA logs) and be willing to share them with researchers for use as outcome measures in a study. Objective measures can also include researcher-assessed environmental/exposure measurements (vibration, noise, particulate, or chemical, eg), biomarkers (such as blood, urine, saliva, or hair samples), anthropometric measurements, physiological assessments, and direct observations of work conditions or practices. Obtaining workers’ permission to collect, analyze, and report data and observing confidentiality and privacy are paramount.

Available Resources for Consideration

It is key to continue to develop reliable, validated, evidence-based TWH resources, assessment tools, scales, and surveys with both quantitative and qualitative approaches to comprehensively assess and evaluate measures and outcomes, including those relevant to integration. In addition to screeners, questions, and checklists about the presence or absence of specific outcomes (such as injuries or disease diagnosis and treatment), investigators should consider the benefits and challenges of brief, simple measures (a single question to measure self-rated safety and health,⁶⁶ for instance) and more broad measures (multiple questions used to measure a single construct,⁶⁷ eg), which collect deeper data but may be more costly and time-consuming to administer and more complicated to analyze. Relevant resources to aid in this process and guide intervention efforts are available on the NIOSH TWH website and elsewhere.^3,62,68,69

Key Conclusions

It is hoped that, over time, a core set of relevant measures and outcomes will be adopted by TWH researchers. Meanwhile, it is appropriate and even valuable to have a variety of independent and dependent variables under study in this still emerging and wide-ranging discipline. Ultimately, focusing on the specific issues to be solved should govern the selection of metrics and measures needed to respond to those very issues. The prime focus of this section is on organizational level and worker level outcomes and measures; however, the authors recognize that there are implications of addressing these that go beyond the scope of the worksite, working environment, and work itself, not underscored in great depth in this report. Future TWH studies might consider including measures and outcomes capturing these additional levels of influence, using some of the emerging resources highlighted.

CONCLUSION

Final Thoughts and Next Steps

This Workshop proceedings summary of methodological and measurement considerations in TWH research is intended to highlight existing approaches, their advantages and disadvantages, as well as propose additional options that hold promise for advancing the field. Many of these principles are understood and articulated in other sources but may not be consistently observed or practiced in research reported in the TWH literature. Comparability of scientific findings is an important focus to solidify any field but cannot be sought at too early a stage. The TWH field has broad applicability, making a narrowing of its scope by the prescription of specific recommendations, methods, measures, and outcomes that must be used potentially too limiting and disadvantageous.

Given its nascency and complexity, further building of the TWH scientific evidence base is essential. Future forums should build upon these Workshop topics by explicitly and directly connecting the methodological best practices offered here to specific methodological limitations of the TWH literature and studies, as the field continues to grow. Finally, additional exploration of TWH research methodological issues is necessary for not only intervention research but also basic/etiologic, translational, and surveillance research, and concrete examples are needed of how challenges can be overcome to drive research practices in the field of TWH.^8,10