The Healthy Work Survey

The Healthy Work Survey is a short, standardized online tool to assess the level of work-related psychosocial risk based on distributions from a U.S. national working population. Occupational medicine professionals can easily access this free tool and report to evaluate individual and workplace psychosocial hazards impacting worker health and safety.

Objectives

Work-related psychosocial stressors have been recognized as occupational hazards and assessed in workplaces in many countries for decades. Identifying tools to measure work-related psychosocial hazards to increase awareness in the United States about the impact on employees’ health and safety is critical (J Occup Environ Med. 2021;63:e245–e249).

Methods

We describe the development and psychometric validation of an online tool, the Healthy Work Survey, utilizing items from the National Institute for Occupational Safety and Health Quality of Work Life questionnaire.

Results

There are 55 items in the final core work section of the HWS. Factor analyses confirmed nine factors, and the subsequent multi-item scales had acceptable internal consistency. A user-friendly, online system and automated report compares individual’s and organization’s scores to distributions from a representative US working population (General Social Survey Quality of Work Life).

Discussion

The HWS is a reliable, valid tool for organizations and individuals to assess psychosocial work hazards.

LEARNING OUTCOMES

• Occupational health practitioners, researchers, and other professionals interested in workplace surveillance for psychosocial hazard risk will be able to utilize the online Healthy Work Survey system after reading this article.

• Upon completion of reading this journal article, occupational health practitioners, researchers, and other professionals will be able to justify and explain why the psychosocial risk assessment tool is valid and helpful to employers and/or other organizations they work with.

Psychosocial work hazards refer to a range of job characteristics and work organization factors that are known to be sources of chronic stress in workplaces. Understanding, evaluating, and mitigating these workplace hazards could significantly reduce the prevalence of chronic stress-related illnesses and injuries among working people.¹ Many of these stressors have been extensively studied in occupational epidemiological observational studies, showing increased risks of psychological disorders, musculoskeletal disorders, injuries, and cardiovascular diseases and mortality.^2,3

Currently, in the United States, there is no standardized way to identify workplace psychosocial hazards. We have developed an assessment tool (the Healthy Work Survey [HWS]) to help organizations understand and evaluate psychosocial work hazards in their workplaces and to encourage them to use this information to reduce sources of stress and improve worker health and safety. To our understanding, this is the first workplace survey in the United States that has the advantage of (1) standardized scales to assess psychosocial hazards, (2) ease of administration online, (3) an automated report of results, (4) comparison to US national scores, (5) reasonably short, and (6) being free. This article describes the justification, development, and validation of this new “psychosocial hazard assessment tool” in comparison to other existing work-related psychosocial questionnaires.

The HWS is a key component of the Healthy Work Campaign (www.healthywork.org), a public health campaign to increase awareness in the United States about the health impacts of work stressors on working people. The HWS project was initiated in January 2018 with a grant from the nonprofit, occupational health research foundation, the Center for Social Epidemiology. The HWS was developed to include items and scales from theory-based models of job stress that have been empirically tested for decades and that are part of most nationally representative working population surveys (eg, job demands, job control, social support, rewards, job insecurity, work-family conflict [WFC]), as well as several important “emerging work stressors” (schedule control, electronic monitoring, bullying, emotional labor, organizational justice) that are not as commonly found in national surveys. These sources of work stress are known risk factors for mental health problems and chronic disease (eg, burnout, depression, cardiovascular disease, musculoskeletal disorders),^4,5 as well as with injuries and productivity outcomes (eg, sickness absence, presenteeism).^6–8

The HWS is an anonymous, free, online tool for individual workers and organizations to use to identify work-related stressors. The automated reports allow individuals and organizations to compare themselves to national scores, provide information to help understand the effects of work-related stressors on worker’s physical and mental health, and describe steps that could be taken to address the issues. The HWS may also be used by researchers interested in measuring psychosocial work stressors or for evaluating work organization interventions in various working populations. The HWS for individuals and organizations was launched online in 2021. We believe the HWS is the first of its kind in the United States and could be a useful tool for both individuals and organizations to assess and mitigate psychosocial risks in the workplace and prevent workplace injury and illness.

BACKGROUND

Commonly studied psychosocial job stress models include the following: high job demands and low job control (job strain),⁹ job demands-resources,¹⁰ lack of social support,¹¹ WFC,¹² effort-reward imbalance,¹³ long work hours and shift work,¹⁴ and bullying/harassment.¹⁵ The direct (health care) and indirect (productivity) costs of these work stressors have been calculated in several reports to total hundreds of billions of dollars annually.¹⁶ United States economists at Harvard and Stanford conservatively estimated that work stressors contribute to 120,000 deaths per year and 5%–8% of annual health care costs in the United States.¹⁷

Many countries recognize the impact of psychosocial hazards on health and safety, especially on workplace mental health, with standards or guidelines referring to “psychological health and safety.”¹⁸ A number of countries have also added psychosocial risk management to occupational health and safety regulatory frameworks, including the European Union, United Kingdom, Canada, Japan, Colombia, and South Korea.^19,20 The US Occupational Safety and Health Administration (OSHA Act, 1970) does not yet have a standard or guidelines specifically regulating psychosocial work hazards despite evidence that these hazards negatively impact worker health and safety.¹ In June 2021, the International Organization for Standardization released a new international standard (ISO 45003:2021), which offers “guidelines for managing psychosocial risks within an OSH Management system” (expanding ISO 45001).²¹ It is potentially applicable to organizations of all sizes and in all sectors to prevent work-related injury and ill health and promote mental and physical well-being at work. Unfortunately, unlike Canada’s Psychological Health and Safety Standard, ISO 45003 does not recommend specific resources or methods that organizations could use to identify or manage work-related psychosocial risks. Current psychosocial risk questionnaires that measure various work organization factors are generally designed for large population studies, and some are fairly lengthy. Therefore, we reviewed existing questionnaires to develop an evidenced-based, shorter psychosocial risk questionnaire for the United States that could be broadly available online.

Review of existing psychosocial risk questionnaires:

1. National Institute for Occupational Safety and Health (NIOSH) Quality of Work Life (QWL) Questionnaire²²—This questionnaire was added by the NIOSH as a 76-item module to the General Social Survey (GSS) in 2002 and administered every 4 years as part of a nationally representative population survey. The QWL questionnaire assesses some of the major work organization hazards in the US workforce. The purpose of the NIOSH QWL survey, according to NIOSH, is “to measure how work life and the work experience have changed since the earlier Quality of Employment Surveys and to establish benchmarks for future surveys.” Most of the questions came from the 1977 Quality of Employment Survey and consisted of several categories, including job level (41 items), culture/climate (11 items), healthy outcomes (nine items), other outcomes (six items), hours of work (six items), flexibility (one item), work-family (four items), supervision (three items), benefits (one item), and union (one item).²³ The questionnaire (having more than 60 items for working conditions) is too long for routine use and is missing some of the major emerging work stressors (eg, bullying, emotional labor). The QWL has not been used widely for work stress research and prevention at the workplace in the United States, in part because NIOSH does not own the data, and does not provide it on its Web site; rather, it has to be accessed through the GSS. Only a few empirical studies have examined the psychometric validity of QWL items and scales.^24,25

2. Job Content Questionnaire (JCQ)²⁶ is based on Karasek’s Demand-Control Model⁹ and has been used extensively for workplace research on task-related demands and control in the United States. However, the JCQ does not provide national comparisons due to a lack of recent national survey data using the JCQ in the United States. Furthermore, like the NIOSH QWL, and several other work stress model-based questionnaires, the JCQ is limited in assessing important emerging work stressors. For example, it does not include items for emotional demands/emotional labor, organizational justice, electronic monitoring, and wage theft. Such variables may be available in a new version of the JCQ being developed.

3. National Health Interview Survey–Occupational Health Supplement (NHIS-OHS),²⁷ developed by NIOSH, has a much larger sample size, but has only a few questions on psychosocial working conditions, and is only conducted every 5 years. In 2015, there were national prevalence data on eight single-item workplace psychosocial factors²⁸ and five work organization factors²⁹ available on NIOSH’s Web site.

4. Copenhagen Psychosocial Questionnaire (COPSOQ)³⁰ is used extensively throughout the world, is available in 25 languages, and is in its third version having been evaluated in six countries. It includes the main psychosocial work stressor domains, including the job demand-control (job strain) model and effort-reward imbalance as well as newly emerging psychosocial theories on emotional demands, and organizational injustice. It includes population-based reference values that organizations may use to assess the need for action and preventive measures in the workplace. The CoPsoQ-istas21³¹ is a Spanish version of the COPSOQ used in national surveys in Spain. The StressAssess survey³² in Canada used some items and validated scales from the COPSOQ to measure workplace psychosocial factors. Sponsored by the Occupational Health Clinics for Ontario Workers in 2016, StressAssess uses Canadian population-based reference values so organizations can compare their scores to a Canadian population. StressAssess served as an excellent model for the development of the online HWS in the United States, but it does not provide US national comparisons, which we believe to be important in an instrument intended to identify level of risk for US workplaces.

5. NIOSH Worker Well-being Questionnaire (WellBQ)³³ was released in 2021 to measure a holistic construct of “worker well-being.” Developed by NIOSH and the RAND corporation, it includes 126 questions in five domains: (1) work evaluation and experience, (2) workplace policies and culture, (3) workplace physical environment and safety climate, (4) health status, and (5) home, community, and society, plus optional questions (five on work arrangements and ten on demographics). Psychometric data on the validity of the WellBQ were recently published.³⁴ However, it is less practical and less useful than the HWS, since the WellBQ:

   1. Is nearly twice as long as the HWS. Organizations have frequently asked us for shorter surveys that will contribute to a higher response rate.

   2. Is a “paper and pencil” survey. The HWS, in contrast, can be completed online.

   3. Is scored by the organization or individual completing the survey. The HWS, in contrast, provides a rapid automated report of survey results.

   4. Cannot provide national norms for items and scales, as the HWS can.

The HWS and WellBQ have similar sets of questions to measure health outcomes. The longer WellBQ does include some measures not included in the HWS, such as the following: meaningful work, supportive work culture (three questions in addition to respect and recognition, which are in the HWS), trust in management, health culture, availability of workplace health promotion programs, availability of job benefits (14 WellBQ questions vs two HWS questions), accommodations for disabilities and/or special needs, work-related positive and negative affect, job engagement, financial insecurity, and health behaviors, such as diet and tobacco and alcohol use.

On the other hand, the WellBQ omits questions related to key sources of stress at work, which are included in the HWS, including the following: work hours, mandatory overtime, being a supervisor, how employees are paid, low wages, wage theft, shiftwork, use of job skills, emotional demands, electronic monitoring, understaffing, job physical effort, procedural justice, and union representation. The WellBQ has only one question on job autonomy and one question on workload demands, whereas the HWS has three questions on each of these two key variables.³⁴

The HWS Project

The HWS project aimed to develop a short, standard questionnaire for risk assessment and management of major work organization and psychosocial hazards in US workplaces. After a review of existing psychosocial questionnaires beginning in 2017–2018, the majority of HWS scales and items were chosen from the NIOSH QWL questionnaire. A literature review on important, emerging work stressors also led to additional items being included. The HWS addresses the following needs:

• ➢A short questionnaire of validated items and scales that assesses major traditional and emerging work organization hazards that are applicable to a wide range of occupations and industries in the United States.
• ➢Identifies high-risk work organization hazards at a particular employer, organization, or group of union members, by comparing aggregate scores with national distributions and risk ranges of work organization hazards in the US workforce.

The HWS is intended to be utilized by employers (large, small, private/public), worker organizations (labor unions, occupational health and safety groups, worker labor centers), individuals, and researchers who are interested in evaluating psychosocial risks and work organization hazards and the health of workers and workplace(s). Ideally, organizations can use the online HWS tool and encourage their employees/members to complete the survey, anonymously and confidentially. The Healthy Work Campaign only shares the aggregate information with participating organizations and observes “human subjects” ethical principles, such as: no personal identifiers (anonymity), voluntary participation, the right to withdraw, and consent. There is an informed consent statement at the beginning of the HWS, clicking to continue the survey indicates consent to participate.

METHODS

The HWS development project was launched to determine the best and fewest number of items from the QWL and other sources to put together a comprehensive psychosocial work assessment tool. It had four main phases^35,36: (1) development of a GSS-QWL data set and psychometric testing of the NIOSH QWL items and scales using GSS data (2002–2014), (2) adding or creating new questions for newly emerging work stressors not commonly found in national US working population surveys, (3) calculating national distributions and risk ranges for the selected QWL items and scales using the GSS-QWL data (2002–2018), and (4) developing and pilot testing an online questionnaire and automated report system (2019–2021) and field testing this system with a group of union members.

Developing a NIOSH QWL Data Set 2002–2018 for Psychometric Testing

All data were downloaded from the GSS Web site³⁷ for four waves of the GSS-QWL module (2002, 2006, 2010, and 2014), before the fifth wave in 2018. Analysis was performed on questions that were available in all waves. The response pattern (distribution) of each QWL item was reviewed. Most QWL items had a good range of variation in responses. However, some items did not, such as “I get to do a number of different things in my job” (89% of the respondents agreed or strongly agreed to the item) and “treated with respect at work” (92% of the respondents agreed or strongly agreed to the item). All three items measuring “safety climate” and one item (“the safety and health conditions where I work are good”) showed 90% of respondents agreed or strongly agreed to the items. Further analysis of the psychometric properties of these items within scales was necessary.

The following psychometric tests of the NIOSH QWL items and scales were conducted in 2018–2019: content validity, factor analysis, differential item functioning (DIF) analysis, scale reliability (Cronbach α), and predictive (concurrent) validity.³⁸

1. Content validity—The NIOSH QWL items and scales were first examined to select the core items and scales that had been used in four waves of the GSS in 2002, 2006, 2010, and 2014. The GSS-QWL codebook with all questions and distributions is available online.³⁷ The core items and potential scales were examined for clear item wording and content validity relative to items typically used to measure the well-known job stress scales of demands (workload, conflicting demands), control (skill utilization, decision authority), support (supervisor, coworker), WFC, rewards (promotion opportunities, job insecurity, pay fairness), physical effort, and safety climate, and harassment.

2. Factor analysis—Exploratory factor analyses were conducted with the core multi-item scales using the 2002–2014 GSS-QWL data (n = 5914 workers). A Principal Component Analysis was used for extraction, and the rotation method was Varimax with Kaiser normalization. In addition, we replicated the factor analysis in the following subgroups by age (up to 44 years and 45+ years), sex (men and women), race/ethnicity (White, Black, and other), time, and occupation to test any differential construct validity of the scales. Two periods were tested: 2002–2006 versus 2010–2014. The following five occupational groups were examined: management, business, science, and arts occupations; service occupations; sales and office occupations; natural resources, construction, and maintenance occupations; and production, transportation, and material moving occupations.

3. Scale reliability—Reliability of the QWL-derived work stressor scales was assessed by the Cronbach α statistic using GSS-QWL data from 2002 to 2018.

4. DIF is a technique to evaluate whether different subgroups respond differently to particular items within a scale. The multi-item QWL scales identified in the factor analysis were tested to determine the best and least number of items needed for each scale based on how they performed in different subgroups. To simplify the interpretation of cross-language DIF and detect the most pronounced differences, we chose the criterion, “moderate to large” DIF (category C) over “slight to moderate” DIF (category B).³⁹ Category C was defined as items with partial gamma outside the interval (−0.31 to 0.31) and its 95% confidence interval significantly outside the interval (−0.21 to 0.21); category A (no or negligible DIF) as items with partial gamma within the interval (−0.21~0.21) or its 95% confidence interval including zero; category B as items located between categories A and C.⁴⁰ Analyses for item-level measurement equivalence between the aforementioned subgroups were also conducted.

5. Concurrent (predictive) validity—The selected QWL items and scales were examined for their association with ten health outcomes in the 2002–2014 GSS-QWL data: the four CDC Healthy Work Days⁴¹ measures, exhaustion, back pain, sleep problems, hypertension, depression, and missed work days (absenteeism). Although the CDC measures, exhaustion, and back pain were used in all four QWL surveys, hypertension, depression, and missing days were used only in the 2014 QWL survey. Sleep problems were used only in two surveys (2010 and 2014). The following multi-item scales that met validity criteria were selected for predictive (concurrent) validity testing: WFC, skill discretion, decision authority, job control, psychological demands, supervisor support, coworker support, resource adequacy, safety climate, discrimination, job insecurity, pay fairness, promotion, and reward. Some single QWL items were also included in the predictive validity analysis, including physical effort, work hours, work arrangements, type of payment, and shift schedule. Analyses were conducted with exposures as continuous variables and outcomes as dichotomous variables using a simple Pearson correlation matrix, and chi-squared tests were used to test with unweighted and weighted data considering the complex survey design of the GSS. The weight variable (WEIGHTALL) and the two variables for the complex survey design (VSTRAT and VPSU—the first and second stage sampling variables) were applied for these analyses. Multivariate logistic regression was also used to test predictive validity while controlling for age and sex.

Adding “Emerging Work Stressors”

We conducted a literature review of existing standard psychosocial questionnaires (see Background) and in consultation with our subject matter experts (see Acknowledgments). Based on our and colleagues’ expertise in work-related psychosocial stress research, we determined several important “emerging psychosocial stressors” that should be included in the HWS, that were not included in the GSS-QWL survey, and that have evidence of a relationship with poor health outcomes (psychological and physical).

Calculating National Statistics and Risk Ranges for the Selected QWL Scales

Once GSS-QWL items were selected and multi-item scales met validity criteria based on the psychometric analyses described previously, national distributions and risk ranges were calculated for these major work stressor domains based on GSS-QWL data from 2002 to 2018. This allowed for national comparisons to scores in the individual report and aggregate averages in the organization reports. In addition, high-, intermediate-, and low-risk ranges were established for the major stressor domains based on approximate tertile cut points. These cut points determine whether an organization’s group average (or an individual’s summed score) on any given work stressor scale is in the top tertile (high risk), middle tertile (intermediate risk), or bottom tertile (low risk) of the US national distribution.

Developing and Piloting an Online Questionnaire and Report System

Our team included university-based researchers, occupational health and safety experts, employer and union leaders, and workers who reviewed the HWS during its development process, and gave feedback on its content and areas for improvement. In addition, questionnaire experts completed the survey and commented on the applicability of the survey items, scales, missing domains, and areas that needed improvement. An expert in the development of computerized survey and report systems collaborated with the authors on the online HWS version and reporting system. The HWS along with the individual report was pilot-tested with stakeholders across a range of roles (including workers, labor unions, consultants, researchers, and OSH experts) and included anonymous feedback regarding its function and utility. A pilot test of the HWS system for organizations was completed with a group of union members (n = 51/100) in a small local union in Northern California in August 2021. The local consisted of employees within a single news media organization from several occupations in the arts, media, and entertainment industry, including technical/engineers, membership services (customer service), and facilities workers. The president of the local was approached by a member of our team (a fellow union leader) and expressed interest about launching the survey with her members. We supplied the president a unique link to the survey, which was emailed by the union to all 100 members. Recruitment was guided by our documentation—that participation is voluntary, the survey is anonymous, and individual data are never shared with the union or management. Union stewards who were coworkers conducted follow-up to encourage participation.

RESULTS

Psychometric Testing of the GSS-QWL Data Set

There were 61 core work items used in four waves of the GSS-QWL module (2002, 2006, 2010, and 2014) (see Supplemental Material, Appendix A, http://links.lww.com/JOM/B305). Based on the clarity of item wording and the content validity of standardized scales from dominant theoretical models (demand-control-support [job strain], effort-reward imbalance, WFC, and safety climate and their subscales), there were 49 items and scales selected to be included in the psychometric analyses for possible inclusion in the HWS and are denoted with an asterisk in Appendix A, http://links.lww.com/JOM/B305.

The results of the factor analyses of the 49 GSS-QWL items selected based on the content validity assessment confirmed nine factors: WFC, job control, supervisor and coworker support, psychological demands, physical job demands, safety climate, pay/fairness, discrimination/harassment. However, one item on WFC (“How hard is it to take time off during your work to take care of personal or family matters?”), one item on psychological job demands (“My job requires that I work very fast”), and one item on job insecurity (“How easy would it be for you to find a job with another employer with approximately the same income and fringe benefits as you have now?”) did not load on the supposed underlying constructs in the factor analysis with the whole QWL data. In addition, these items in WFC and psychological job demands scales also appeared to be problematic in the subgroup factor analyses, particularly, within the five occupational groups. Table 1 shows the results of the factor analysis for the job demand-control items and scales. The results of the factor analysis on all 49 items in each subgroup are available upon request.

Table 1.

Factor Analysis Results for the QWL Job Demand-Control Items/Scales (2002, 2006, 2010, and 2014) Rotated Component Matrix^a

	Component
	1	2	3	4
Job requires R to work fast	0.048	0.192	0.223	−0.599b
R has too much work to do well	−0.021	−0.019	−0.020	−0.715b
R has enough time to get the job done	0.373c	−0.001	0.039	0.653b
R free from conflicting demands	0.432c	0.075	0.030	0.477b
Job requires R to learn new things	0.076	0.685b	−0.120	−0.300c
Opportunity to develop my abilities	0.288	0.681b	−0.070	0.095
Job allows R use of skills	0.129	0.736b	0.009	0.106
R does numerous things on job	0.035	0.716b	0.025	−0.185
A lot of freedom to decide how to do job	0.266	0.486b	−0.088	0.343c
Supervisor concerned about welfare	0.786b	0.109	−0.074	0.048
Supervisor helpful to R in getting job done	0.801b	0.075	−0.019	0.073
Coworkers can be relied on when R needs help	0.646b	0.149	−0.061	0.162
Coworkers take a personal interest in R	0.658b	0.248	−0.065	0.024
R do repeated lifting	−0.024	−0.072	0.845b	0.029
R perform forceful hand movements	−0.068	−0.055	0.737b	−0.089
Rate physical effort	−0.076	−0.011	0.841b	−0.039

Extraction method: principal component analysis.

Demand/workload: R has enough time to get the job done (0.373, #1); R free from conflicting demands (0.432, #1, the strongest)

Control/decision making: a lot of freedom to decide how to do job (0.343, #4)

In general, component #4 = demand/workload, component #2 = control/decision making, component #1 = support, component #3: physical effort.

Rotation method: Varimax with Kaiser normalization.^a

^aRotation converged in five iterations.

^bThe strongest.

^cThe second strongest.

Table 2 presents the results of the reliability analyses of the multi-item scales. Most scales met the criteria for an acceptable level of internal consistency⁴² with Cronbach α scores of 0.60 and above, except psychological job demands (0.579) and coworker support (0.578). Several scales had somewhat improved Cronbach α scores after removing items including WFC (after removing one item “How hard is it to take time off during your work to take care of personal or family matters?”) and psychological job demands (after removing the item “My job requires that I work very fast”); therefore, these items were not used in the final instrument. In addition, since the reliability scores for the four item “job insecurity” scale and the two item “pay fairness” scale were unacceptably low (0.30 and 0.33, respectively), these were removed as multi-item scales and the single items “Job security is good” and “How fair is what you earn on the job in comparison to others doing the same type of work you do?” were included in the five item “Reward” scale. Cronbach α values were similar across subgroups.

TABLE 2.

Reliability (Cronbach α) Results for Multi-Item QWL Scales

Scale	Number of Items	Cronbach α	Cronbach α If an Item Is Removed
Work/family	3	0.568	0.633 (if the “FAMWKOFF” item is removed)
Psychological job demands	4	0.579	0.591 (if the “WORKFAST” item is removed)
Resource adequacy	3	0.617	0.665 (if the “TOOFEWWK” item is removed)
Skill discretion	4	0.716
Decision authority	3	0.602	0.614 (if the “WKDECIDE” item is removed)
Supervisor support	2	0.774
Coworker support	2	0.578
Physical demands	3	0.751
Safety climate	4	0.895
Job insecurity	4	0.300	0.401 (if the JOBFIND1 item is removed)
Pay/fairness	2	0.336
Promotion	2	0.625
Reward	5	0.688

The results of the DIF analysis using the criteria for a DIF item of slight to moderate DIF (>0.21) and also moderate to large DIF (>0.31) supported the removal of items that improved reliability scores. There were no slight to moderate DIF items in the decision authority or resource adequacy scales. There were some slight to moderate DIF items (but no moderate to large DIF items) between White and Black QWL participants, age groups, and between the occupational groups in the following scales: WFC, supervisor support, coworker support, promotion, and rewards.

There were some moderate to large DIF items in the following scales (mostly between racial groups and some occupational groups): psychological job demands (four items), workload (three items), skill discretion (three items), physical demands (three items), safety climate (four items), and job security (four items). Therefore, one item, “learn new things on the job,” was dropped from the final skill discretion scale. Two physical demands items “heavy lifting” and “hand movements” had moderate to large DIFs among occupations; therefore, it was decided to use only one item “physical effort” for the final instrument. Two items in the safety climate scale had moderate to large DIF between some occupational groups or between racial groups (White vs others); therefore, we used only two items for our final instrument. Finally, job security had three items with moderate to large DIF among men and women, age groups, race, or occupational groups; therefore, only one item (“job security is good”) was used in the final instrument and as part of the “reward” scale.

Table 3 shows the results of predictive/concurrent validity analyses of the QWL multi-item work stressor scales with ten health outcomes. Most of the multi-item QWL scales were significantly correlated (P < 0.05) with the four CDC healthy work days measures, and exhaustion, back pain, and sleep problems. The psychological (workload) demands, control, and support scales were not significantly correlated with self-reported hypertension, depression, or missing work days. Low coworker support was significantly correlated with hypertension and depression. Resource adequacy, job insecurity, and low rewards were significantly correlated with hypertension, depression, and missing work days. The results were similar in multivariate logistic regression analyses after controlling for age and sex. The results with complex survey design-based correlations and multivariate logistic regression analyses were also similar.

TABLE 3.

GSS Health Outcomes Used in Concurrent/Predictive Validity Results (10 Items)

Item #	Domains	Item Wording	Dichotomous Variables	Correlationb (P < 0.05)
1	CDC/general health (2002–2014)	(HEALTH1) Would you say that in general your health is (1) excellent, (2) very good, (3) good, (4) fair, or (5) poor?	Fair or poor (14.8%) vs excellent/very good/good	All work scalesa significantly correlated at P < 0.05
2	CDC/physical health (2002–2014)	(PHYSHLTH) Now thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good? Days 0–30	Physically unhealthy days: 14 or more days (6.7%)	All work scales significantly correlated at P < 0.05
3	CDC/mental health (2002–2014)	(MNTLHLTH) Now thinking about your mental health, which includes stress, depression, and problems with emotions, for how many days during the past 30 days was your mental health not good? Days 0–30	Mentally unhealthy days: 14 or more days (10.1%)	All work scales significantly correlated at P < 0.05
4	CDC/activity limitations (2002–2014)	(HLTHDAYS) During the past 30 days, for about how many days did your poor physical or mental health keep you from doing your usual activities, such as self-care, work, or recreation? Number of days in past 30 days: Days 0–30	Activity Limitations Days: 14 or more days (3.5%)	All work scales significantly correlated at P < 0.05 (except WFC, workload, psychological demands)
5	Exhaustion (2002–2014)	(USEDUP) How often during the past month have you felt used up at the end of the day? (1) never, (2) rarely, (3) sometimes, (4) often, and (5) very often	Very often/often (41.5%) vs never/rarely/sometimes	All work scales significantly correlated at P < 0.05
6	Back pain (2002–2014)	(BACKPAIN) In the past 12 months, have you had back pain every day for a week or more? (1) yes, (2) no	Yes (26.5%) vs no	All work scales significantly correlated at P < 0.05
7	Sleep problem (2010 and 2014)	(SLPPRBLM) During the past 12 months, how often have you had trouble going to sleep or staying asleep? (1) never, (2) rarely, (3) sometimes, (4) often	Often (22.1%) vs never/rarely/sometimes	All work scales significantly correlated at P < 0.05
8	Hypertension (2014 only)	Has a doctor, nurse, or other health professional ever told you that you had hypertension or high blood pressure? (HYPERTEN) (1) yes, (2) no	Yes (27.3%) vs no	Low coworker support Resource inadequacy Job insecurity Low rewards (P < 0.05)
9	Depression (2014 only)	Has a doctor, nurse, or other health professional EVER told you that you had depression? (DEPRESS) (1) yes, (2) no	Yes (15.5%) vs no	Low coworker support Resource inadequacy Job insecurity Low rewards (P < 0.05)
10	Missed work days (2014 only)	During the past 30 days, about how many days did you miss work due to your mental or physical health? Days 0–30	1 or more (14.5%) vs never	Resource inadequacy Job insecurity Low rewards (P < 0.05)

^aAll work scales included: WFC, skill discretion, decision authority, job control, psychological demands, physical effort, supervisor support, coworker support, resource adequacy, safety climate, discrimination, job insecurity, pay fairness, promotion, and rewards.

^bThere was no significant difference in analyses with unweighted and weighted data. In the simple correlation analysis, almost all correlations between the aforementioned scales and the four CDC measures were significant at the 0.05 level of P value or lower.

Several single items were also tested for predictive validity. Total work hours per week were correlated with the four CDC healthy days measures. As with the psychological demands, control, and support (supervisor support) scales, total work hours were not correlated with hypertension, depression, and missing work days. Work arrangements (regular, independent contractor, on-call workers, temp agency-hired workers, etc), work shift schedule (day shift, night shift, rotating shift, etc), and “way of payment” (salaried workers, hourly workers, paid by the job, etc) were significantly associated with three CDC healthy days measures (general health, physical health, and mental health) but not with the last CDC health measure (activity limitation), which was left out of the final instrument. Their predictive validity with the other health outcomes was not strong.

Some reduced scales were tested with the health outcomes. For example, there were three QWL items measuring WFC; however, during factor and DIF analyses, one item “able to take time off…” did not load on WFC and met the “moderate to large” DIF criteria, so a reduced WFC scale with only two items was tested. The correlations of the three-item and two-item WFC scales with health outcomes were very similar to each other; therefore, the two-item scale was used in the final instrument. In the QWL, there were four psychological job demands items; however, the “work fast” and “role conflict” items were excluded after the factor analysis. A reduced two item scale with only “workload” items, time pressure and amount of work, were correlated with the health outcomes, similarly to the four items job demands scale. Given our objective of creating a shorter questionnaire but a valid construct, we included a two-item workload scale and a three-item psychological demands scale that included workload and role conflict. A reduced “job control” scale was used in the final HWS, with all three “skill discretion” items, but only two of the three “decision authority” items. The item “I have a lot of say on my job” was not measured in the 2014 QWL, and removing this item did not change the predictive validity of the five-item job control scale.

Adding Emerging Work Stressors

After examination of the selected GSS-QWL items included after the psychometric testing, we found that several important, emerging work stressors were not included in the GSS-QWL questionnaire. In the GSS-QWL questionnaire (2002–2014), there were no questions about work schedule changes/notice (other than starting or quitting times), emotional demands or emotional labor, electronic surveillance (a type of demand), toxic physical exposures, exposure to dangerous equipment, workplace physical violence, low wages, wage theft, and organizational (procedural) justice. Thus, we added or created questions items for the missing work stressors based on our literature review of existing psychosocial questionnaires.

Work time changes and schedule change notice were included from the 2015 European working conditions survey (and used in the 2015 American Working Conditions Survey). On-call scheduling and insufficient advanced notification of schedule changes can be significant stressors for many workers in different sectors, but especially for low-wage workers.⁴³ An indicator of whether someone is a “low-wage worker” was included because it is a well-known risk factor for poor health.⁴⁴ The widely accepted definition of “low-wage work” (two thirds of the median hourly wage rate) was used for the United States ($12.39/hour in 2018).⁴⁵ Wage theft, including having tip money stolen or not being paid overtime, is a major contributor to low wages and income, which adversely affect worker health being linked to hypertension, diabetes, and obesity.⁴⁶ A four-part question on wage theft was developed by Meredith Minkler (UC Berkeley) and was included in the HWS (with permission) to measure this form of economic injustice.

We included one item on emotional demands (“Is your work emotionally demanding”) and one item on emotional labor. The job requirement to hide one’s feelings while performing work (“surface acting”) to manage or give care to the public or clients is an important component of emotional labor.⁴⁷ These items are not measured in the GSS-QWL but have a large literature showing a relationship to burnout⁴⁸ and were available in the COPSOQ II.

Electronic monitoring of workers is becoming increasingly common and may occur in a variety of ways, including the following: recording of employees’ telephone conversations, use of video cameras to monitor employees’ activities, monitoring the contents of employees’ emails, and electronically tracking the location of employees when using company vehicles (or hand-held scanners). The literature suggests electronic monitoring can be a significant stressor to workers who may feel pressured to meet performance goals to keep their jobs.⁴⁹ A form of job demands, electronic monitoring may cause workers to have to work faster and complete work tasks in ever decreasing durations of time. An item measuring electronic monitoring was included from a draft of the 2021 NHIS-OHS.

Freedom from physical and chemical hazards is a principle of healthy work; therefore, we consulted experts regarding the minimum number of physical factors that should be included while still maintaining the core goal of creating a psychosocial hazard assessment tool. Physical and chemical exposures (including vapors/gas/fumes, extreme temperatures, excessive noise, heavy lifting [50 lb]) were included but modified from the 2010 NHIS-OHS. Exposure to safety hazards, such as dangerous tools, equipment, or machinery, was also included from the Quality of Employment Survey (1977).⁵⁰ Physical violence and witnessing physical violence have become a growing concern in many sectors, especially among health care workers and those “front-line” jobs that have interactions with the general public, for example, flight attendants, food service/grocery workers, delivery workers.⁵¹ Items modified from the COPSOQ II that measure personally experiencing physical violence at work (assault) and the sources of that violence, as well as witnessing physical violence at the workplace and the sources of that violence, were added. In addition, we added a “source of sexual harassment” based on the modified COPSOQ II item to the GSS-QWL sexual harassment question.

Workplace bullying is also a major workplace stressor that has been shown to increase the risk of mental and physical illnesses, including depression and heart disease.^2,52 The Workplace Bullying Institute conducts a nationally representative survey⁵³ specific to workplace bullying in the United States and defines bullying as “repeated mistreatment: abusive conduct that is threatening, intimidating, humiliating, work sabotage, or verbal abuse.” We used this question after seeking permission so that individual and organization’s scores could be compared with a national score. We slightly modified the question from the WBI to create items on personally experiencing bullying using the definition above, witnessing bullying, and the sources of that bullying.

Organizational (procedural) justice has a large body of evidence linking it to various health outcomes⁵⁴ and is considered an important emerging stressor. A single item for procedural justice, the perceived fairness of applying policies and procedures, a strong component of organizational justice, was added.⁵⁵

Final Characteristics of the HWS (Version 1.0)

After determining the validity of the existing QWL-GSS items and conducting the various psychometric tests, reviewing other psychosocial questionnaires, and consulting with subject matter experts, we selected 55 questions to assess psychosocial work stressors and other working conditions (see Table 4, HWS Core Work Items).

TABLE 4.

Final HWS: Core Work Scales and Items (55 Questions)

Google Form #	Domains	Item Wording	Response Set	Sources
W1	Labor force status	(WRKSTAT) Last week were you working full time, part time, or what?	1. Working full time 2. Working part time 3. With a job, but not at work because of temporary illness, vacation, strike 4. Unemployed, laid off, looking for work 5. Other	GSS (modified)
W2	Paid work hours in your main job	(HRS1) How many hours did you work last week at your main job?	___ hrs	GSS-QWL (modified)
W3	Second job	Do you have another job(s)?	1. Yes 0. No	GSS-QWL (modified)
W4	Paid work hours in your second job(s)	If yes to question 3, how many hours did you work last week at your other job(s)?	___ hrs	GSS-QWL (modified for TotalWorkHrs)
W5	Mandatory overtime	(MUSTWORK) When you work extra hours on your main job, is it mandatory (required by your employer)?	1. Yes 2. No 0. Not applicable	GSS-QWL
W6	Work arrangement	(WRKTYPE) How would you describe your work arrangement in your main job?	1. I am a regular, permanent employee (standard work arrangement). 2. I work as an independent contractor, independent consultant, or freelance worker. 3. I am on-call and work only when called to work. 4. I am paid by a temporary agency. 5. I work for a contractor who provides workers and services to others under contract.	GSS-QWL
W7	Supervisor	Do you supervise others at work as part of your job?	1. Yes 0. No	GSS-QWL (Modified)
W8	How paid in main job	(WAYPAID) In your main job, how are you paid?	1. Salaried 2. Paid by the hour 3. Self-employed	GSS-QWL (modified)
W9	How paid in main job	In your main job, what are other ways in which you receive pay?	0. Not applicable 1. Tips 2. Commission/bonus 3. Piece work 4. Paid by the job 5. Paid by day 6. Paid by percent 7. Paid by contract	GSS-QWL (modified)
W10	Low wages	Is your hourly wage rate lower than $12.39? (Two thirds of the median hourly wage rate in US workers as of 2018 is considered “low wage.”)	1. Yes 2. No 0. Not applicable	Bateman R. Low Wage Workforce, Technical Appendix B. Brooking Institute, Nov 2019
W11	Wage theft	Have you experienced any of the following? (You can select more than one option.)	1. Been paid late or not paid at all (eg, time for training) 2. Did not receive extra pay for overtime 3. Receive less than minimum wage 4. Your boss takes part of your tip money 0. Not applicable/none of the above	Minkler et al. Am J Public Health. 2014;104(6):1010–1020
W12	Work schedule/shift work	(WRKSCHED) Which of the following best describes your usual work schedule?	1. Day shift 2. Afternoon shift 3. Night shift 4. Split shift 5. Irregular shift/on-call 6. Rotating shifts	GSS-QWL
W13	Working time arrangements	How are your working time arrangements set?	1. They are set by the company/organization with no possibility for changes. 2. You can choose between several fixed working schedules determined by the company/organization. 3. You can adapt your working hours within certain limits (eg, flextime). 4. Your working hours are entirely determined by yourself	Sixth European Working Conditions Survey
W14	Schedule changes to working time	Do changes to your working time arrangements occur regularly?	1. Yes 0. No
W15	Notice of schedule change	If yes to Q.14, how long before are you informed about these changes?	1. Several weeks in advance 2. Several days in advance 3. The day before 4. The same day 0. Other (spontaneous)
W16	Use of skills on the job (CONTROL)	(OPDEVEL) I have an opportunity to develop my own special abilities.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	GSS-QWL
W17	(CONTROL)	(WORKDIFF) I get to do a number of different things on my job.	1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree	GSS-QWL
W18	(CONTROL)	(MYSKILLS) My job lets me use my skills and abilities.	1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree	GSS-QWL
W19	Decision making opportunity (CONTROL)	(WKDECIDE) In your job, how often do you take part with others in making decisions that affect you?	1. Often 2. Sometimes 3. Rarely 4. Never	GSS-QWL
W20	(CONTROL)	(WKFREEDM) I am given a lot of freedom to decide how to do my own work.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	GSS-QWL
W21	N/A	(ALOTOFSAY) I have a lot of say about what happens on my job.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	Not included in final Job Control Scale
W22	Supervisor support	(SUPCARES) My supervisor is concerned with the welfare of those under him or her.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true 0. Not applicable	GSS-QWL
W23	(SUPPORT)	(SUPHELP) My supervisor is helpful to me in getting the job done.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true 0. Not applicable	GSS-QWL
W24	Coworker support	(COWRKHLP) The people I work with can be relied on when I need help.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true 0. Not applicable	GSS-QWL
W25	(SUPPORT)	(COWRKINT) The people I work with take a personal interest in me.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true 0. Not applicable	GSS-QWL
W26	Respect (REWARDS)	(RESPECT) At the place where I work, I am treated with respect.	1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree	GSS-QWL
W27	Promotion opportunities (REWARDS)	(PROMTEFR) Promotions are handled fairly.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true 0. Not applicable	GSS-QWL
W28	(REWARDS)	(PROMTEOK) The chances for promotion are good.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true 0. Not applicable	GSS-QWL
W29	Fair Earning (REWARDS)	(FAIREARN) How fair is what you earn on your job in comparison to others doing the same type of work you do?	1 Much more than you deserve 2 Somewhat more than you deserve 3 About as much as you deserve 4 Somewhat less than you deserve 5 Much less than you deserve	GSS-QWL
W30	Job Security (REWARDS)	(JOBSECOK) The job security is good.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	GSS-QWL
W31	Workload (DEMANDS)	(OVERWORK) I have too much work to do everything well.	1. Strongly disagree 2. Disagree 3. Agree 4. Strongly agree	GSS-QWL
W32	(DEMANDS)	(WRKTIME) I have enough time to get the job done.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	GSS-QWL
W33	Role conflict (DEMANDS)	(CONDEMND) I am free from the conflicting demands that other people make of me.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	GSS-QWL
W34	Emotional demands	Is your work emotionally demanding?	1. Never/hardly ever 2. Seldom 3. Sometimes 4. Often 5. Always 0. Not applicable	COPSOQ II (modified)
W35	Emotional labor	Does your work require that you hide your feelings?	1. Never/hardly ever 2. Seldom 3. Sometimes 4. Often 5. Always 0. Not applicable
W36	Electronic monitoring	In your workplace, how much do your supervisors use electronic monitoring to keep track of what you do?	1. Not at all 2. A little 3. Somewhat, but mostly they use other ways of keeping track of what I do 4. A lot 0. Don’t know/not applicable	NHIS-OHS 2021 (draft)
W37	Resource adequacy	(HLPEQUIP) I receive enough help and equipment to get the job done.	1. Very true 2. Somewhat true 3. Not too true 4. Not at all true	GSS-QWL
W38	(RESOURCE ADEQUACY)	(TOOFEWWK) How often are there not enough people or staff to get all the work done?	1. Never 2. Rarely 3. Sometimes 4. Often	GSS-QWL
W39	Work-family conflict	(WKVSFAM) How often do the demands of your job interfere with your family life?	1. Never 2. Rarely 3. Sometimes 4. Often	GSS-QWL
W40	(WFC)	(FAMVSWK) How often do the demands of your family interfere with your work on the job?	1. Never 2. Rarely 3. Sometimes 4. Often	GSS-QWL
W41	Physical effort	(PHYEFFRT) Please rate the overall physical effort at the job you normally do.	1. Very light 2. Fairly light 3. Somewhat hard 4. Hard 5. Very hard	GSS-QWL (2018)
W42 A-D	Exposure to physical hazards	In your main job, how often are you exposed to: a. Vapors, gas, dust, or fumes b. Extreme temps (hot/cold) c. Excessive noise d. Heavy lifting (>50 lb)	1. Never 2. Monthly 3. Weekly 4. Daily	NHIS-OHS 2010 (modified)
W43	Dangerous equipment/machinery	In your main job, how much of a problem are dangerous tools, machinery, or equipment for you?	0. Not applicable, do not operate dangerous equipment 1. No problem at all 2. Slight problem 3. Sizable problem 4. Great problem	Quality of Employment Survey, 1977 (modified)
W44	Safety climate	(SAFETYWK) The safety of workers is a high priority with management where I work.	1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 0. Not applicable	GSS-QWL
W45	(SAFETY CLIMATE)	(SAFEFRST) There are no significant compromises or shortcuts taken when worker safety is at stake.	1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree 0. Not applicable	GSS-QWL
W46-46a.47-47a	Physical violence (4 items)	Have you personally been exposed to [witnessed] physical violence (eg, been physically assaulted) at your workplace during the last 12 months? (2) Who was the source of the physical violence? (2)	1. Yes 0. No 1. Coworkers/colleagues 2. Manager/supervisor 3. Subordinates 4. Client/customer/patient/students 5. None of the above	COPSOQ (modified)
W48-48a 49-49a	Workplace bullying (4 items)	At work, have you personally experienced [witnessed] any of the following types of repeated mistreatment: abusive conduct that is threatening, intimidating, humiliating, work sabotage or verbal abuse? (2) Who was the source of this repeated mistreatment? (2)	1. Yes 0. No 1. Coworkers/colleagues 2. Manager/supervisor 3. Subordinates 4. Client/customer/patient/students 5. None of the above	2017 Workplace Bullying Institute US Workplace Bullying Survey (modified)
W50-50a	Sexual harassment (2 items)	In the last 12 months, were you sexually harassed by anyone while you were on the job? Who was the source of this sexual harassment?	1. Yes 0. No	GSS-QWL
			1. Coworkers/colleagues 2. Manager/supervisor 3. Subordinates 4. Client/customer/patient/students 5. None of the above	COPSOQ
W51	Discrimination	Do you feel in any way discriminated against on your job because of your:	1. Gender 2. Age 3. Race or ethnic origin 4. Religion or Country 5. Sexual orientation or gender identity 0. None of the above or not applicable	GSS-QWL (modified)
W52	Procedural justice	To what extent have the policies and procedures been applied consistently?	1. To a very large extent 2. To a large extent 3. To a moderate extent 4. To a small extent 5. To a very small extent 0. Not applicable	Colquitt (2001)
W53	Union representation	In my workplace I am represented by a union.	1. Yes 2. No 0. Don’t know	Modified GSS-QWL items on unions and benefits (by authors)
W54	Access to medical insurance and sick pay	Does your company (organization) offer:	1. Medical Insurance 2. Paid sick/family leave 0. None
W55	Open-ended question	Please comment briefly about parts of your job that you think were not addressed by these questions or if you have anything more you would like to add.		Authors

• A. GSS-QWL questions (34 questions)

There are 23 GSS-QWL questions that make up the multi-item scales and subscales; job demands (workload: two items, conflicting demands: one item), resource adequacy (two items), job control (decision authority, skill utilization: five items), supervisor and coworker support (four items), WFC (two items), rewards (respect, promotion, fair earnings, job insecurity: five items), and safety climate (two items).

There were also 11 GSS single items including the following: labor force status (full-time, part-time, unemployed, etc), work hours in main job, second job, mandatory overtime, work arrangements (eg, regular employee, independent contractor, on-call, temp worker), how paid, type of pay, work shift schedule, physical effort, sexual harassment, and discrimination.

• B. Non–GSS-QWL questions for missing work stressors (21 questions)

Twenty-one additional questions were included either from existing national questionnaires or from colleagues that were experts in their field. These included work time changes and schedule change notice (two items), low-wage work (is your hourly wage rate lower than $12.39?) (one item), wage theft (one item, four parts), and an additional work hours question (one item) for those reporting a second job (otherwise, it could be skipped). This question could then be added with the work hours (main job) question for a more accurate “total work hours” variable. Emotional demands and emotional labor (two items) and electronic monitoring (one item) were also included. Physical exposures (including vapors/gas/fumes, extreme temperatures, excessive noise, heavy lifting [>50 lb]) (one item, four parts), exposure to safety hazards (dangerous equipment) (one item), exposure to physical violence (personal experience, witnessing, and sources of violence) (four items), and workplace bullying (personal experience, witnessing, and sources) (four items). The questions on sources of physical violence, workplace bullying, and sexual harassment are only available to answer if someone answers “yes” to the items on personal experience or witnessing (an optional two part question). Owing to the relatively small number of people who typically answer yes to these questions, we felt the additional items on “source” were justified.

An item measuring procedural justice was also added (one item). Although there is a GSS-QWL question regarding union representation, it asked more about how a respondent felt about their union rather than just whether they were represented by a union or not. We added one item “In my workplace, I am represented by a union” that would allow stratification of a workplace by whether they are unionized or not. In addition, one item was created by the authors to measure whether a respondent’s company or organization offers medical benefits or sick leave, since inadequate access to health care or sick pay can be a significant stressor.⁵⁶ Lastly, we added an open-ended comment box.

• C. HWS Supplemental Questions (Appendix B, http://links.lww.com/JOM/B306)—health and productivity outcomes (16 questions), COVID-19 impacts (16 questions), and sociodemographics (nine questions)

A “Supplemental” section on health outcomes, quality of life, and productivity is included in the HWS along with the core psychosocial and work organization questions (16 items). There were 13 health items selected from the GSS-QWL, including three questions from the well-validated, CDC Healthy Days Core Module (CDC HRQOL-4).⁴¹ The general health question was dichotomized at “poor/fair” health versus “excellent, very good, good.” The two items on the number of unhealthy days in the past 30 days were dichotomized at “14 or more days.” The CDC activity limitations question was not included in the final version of the survey due to its tendency to measure extreme levels of functional disability, which is very uncommon in working age populations and has a small prevalence (3.5%). Ten other GSS-QWL health questions were also included. National GSS-QWL data provided percent of the US population exposed to these health problems to compare with the % exposed in an organization. Three non-GSS questions were also added to the supplemental health section. “Stress at home” (one item) was worded by the authors similarly to the QWL item “How often do you find your work stressful?” question. Two items were added from the Work Limitations Questionnaire that measure “presenteeism” or the extent to which physical or mental health problems may interfere with work performance.⁵⁷

The launch of the HWS coincided with the global pandemic; therefore, an optional section on the impacts of COVID-19 on work, health, and safety was added (16 items). These questions were borrowed with permission from colleagues in Canada who developed questions for a national survey of health care workers and other occupations in Canada during the pandemic.⁵⁸

Nine items were added to collect basic sociodemographic data including age (ranges), gender, race/ethnicity, marital status, education, household income (optional), workplace location (state), industry, and occupation (ICD classifications).

National Distributions and Risk Ranges for the Selected Core Work QWL Scales

Table 5 shows the averages for the key work stressor domains confirmed in the psychometric testing, calculated from the GSS-QWL data set (2002–2018). Based on the distribution of these multi-item scales, tertile cut points were determined to be the most appropriate for grouping the population into distinct “risk” groups: low risk, intermediate risk, and high risk. Approximately 20%–35% of the population were classified into each of the three risk groups, except for “low coworker support” and “low safety climate,” which had only 15% and 11%, respectively, in the high-risk groups. In the case of “safety climate,” this was probably due to the high incidence of individuals in the QWL who reported they agreed or strongly agreed that management prioritized health and safety of workers and did not take “short cuts.” In the case of coworker support, a high percent of the QWL population agreed or strongly agreed that coworkers took a personal interest in each other and that they could be relied on. These averages and risk groups were ultimately used in an algorithm that helps to prioritize an organization’s survey results.

TABLE 5.

Three-Group Risk Classification of Multi-Item QWL Scales

Scale	Average	Low Risk	Intermediate Risk	High Risk
Low use of skills on the job (3 items)	5.2924	3–4 (34.1%)	5–6	7–12 (20.3%)
Low decision-making opportunities (2 items)	3.5244	2 (25.9%)	3–4	5–8 (21.5%)
Low job control (5 items)	8.8132	5–7 (33.8%)	8–9	10–20 (35.2%)
Low supervisor support (2 items)	3.4084	2 (38.2%)	3–4	5–8 (19.1%)
Low coworker support (2 items)	3.3496	2 (31.5%)	3–4	5–8 (15.0%)
Low promotion opportunities (2 items)	4.4631	2–3 (29.9%)	4–5	6–8 (25.2%)
Low reward (5 items)	11.2235	5–9 (28.0%)	10–12	13–21 (28.1%)
High workload (2 items)	4.0404	2–3 (38.7%)	4	5–8 (29.6%)
High psychological demand (3 items)	6.1160	3–5 (39.8%)	6–7	8–12 (19.7%)
Low resource adequacy (2 items)	4.4161	2–3 (28.6%)	4–5	6–8 (21.6%)
High work-family conflict (2 items)	4.3219	2–3 (29.0%)	4–5	6–8 (25.8%)
Low safety climate (2 items)	3.3483	2 (35.6%)	3–4	5–8 (11.4%)

Developing and Piloting an Online Questionnaire and Report System

• A. Online reporting system

The HWS was converted into an online version using Google Forms and Google Data Studio, which transforms the collected survey data into an automated report for either individuals or organizations (aggregated). No identifiers are collected in the survey so the reports are completely anonymous. Organizations do not receive raw data and cannot connect data to individuals.

The HWS for individuals is available online⁵⁹ and allows individuals to click a link to the online survey and complete it independently at any time. Individuals who choose to complete the survey may elect to receive an automatic, confidential report of their personal results by providing an email address. Once the survey is completed, an individualized report (see Appendix C, http://links.lww.com/JOM/B307) can be emailed directly to that individual, which summarizes the level of their work stressors compared with national averages from the QWL. This option is also available to individuals who complete the HWS as part of an organization.

An organization may “request access” to the HWS by filling out a short form (available for “employers” or “unions and worker advocates”)⁶⁰ with contact information for a designated representative. Once received, a unique ID for the HWS is generated for that organization. The organization is provided an email template and the unique link that they can then email to employees. Once enough individuals submit the survey as part of an organization (recommended between 50% and 80%, depending on the size of the organization), data are saved in a spreadsheet. The spreadsheet data are linked to a report template created in Google Data Studio, and a report can be generated just for that organization. The automated report can be generated and emailed to the organization’s designated representative within 24–48 hours (see Appendix D, http://links.lww.com/JOM/B308).

The HWS organizational report automatically generates group averages for the multi-item work stressor scales and compares the organization’s averages with national distributions from the same QWL scales. In addition, an algorithm was created to “red flag” an organization’s work stressors if (a) the group average is higher than the national average, and (b) if the organization’s percent of workers at “high risk” is greater than the national percent at high risk. The report also summarizes single items from the HWS in the following areas: workplace injustice (bullying, sexual harassment, feelings of discrimination), physical hazards, work arrangements, precarious employment, COVID-19 impacts, health outcomes, and sociodemographics, using pie charts, simple percentages based on dummy variables, and tables. Explanations for the major work stressors and links to a range of tools, “solutions and strategies” available on the HWC Tools pages,⁶¹ are provided in the report so organizations have guidance on how to pursue improvements in work organization.

• B. Expert feedback from stakeholders and pilot testing the HWS system

In phase 4 of the HWS project, occupational health experts familiar with psychosocial stressors and work organization were contacted to provide feedback about the content of a draft version of the HWS. Since the selected questions, all came from previously validated questionnaires that have been used in multiple working populations, we believed it was unnecessary to do cognitive testing on the QWL items. Also, since we planned on using the current QWL items as a benchmark for organizations to compare with, it was necessary for psychometric validity to leave the QWL question wording the same. However, in January 2020, a panel of experts was asked to review the content of the HWS 1.0 using a version designed in Survey Monkey, for content wording and accessibility.

1. Do you think we missed any important domains/scales or items?

2. Do you know of any better items for the “new” items that we have included on top of the core QWL items?

3. Are there any items/scales that you think may not really be essential for identifying key work stressors that we could do without to shorten the survey further?

4. Any other general thoughts you have about the HWS or the project as a whole that might be helpful as we move toward the analysis and the computerized report and beta testing in the field with workers?

Four experts on psychosocial work stress questionnaires responded, and where possible, the order of questions and the wording of non GSS-QWL questions were improved. For example, we separated the exposure to physical violence, bullying, and sexual harassment questions into two questions, personally experienced or witnessed and also added the “source” (eg, client, supervisor, coworker, etc) to each of these questions. We changed the wording to describe emotional labor and added wording to the organizational justice question to clarify it.

In December 2020, 29 occupational health and safety researchers and experts from labor, employer consulting, and the scientific community and workers were emailed a link to the final HWS Google Form. They were asked to complete the anonymous HWS for individuals, receive the automated report for individuals, and complete a brief questionnaire (data available upon request). Thirteen responded, and eight completed the HWS and received the report (see Appendix E for summary, http://links.lww.com/JOM/B309). The majority (six of eight) agreed that we had achieved a short, validated work organization assessment tool and that the content would apply to a wide range of sectors and occupations. Seven of eight responded that they would or “maybe would” use this tool in their occupational role. However, five of eight mentioned the linked information in the individual report explaining that the findings needed some improvement, including that individuals might need further in-person contact to learn how to utilize these findings. In response, short videos geared to individuals with information about the HWS have been produced, and others are planned in lieu of one-on-one consulting, as this is beyond the resources of the HWC.

Finally, in August 2021, the organizational report system that automatically calculates aggregate scores after completion of the HWS by participants of an organization was completed. The system was field-tested by the study authors. Coding of all variables used in the data source that is the basis of the report in Google Data Studio was cross-checked, and test reports were generated and reviewed by the study authors. Also in August 2021, a labor union representing 100 members from one employer in California pilot-tested the entire system, successfully collecting 51 surveys and receiving the HWS report for organizations. Within a few days of closing the survey, we generated results for the union and presented the report to the president of the local and two stewards. We were able to discuss the ease of the survey process and the clarity of results represented in the report. While discussing the survey results with the president and stewards, we were able to improve the aggregate group-level report by adding a cover page that summarized the major findings in each section (see Appendix D, http://links.lww.com/JOM/B308). The stewards said that the online survey had been very easy to use and understandable. Within a week or so, we joined the local union president and stewards to present the results to management within a labor-management committee structure, which were also well received by those at the meeting. We intend to summarize these findings and those of other organizations that have used the survey in future research articles. As of August 2022, 37 organizations have requested access to use the HWS, six have completed, and others are in various stages of readiness to distribute the survey to participants. Refinement of the report and survey system, based on feedback from organizations, will continue.

DISCUSSION

Psychometric Validation of the HWS

The HWS was developed systematically by experts evaluating the content validity of the NIOSH QWL questionnaire items based on the major theoretical models of work-related psychosocial stressors and their subscales, including the following: the Karasek’s Job Demands-Control and Support model, Siegrist’s Effort-Reward Imbalance model, WFC/spillover theory, and organizational injustice (discrimination/harassment). NIOSH has not previously constructed work stressor scales from QWL items. The psychometric analysis of the selected QWL items and scales that were performed provided the shortest possible scales while maintaining the validity of the underlying constructs measured by the scales. Some scales did not meet reliability criteria (0.60 and above) and were reduced to single item measures or grouped into another scale (eg, job insecurity became part of the “rewards” scale, physical demands became a single item, discrimination and harassment questions were used as single items because of low reliability as a multivariate scale).

Based on a literature review and consultation with OSH experts, 21 items measuring several relatively recent, but accepted, psychosocial stressors (emotional demands/labor, organizational justice, workplace bullying, and precarious work arrangements, scheduling) were added to the 34 QWL items to give a final HWS questionnaire of 55 core work items. The additional items are not multi-item scales, so we are not able to test them for internal consistency, reliability, or predictive validity; however, they are from widely used questionnaires. It is possible, as we continue to collect survey data, that some of these items may be grouped together into a multi-item construct, such as “precarity,” “emotional labor,” or “injustice” for testing and validation in the future.

Although the core of the HWS is focused on psychosocial work stressors and work organization, we also added “Supplemental” sections to the HWS that measure COVID-19–related impacts on work and self-reported health. Most of the self-reported health items are derived from the GSS, so the HWS report provides an organization with comparison of their organization’s health outcomes to US workers. We also added nine sociodemographic items so that a workplace could compare their survey respondents to the entire organization to better confirm sample generalizability.

Final Version of the Online HWS: Pros and Cons

We were successful in creating a short, standardized questionnaire to assess workplace psychosocial hazards. The final version of the HWS has 55 core work-related items. When field tested, the online survey could be completed within 15 minutes and within 20 minutes when including the supplemental COVID-19 questions, as well as self-reported health and sociodemographics questions. We anticipate that this will encourage workplaces to participate by reducing the time burden for completion of the survey.

At this time, the HWS cannot be tailored based on industry or occupation-specific factors (eg, for educators, health care workers, or manufacturing workers); that kind of flexibility was sacrificed for the benefit of providing automatic, standardized results. The QWL questionnaire scales and other variables are coded to calculate comparisons to national distributions, and to determine risk ranges, so it is not possible to easily revise, remove, or add questions without significantly interfering with the report system.

Future Aims

We plan to return to the HWS annually and determine if changes need to take place to improve the quality of the survey, remove questions that might show little variability, or add questions that might be needed based on feedback during meetings with workers, union representatives, or employers. We also intend to conduct further quantitative analysis based on the new data we have collected over the last 12 months since the launch of the HWS and will seek publication of those results. In addition, we continue to request anonymous feedback using two open-ended questions available at the end of the survey since its launch, to better understand the user experience. However, we will also conduct an active investigation soliciting feedback from individual users, who have agreed to have us contact them in the future, about their experience with the survey. We will conduct a qualitative analysis including interviews with representatives of the organizations that have utilized the HWS about their experience conducting the survey and reviewing the report. Also, we will send an anonymous feedback survey to those organizations that requested access to the survey but have not yet chosen to use the survey regarding reasons for hesitancy.

Encouraging “Next Steps”: Workplace Intervention

The major goals of the Healthy Work Campaign are to raise awareness about the health damaging effects of work stressors and to encourage work organization improvements. We see the HWS and the automated report of results as educational tools and “interventions” in themselves. We continue outreach to encourage the dissemination of the HWS, other tools, and resources to encourage workplace change.

Although the report does display statistics such as “averages” and percentages, it also automatically calculates the “level of risk” of an organization and the percent of people at high risk. If both are high (average and % of people), the stressors are “red flagged.” Thus, we believe the red flags provide a visual way of prioritizing the problems and translating the findings fairly easily. However, in addition to providing the survey as a free service of the nonprofit Center for Social Epidemiology, we also meet with organizations (labor unions, businesses, and nonprofits) or provide webinars and short videos to educate them about the use of the survey and also to follow-up with them once they complete the survey to help interpret the results and encourage employee input. Enlisting employee participation (and their representatives) and management buy-in, or collective bargaining, is essential to implementing effective workplace changes to reduce psychosocial stressors and improve worker health.^62,63

To help promote workplace interventions, we also provide tools pages for individuals, unions, and employers on the Healthy Work Campaign Web site with resources and steps to follow. We have also published on the Web site over 40 case reports of various interventions. These reports include examples from “workplace research, policies, and programs,” “laws and regulations,” and “collective bargaining” of successful “healthy work strategies” that reduce psychosocial work stressors and/or improve worker health.⁶⁴ It is an essential goal to encourage organizations to not just “diagnose” work stressors, but also to take action to improve work organization and health.

CONCLUSIONS

The HWS is a reliable, valid tool for measuring work-related psychosocial stressors and work organization. We have successfully validated several common workplace psychosocial stressor scales based on items available in the NIOSH QWL questionnaire. This allows organizations to compare their scores on psychosocial factors, such as job demands/workload, job control, social support, work-life balance, rewards, and safety climate, to the US national employed population. The HWS provides a comprehensive assessment of work-related psychosocial and other work organization factors that are known contributors to poor health and well-being, for both organizations and individuals. This instrument is the first of its kind in the United States that is free, online, and anonymous; provides automatic reports of results; and can be used autonomously by organizations seeking to create healthier working conditions and improve the health of working people.