Development of a Short Version of the New Brief Job Stress Questionnaire

Abstract

This study was aimed to investigate the test-retest reliability and validity of a short version of the New Brief Job Stress Questionnaire (New BJSQ) whose scales have one item selected from a standard version. Based on the results from an anonymous web-based questionnaire of occupational health staffs and personnel/labor staffs, we selected higher-priority scales from the standard version. After selecting one item with highest item-total correlation coefficient from each scale, a 23-item questionnaire was developed. A nationally representative survey was administered to Japanese employees (n=1,633) to examine test-retest reliability and validity. Most scales (or items) showed modest but adequate levels of test-retest reliability (r>0.50). Furthermore, job demands and job resources scales (or items) were associated with mental and physical stress reactions while job resources scales (or items) were also associated with positive outcomes. These findings provided a piece of evidence that the short version of the New BJSQ is reliable and valid.

In Japan, the number of workers with mental health problems is increasing¹⁾ and primary prevention of mental health problems is a high priority for employers as well as employees. Previous studies have shown that “assessing and improving work environment” effectively reduces mental health problems^{2, 3)} and thus the Brief Job Stress Questionnaire (BJSQ)⁴⁾ was developed to assess work environment in Japan.

However, considering increasing needs to include a more extensive set of psychosocial factors at work (e.g., effort-reward imbalance [ERI], organizational justice, and work-family interface) and positive mental health outcomes (e.g., work engagement) in the assessment of psychosocial work environment^{5, 6)}, we recently developed the new version of the Brief Job Stress Questionnaire (New BJSQ) standard version and reported its reliability and validity⁷⁾. The New BJSQ can assess job demands and job resources as well as employee and organizational outcomes multidimensionally and comprehensively by adding its scales/items to the current BJSQ. However, because the New BJSQ has 30 scales and 84 items (49 scales and 141 items in total when combined with the current BJSQ), it would be burdensome to use it in practice. To cope with this dilemma, we developed a short version of the New BJSQ whose scales have one or two item(s) selected from the standard version. The purpose of the present study was to investigate the test-retest reliability and validity of the short version of the New BJSQ.

Prior to the development of the short version, occupational health staffs and personnel/labor staffs who participated in conferences on occupational health (e.g., Annual Meeting of the Japan Society for Occupational Health) were invited to complete an anonymous web-based questionnaire in May 2010. The questionnaire asked participants to choose “important scale(s)” and “unnecessary or hard-to-use scale(s)” from the New BJSQ scales (multiple answers were possible). Based on 103 valid responses, we selected 22 higher-priority scales from 30 scales of the standard version. These were “emotional demands”, “role conflict”, and “work-self balance (negative)” classified as “job demands” (three scales); “role clarity” and “career opportunity” classified as “task-level job resources” (two scales); “monetary/status reward”, “esteem reward”, “job security”, “leadership”, “interactional justice”, “workplace where people compliment each other”, and “workplace where mistakes are acceptable” classified as “workgroup-level job resources” (seven scales); “trust with management”, “preparedness for change”, “respect for individuals”, “fair personnel evaluation”, “diversity”, “career development”, and “work-self balance (positive)” classified as “organizational-level job resources” (seven scales); and “workplace harassment”, “workplace social capital”, and “work engagement” classified as “outcomes” (three scales). For “workplace harassment” and “workplace social capital”, they were initially classified as “job demands” and “workgroup-level job resources”, respectively. However, according to a series of stakeholder meetings, which were held twice a year attended by researchers from five institutes/departments of occupational safety and health, occupational health staffs (physicians, nurses, and hygienists), and representatives of two employer associations and one employee association, they were finally classified as “outcomes” of the job demands and resources. The decision was made because in the stakeholder meetings, a workplace with greater social capital and without workplace harassment was considered as one of the current business goals of a company and also because the workplace social capital scale seemed to measure employees’ evaluation of connectedness in a workplace, which was considered to be an outcome of social capital rather than of work environment⁸⁾.

For the 15 scales comprising three or more items, the selection of items for the short version was based on item-total correlation coefficients (ITCs) calculated for each scale. One item with highest ITC was selected from each scale (data available upon request). There were few exceptions. For the role clarity scale, the ITC was lower for the selected item (“knowing own role and responsibility”) (0.478) than that for another item (“knowing how much authority I have”) (0.481) among men. However, the ITC was better for selected item (0.453) than that for the other item (0.380) among women. We thus selected the former item. For the diversity scale, the ITC was lower for the selected item (“good for workers with different employment positions”) (0.504) compared to another item (“good for younger workers”) (0.513). However, considering a current increasing concern about non-regular or precarious employment^{9, 10)}, we selected the former item. The average ITC for the new 15 single-item scales of the short version was 0.693, ranging from 0.478 to 0.882. For six of the seven two-item scales, i.e., work-self balance (negative), monetary/status reward, esteem reward, workplace where mistakes are acceptable, work-self balance (positive), and workplace harassment, one of the two items was selected for the short version based on a discussion among the authors, considering its content validity, representativeness of the items (i.e., applicable to most situations), and comprehensibility. We decided to keep two items to measure work engagement in the short version because the two items measure different dimensions of this concept (i.e., vigor and dedication). Through the item reduction described above, we fixed a 23-item short version of the New BJSQ on a four-point response option: 1 = Definitely, 2 = Somewhat so, 3 = Not exactly, and 4 = Not at all (Table 1). These items are available at http://jstress.net (only in Japanese language).

Table 1. Averages (and standard deviations [SDs]), correlation with a standard version, and one-year test-retest of the short version of the New BJSQ obtained from a nationally representative survey of employees of Japan in 2010/2011 †.

Scales ‡	No. of items	Average (SD)	Correlation with a standard version (Polycholic correlation coefficient) n=1,606–1,626	One-year test-retest (Pearson correlation coefficient) n=385–389
Job demands
1. Emotional demands	1	2.66 (0.96)	0.962***	0.566***
2. Role conflict	1	2.87 (0.93)	0.916***	0.549***
3. Work-self balance (negative)	1	2.83 (0.89)	0.976***	0.576***
Task-level job resources
4. Role clarity	1	3.41 (0.63)	0.819***	0.343***
5. Career opportunity	1	2.62 (0.94)	0.947***	0.623***
Workgroup-level job resources
6. Monetary/status reward	1	2.25 (0.92)	0.956***	0.633***
7. Esteem reward	1	2.59 (0.80)	0.960***	0.551***
8. Job security	1	2.84 (1.02)	0.870***	0.620***
9. Leadership	1	2.25 (0.92)	0.930***	0.549***
10. Interactional justice	1	2.65 (0.86)	0.974***	0.516***
11. Workplace where people compliment each other	1	2.59 (0.91)	0.987***	0.546***
12. Workplace where mistakes are acceptable	1	2.45 (0.85)	0.957***	0.562***
Organizational-level job resources
13. Trust with management	1	2.58 (0.81)	0.948***	0.553***
14. Preparedness for change	1	2.35 (0.86)	0.913***	0.439***
15. Respect for individuals	1	2.14 (0.82)	0.939***	0.587***
16. Fair personnel evaluation	1	2.04 (0.86)	0.935***	0.524***
17. Diversity	1	2.72 (0.87)	0.831***	0.515***
18. Career development	1	2.23 (0.87)	0.899***	0.613***
19. Work-self balance (positive)	1	2.07 (0.87)	0.960***	0.578***
Outcomes
20. Workplace harassment	1	3.70 (0.63)	0.941***	0.428***
21. Workplace social capital	1	2.66 (0.77)	0.959***	0.509***
22. Work engagement	2	2.52 (0.77)	SS	0.664***

*** p<0.001. † The number of respondents varied from 1,606 to 1,628 because of missing values. ‡ Each scale (or item) score was converted so that the higher score indicates better state and ranges from 1 to 4. See text for more details on scoring. SS: Same items as a standard version.

To examine the test-retest reliability and validity and obtain normative scores of the short version of the New BJSQ, cross-sectional and one-year prospective studies were conducted of a nationally representative sample of Japanese employees. In November 2010, a self-administered questionnaire, including scales on demographic characteristics and all scales of the current BJSQ and the New BJSQ, was mailed to 5,000 Japanese people aged 20–60 yr selected by a two-stage random sampling. By February 2011, we received 2,400 completed questionnaires, of which 2,384 were valid (response rate, 47.7%). Among them, 1,633 respondents (847 men and 786 women) answered “I am an employee and on the payroll of a company” in the questionnaire. Out of these 1,633 employed respondents, 479 agreed to participate in a follow-up survey. In November 2011, the same questionnaires were sent to these participants and 417 questionnaires (202 men and 215 women) were returned by December 2011 (response rate, 87.1%). Detailed sampling methods and demographic characteristics of participants are shown in elsewhere⁷⁾. The Ethics Committee of the Graduate School of Medicine/Faculty of Medicine, The University of Tokyo reviewed and approved aims, designs, and procedures of the present study (No. 2953).

Based on the baseline cross-sectional data (1,633 employees), a national average and standard deviation of each scale (or item) for the short version of the New BJSQ were calculated. For the work engagement scale, instead of calculating a scale score as a sum of the item scores, a scale score was calculated as an average item score ranging from 1 to 4. Each scale (or item) score was converted so that higher scores indicated better status (e.g., a higher score of job demands means lower job demands and a higher score of workplace harassment means low level of workplace harassment; on the other hand, a higher score of job resources means higher job resources), which allowed us to standardize averages and ranges of scores across scales (or items) and to interpret scale (or item) scores easier, making the comparison of the scale (or item) scores more convenient.

Polychoric correlation coefficients between scales (or items) of the short version and scales of the standard version were calculated to examine validity of the short version. Based on the data from 417 respondents who completed the one-year follow-up questionnaire, Pearson correlation coefficients were calculated to evaluate one-year test-retest reliability. In consideration of the influence of changes in working life (e.g., promotion, transfer, change in contents of work, change in employment status, job change, etc.) during the follow-up period on the test-retest reliability, the partial correlation coefficients adjusted for experience of changes in working life during the follow-up period, which was obtained from the self-administered questionnaire, were also calculated. Furthermore, as sub-analyses, the correlation coefficients only among those who did not experience any changes in working life during the follow-up period (n=248) were also calculated. For these correlation analyses, a pair-wise deletion of cases, rather than list-wise deletion, was used when items had a missing response. Furthermore, to examine whether the data fit the Job Demands-Resources (JD-R) model¹¹⁾, which assumes that job demands predict negative emotional reactions (such as psychological stress) while job resources, including task-level, workgroup-level, and organizational-level, predict both negative and positive emotional reactions (such as work engagement), polychoric correlation coefficients were calculated between 19 scales (or items) of psychosocial work environment and selected outcomes (i.e., psychological and physical stress reactions, work engagement, workplace social capital, and workplace harassment) using 1,398 respondents who completed all scales at baseline. The level of significance was 0.05 (two-tailed). All the analyses were conducted using the IBM SPSS Statistics version 19.

As a result, for the nationally representative sample of 1,633 employees, average scores for most scales (or items) of the short version of the New BJSQ fell between 2.0 and 3.0 (Table 1). The average score was higher for workplace harassment (3.70) and role clarity (3.41) and lower for respect for individuals, fair personnel evaluation, and work-self balance (positive) (2.04–2.14). More detailed information on the national average scores by gender, occupation, employment type, and industry is available at http://www.jstress.net (only in Japanese language). Polychoric correlation analyses showed that all scales (or items) in the short version correlated highly with scales of the standard version (r>0.80) (Table 1). Among 417 employees who completed the one-year follow-up questionnaire, one-year test-retest reliability as measured by Pearson correlation coefficient was over 0.50 for most scales (or items). Almost same correlation coefficients were observed after adjusting for experience of changes in working life during the follow-up period and excluding those who experienced changes in working life during the follow-up period (data available upon request). Furthermore, Polychoric correlation coefficients between psychosocial work environments and outcomes showed that job demands scales (or items) correlated highly with psychological and physical stress reactions; but modestly with work engagement and workplace social capital (Table 2). Job resources scales (or items) correlated with psychological and physical stress reactions to a similar extent, while these scales (or items) correlated with work engagement and workplace social capital more strongly than did job demands. However, monetary/status reward and job security had relatively weaker correlations (r=0.264 and 0.181, respectively) with work engagement among the job resources scales (or items).

Table 2. Polychoric correlation coefficients between psychosocial work environment (job demands and job resources) and outcomes measured by the current BJSQ and the short version of the New BJSQ scales: a national representative sample of employees of Japan in 2010/2011 †.

Scales ‡	Psychological stress reactions	Physical stress reactions	Work engagement	Workplace social capital	Workplace harassment
Job demands
1. Emotional demands	0.530**	0.331**	0.164**	0.200**	0.379**
2. Role conflict	0.448**	0.286**	0.195**	0.376**	0.420**
3. Work-self balance (negative)	0.503**	0.305**	0.167**	0.226**	0.298**
Job resources: task-level
4. Role clarity	0.156**	0.047	0.329**	0.278**	0.130**
5. Career opportunity	0.329**	0.162**	0.594**	0.402**	0.158**
Job resources: workgroup-level
6. Monetary/status reward	0.317**	0.243**	0.264**	0.378**	0.173**
7. Esteem reward	0.370**	0.224**	0.429**	0.454**	0.302**
8. Job security	0.237**	0.154**	0.181**	0.239**	0.215**
9. Leadership	0.293**	0.149**	0.449**	0.471**	0.222**
10. Interactional justice	0.375**	0.209**	0.423**	0.484**	0.340**
11. Workplace where people compliment each other	0.326**	0.189**	0.429**	0.437**	0.301**
12. Workplace where mistakes are acceptable	0.314**	0.180**	0.413**	0.414**	0.256**
Job resources: organizational-level
13. Trust with management	0.358**	0.207**	0.391**	0.517**	0.314**
14. Preparedness for change	0.292**	0.153**	0.365**	0.465**	0.207**
15. Respect for individuals	0.377**	0.237**	0.506**	0.536**	0.275**
16. Fair personnel evaluation	0.291**	0.194**	0.359**	0.444**	0.196**
17. Diversity	0.269**	0.142**	0.353**	0.414**	0.207**
18. Career development	0.301**	0.176**	0.489**	0.513**	0.194**
19. Work-self balance (positive)	0.486**	0.259**	0.677**	0.435**	0.204**

** p<0.01. No asterisk means p>0.05. † Based on data from 1,398 respondents who completed all the scales at baseline. ‡ Each scale (or item) score was converted so that the higher score indicates better state. See text for more details on scoring.

In the present study, we developed the short version of the New BJSQ, which can assess a higher-priority extensive set of job demands, job resources, and outcomes more briefly, by adding scales (or items) to the current version of the BJSQ. For the one-year test-retest reliability, Pearson correlation coefficients were over 0.50 for most scales (or items), which is similar to one-year test-retest correlations (r=0.47 to 0.66) previously reported for job demands, decision authority, and social and management support¹²⁾. These levels of test-retest reliability are considered to be adequate for a group comparison purpose^{13, 14)}. Correlation analyses showed that all scales (or items) in the short version were highly associated with scales of the standard version. Furthermore, job demands and job resources scales (or items) were associated with psychological and physical stress reactions while job resources scales (or items) were also associated with positive outcomes, i.e., work engagement and workplace social capital, which is consistent with the theoretical framework of the JD-R model¹¹⁾. However, monetary/status reward and job security had relatively weaker associations with work engagement among the job resources scales (or items). This finding may be explained by a traditional two-factor theory, sometimes known as Herzberg’s motivation-hygiene theory¹⁵⁾. In this theory, hygiene factors including status, job security, and salary do not provide positive satisfaction, though dissatisfaction results from their absence, which may lead to weaker associations of monetary/status reward and job security with work engagement. The present findings provided a piece of evidence that the short version of the New BJSQ is reliable and valid and fit expectations from the JD-R model.

The important feature of the short version of the New BJSQ is that users can assess the higher-priority set of psychosocial work environment and its employee and organizational outcomes by only 80 items if combined with the current 57-item BJSQ, which may be easy-to-use in practice. However, because the short version may not have sufficient measurement accuracy, it may not be suitable for scientific research. The standard version may be more desirable for use in scientific research rather than the short version.

The present study has some limitations that should be considered. Among the most important limitation is that each scale (or item) of the short version of the New BJSQ showed only modest test-retest reliability while it is adequate levels for a group comparison purpose^{13, 14)}. Although we also calculated partial correlation coefficients adjusted for experience of changes in working life during the follow-up period, the “magnitude” and “timing” of the changes in working life were not obtained in the present study. The lack of this information might make it difficult to investigate test-retest reliability. Further review of the scale items and more accurate assessment of test-retest reliability should be conducted to achieve higher measurement accuracy. Although the short version of the New BJSQ remains some limitations, it can assess a set of psychosocial factors at work comprehensively and briefly. Because of its limited burden for respondents, the short version of the New BJSQ would be useful to occupational health staffs as well as researchers interested in assessment and improvement of psychosocial work environment. To confirm more detailed validity of the short version of the New BJSQ, convergent and discriminant validities using other reliable and valid measurements should be investigated in a future study.