Assessing the psychometric properties of the Guarding Minds @ Work questionnaire recommended in the Canadian Standard for Psychological Health and Safety in the Workplace

Objectives

This study examines the item and dimension distribution and factorial reliability and validity of the GM@W questionnaire for assessing the 13 dimensions of the work environment outlined in The Canadian National Standard for Psychological Health and Safety in the Workplace (The Standard). Methods An internet survey of 1,006 Ontario workers was conducted between February 10th and March 5th, 2020. Respondents had to be employed in a workplace with five or more employees. The survey included the 65 items from the GM@W questionnaire, and questions to assess sociodemographic characteristics and employment arrangements. Analyses examined the distribution of scores for items and for overall dimensions. A confirmatory factor analysis (CFA) examined the relationship between the 13 proposed dimensions and each of the 65 questions, using only respondents with complete information (N = 900). Results Low levels of missing responses were observed, although 14 of the 65 items had potential ceiling effects. CFA analyses demonstrated poor fit for the conceptual model linking the 13 dimensions of The Standard to the 65-items. High correlations between dimensions were also noted. The GM@W questionnaire displayed poor discriminant in measuring the specific dimensions proposed in The Standard. Conclusions Our results suggest the GM@W survey is unable to isolate the proposed dimensions of the psychosocial work environment as outlined in The Standard. These limitations are important, as workplaces using the GM@W survey will not be able to identify dimensions of the work environment which require attention or assess changes in particular dimensions over time.

Introduction

The psychosocial working environment has been recognised as an important determinant of health (Landsbergis 2010; Smith et al. 2008). In particular in relation to mental health, aspects of work such as work demands, autonomy, ability to use skills, and organisational justice have been associated with various mental health outcomes (Harvey et al. 2017; Stansfeld and Candy 2006). In 2013, to help address the relationships between the psychosocial work environment and metal health, the Mental Health Commission of Canada and the Canadian Standards Association released the National Workplace Psychological Health and Safety Standard (The Standard) (Canadian Standards Association 2013). The objective of The Standard was to outline the steps workplaces could follow to 'develop and sustain a psychologically healthy and safe workplace' (Canadian Standards Association 2013). To do this The Standard provides guidance on how to identify and reduce (or control) exposures at work that may be associated with mental health risks; and how to implement practices, structures and workplace cultures that support and promote psychological health and safety in the workplace (Canadian Standards Association 2013).

The Standard proposes 13 dimensions of the work environment which have the potential to impact worker mental health. These are: psychological support; organisational culture; clear leadership and expectations; civility and respect; psychological job demands; growth and development; recognition and reward; involvement and influence; workload management; engagement; work/life balance; psychological protection from violence, bullying, and harassment; and protection of physical safety. In addition, The Standard suggests ‘other chronic stressors as identified by workers’ should also be addressed. To assess these 13 dimensions, The Standard (Canadian Standards Association 2013), along with subsequently published implementation guides (Canadian Standards Association and Mental Health Commission of Canada 2017), recommends that workplaces should use a 65-item survey instrument developed by Guarding Minds @ Work (GM@W). Despite the prominent promotion of the GM@W instrument, very little information is available concerning the psychometric properties of this instrument. In particular, while it appears that 12 of the 13 dimensions of the work environment identified in The Standard were based on the GM@W instrument, there is a brevity of evidence in the peer-reviewed literature, that the theoretical relationship between the 13 dimensions and the 65 items in the GM@W questionnaire is supported by data collected from workers. While employers recognise the importance of the psychosocial work environment, the number of dimensions contained in The Standard and complexity of dimensions have been identified as barriers for employers to implement The Standard (Kunyk et al. 2016). One Canadian study has noted that while 17% of employers sampled were aware of The Standard, only 2% of these had implemented The Standard in its entirety, with another 20% implementing some elements of The Standard but not others (Sheikh et al. 2018).

Given the challenges associated with the implementation of The Standard, it is important to understand if the instrument promoted and recommended in The Standard can adequately assess the 13 dimensions it proposes to asses. In addition, it is important to document general information about the distribution of scores across items contained within the GM@W instrument, as well as the distribution of scores for each of the 13 dimensions in a sample of workers across different industries. By generating this evidence, workplaces who opt to implement The Standard, can be confident that they have an instrument that can adequately distinguish between particular dimensions of the work environment as outlined in The Standard. This will then enable them to identify specific dimensions, as part of the 13 assessed, which warrant greater focus in their organisation, and the ability to monitor change in these dimensions over time. To address this important information gap in the development of The Standard, the objective of this paper is to assess the psychometric properties of the GM@W 65 item survey in a sample of employed workers in Ontario. Specifically, we assess the reliability of the items contained within each dimension of the GM@W instrument, the relationships between items contained within the standard, and the discriminant validity of the dimensions within The Standard. Discriminant validity in a multi-dimensional construct, such as The Standard, can be defined as the ability of each dimension to account for more variation in the items associated with it, than it does for measurement error or other dimensions contained within The Standard (Farrell 2010).

Methods

In February 2020 a survey was conducted in collaboration with EKOS Research Associates, to assess the GM@W survey instrument. Respondents were recruited using a pre-existing panel of over 100,000 households maintained by EKOS, where participants have agreed to participate in surveys from “time-to-time”. This sample has been drawn using both landline and cellular telephones, and the distribution of the target sample is meant to mirror the actual population in Canada (based on Census data). The addition of cell phones in the EKOS sample enables the inclusion of more low-income respondents (Blumberg and Luke 2010; Call et al. 2011). To be eligible to participate respondents had to be employed in a workplace with 5 or more employees, able to complete the survey online, in English. A total of 1,006 respondents completed the survey, which represents a conservative response rate of 11.8% (given not all respondents approached would have been eligible to participate in the study).

Guarding minds @ work survey

The GM@W Survey is a 65-item instrument, proposed to capture information on the 13 dimensions of the psychosocial work environment as outlined in The Standard. Each item in the GM@W survey is a positively worded statement about the work environment, with 4-level agreement scale used as response options (strongly disagree, disagree, agree, strongly agree). Higher scores on each item reflect greater agreement, and therefore more positive assessment of the work environment. Certain dimensions in the GM@W questionnaire are named differently than the names originally provided in The Standard. These are: ‘Psychological Competencies & Requirements’ for ‘Psychological Job Demands’; ‘Psychological Protection’ for ‘Psychological Protection From Violence, Bullying, and Harassment’; and ‘Balance’ for ‘Work/Life balance’. In this paper we have used the names as provided as part of the GM@W instrument. Items are evenly distributed across dimensions, with each dimension in The Standard being reflected by five items in the GM@W questionnaire. Items from the survey were administered in the same order as they are provided on the GM@W website (Guarding Minds @ Work 2016).

Analysis

Initial analyses examined the distribution of each of the survey items to identify floor or ceiling effects. While having no more than 15% of the sample in the top or bottom of the response scale has been suggested to assess the absence of floor/ceiling effects (McHorney and Tarlov 1995), given the GM@W survey items are positively worded statements on an agreement scale, we instead used a threshold of having no more than 80% of the sample agreeing or disagreeing with a particular statement as evidence of the absence of floor/ceiling effects (Cadarette et al. 2004). We then examined the correlations between items, both within the same dimension, and between items theoretically influenced by different dimensions. We also examined the distribution of scores across each of the 13 dimensions, to allow comparability between our sample and previous data collected by GM@W. Finally, we examined factor structure of the GM@W scale, using confirmatory factor analysis (CFA). In a multi-dimensional scale, responses to items assigned to a given theoretical dimension should be most strongly influenced by that latent construct/dimension and be less strongly influenced by other latent constructs/dimensions. As part of the CFA paths between each of the 13 latent constructs/dimensions proposed as part of The Standard and the 5 items identified to measure that dimension were specified. Paths between dimensions and items not measuring those dimensions were set to zero. Correlations were allowed between each of the 13 dimensions in The Standard, and error correlations were included if they belonged to items within the same dimension. The degree to which a proposed theoretical model is supported by the data was assessed using various model fit indices. We used three types of model fit indices: absolute fit; incremental fit; and parsimonious fit (Hatcher 1996; Kline 1998). Absolute fit is focused on the ability of the proposed model to reproduce the data, and is measured in this paper using the chi-square (χ²) statistic, as measure of the deviation between the proposed model and the actual correlation or covariance matrix. Incremental fit is concerned with comparing two competing models, with the first usually being a model where no relationships are specified and the comparison being the proposed model. Incremental fit was assessed using the comparative fit index (CFI), and the non-normed fix index (NNFI), also known at the Tucker-Lewis Index. Parsimonious fit assesses the tradeoff between the number of parameters estimated (noting that a better model fit can always obtain a better fit by estimating more parameters) and model fit. Parsimonious fit was assessed using the Root Mean Square Error of Approximation (RMSEA). We used guidelines proposed by Hu and Bentler (1999) to assess goodness of fit. These are a p-value of greater than 0.05 for the chi-square statistic, values of 0.95 and higher for the CFI and NNFI and values of 0.08 and lower for the RMSEA. A final set of analyses examined the reliability, and convergent and divergent validity of each latent construct using estimates from the CFA procedure. For reliability we used coefficient H (Hancock and Mueller 2001), and for convergent and divergent validity we used the Average Variance Extracted (AVE), Maximum Shared Variance (MSV), and inter-factor correlations (Farrell 2010; Fornell and Larcker 1981). For good reliability all coefficient H values should be 0.8 or above (Hancock and Mueller 2001), AVE values of 0.5 and higher indicate convergent validity, and divergent validity is indicated when the AVE is greater than the MSV, and the square root of the AVE value for a dimension (factor) is higher than correlations between that factor and other factors (Farrell 2010; Fornell and Larcker 1981). Given items are on a four-point scale, we used a Spearman correlation matrix as the input for the CFA, rather than a Pearson correlation matrix, which assumes normally distributed items. Analyses were conducted in SAS Version 9.4 (The SAS Institute 2017). CFA was conducted using PROC CALIS with a maximum likelihood estimation procedure.

Results

Of the initial sample of 1,006 respondents, 106 (10.5%) were missing information on one or more of the GM@W survey items, leaving a sample of 900 respondents with complete information on all 65 GM@W questions. Of these 106 respondents the majority (N = 77.73% of the 106) were missing information on only one of the 65 items. Table 1 presents the distribution of the sample by sex, age, employment status, industry and overall psychological job quality. Respondents with missing data were more likely to be male (compared to female). There was trend for greater missing data (as assessed by a Mantel-Haenszel Chi-Square test) with older age, and better psychological health and safety climate. No differences were observed in missing data across employment status or industry categories.

Table 1.

Distribution of missing data on GM@W questions across sample demographic and work characteristics

Variable	N	Complete data (N = 900) (%)	Missing data (N = 106) (%)
All	1006	89.5	10.5
Sex
Male	469	87.0	13.0
Female	526	92.2	7.8
Other	7	85.7	14.3
Age group
Under 25 years	35	94.3	5.7
25 to 34 years	162	92.6	7.4
35 to 44 years	240	92.1	7.9
45 to 54 years	283	89.4	10.6
55 to 64 years	223	86.1	13.9
65 + years	60	81.7	18.3
Employment status
Full-time	794	90.6	9.5
Part-time	115	84.4	15.7
Casual/seasonal/temp/other	97	86.6	13.4
Industry
Primary industry	32	90.6	9.4
Construction	32	87.5	12.5
Manufact/Trade/Transport	128	89.8	10.2
Education	159	89.9	10.1
Healthcare and social services	110	88.2	11.8
Arts/Rec/Accom/Food/Retail	124	91.9	8.1
Public administration and Public Services	142	87.3	12.7
Other Service Industry	278	89.9	10.1
How would you rate the psychological health and safety climate in your workplace?
Healthy/supportive	218	88.5	11.5
Good	262	86.6	13.4
Fair	186	91.9	8.1
Neutral	91	86.8	13.2
Not so good	105	88.6	11.4
Poor	71	95.8	4.2
Toxic	72	95.8	4.2

Employed Ontario respondents in workplaces with 5 or more employees (N = 1,006)

The distribution of responses across each of the 65 items in the GM@W measure is provided in the Appendix (Table A1). In general, missingness across items was small (less than 1.3% of respondents for all items). A total of 14 or the 65 items had 80% or more of the sample in the agree or strongly agree category, including 4 out of the 5 items in the engagement dimension, and 3 out of the 5 items in the physical protection & safety dimension, indicating potential ceiling effects.

Table 2 presents the average Spearman correlations for the five items within each of the 13 dimensions of the GM@W questionnaire, and the average spearman correlations between the five items within that dimension and the other 60 items outside of that dimension. In general, correlations were slightly higher with items within the same dimension, than between items outside of that dimension. For one dimension (psychological competencies and requirements), the average correlation between items and other items outside of that dimension was slightly higher than the correlation between items within that dimension. In addition, for three other dimensions (recognition and reward, involvement and influence and workload management) correlations within and outside of the dimension were similar. Table 3 provides examples of four pairs of items where the Spearman correlations were noted as high. In each case, the wording of the item seems similar, despite the items theoretically being influenced by different dimensions of The Standard.

Table 2.

Average Spearman correlations between items within the same dimension, and average Spearman correlations between items within a dimension with items outside of that dimension. Employed Ontario respondents working in workplaces with 5 or more employees

Dimension	Items within same dimension	Items outside of dimension
Psychological support	0.53	0.47
Organisational culture	0.58	0.49
Clear leadership & expectation	0.55	0.48
Civility and respect	0.56	0.47
Psychological competencies & requirements	0.40	0.43
Growth and development	0.50	0.46
Recognition and reward	0.51	0.47
Involvement and influence	0.52	0.48
Workload management	0.45	0.43
Engagement	0.49	0.34
Balance	0.51	0.44
Psychological protection	0.66	0.51
Protection of physical safety	0.63	0.43

Table 3.

Examples of highly correlated items, across dimensions

Item one	Item two	Spearman correlation
I am informed of important changes that may impact how my work is done (Involvement & influence)	I am informed about important changes at work in a timely manner (Clear leadership & expectations)	0.78 (0.75–0.81)
Our workplace effectively handles “people problems” that exist between staff (Civility and respect)	Difficult situations at work are addressed effectively (Organisational culture)	0.75 (0.72–0.78)
My immediate supervisor cares about my emotional well-being (Psychological protection)	My supervisor would say or do something helpful if I looked distressed while at work (Psychological support)	0.77 (0.74–0.80)
I am able to talk to my immediate supervisor about how I do my work (Involvement & influence)	My immediate supervisor appreciates my work (Recognition & reward)	0.77 (0.73–0.80)

Guarding minds @ work questionnaire

Figure 1 presents the distribution of scores across each of the 13 dimensions and overall, using the four categories suggested by GM@W. Scores between five and nine are labelled ‘Serious Concerns’, scores of 10 through 13 are ‘Significant Concerns’, scores of 14 to 16 are ‘Minimal Concerns’ and scores of 17 through 20 are ‘Relative Strengths’. Dimensions with the most positive profile (greatest percentage with relative strengths) were engagement and protection of physical safety, while the dimensions with the most negative profiles (highest percentage in serious concerns) were organisational culture, clear leadership and expectations, and psychological protection.

Fig. 1.

Distribution of scores across each of the 13 dimensions contained within the GM@W questionnaire (N = 900)

Table 4 presents the model fit statistics from two confirmatory factor analysis models. The first is a model with each item influenced by its theoretical dimension, and covariances specified between each of the 13 dimensions (latent constructs). The second model includes two correlated error terms for items within a particular dimension, based on modification indices. Due to these correlated errors the second model has two fewer degrees of freedom. In general, both models had challenges with convergence due to the predicted covariance matrix not being positive-definite. Further investigation suggested this was due to the latent constructs being highly correlated with each other. The degree to which the theoretical model from GM@W questionnaire was supported by the data was poor. All fit indices were below recommended cut offs, although the upper bound of the RMSEA was below the 0.08 threshold. It should be noted that the absolute fit indices (chi-square statistic) are also strongly influenced by sample size (Kline 1998). In the CFA model with the two correlated errors (GM@W (mod)—the second model referred to above), 18 of the 78 correlations between the 13 latent factors were 0.95 and higher, and only one (between engagement and physical safety) was below 0.60.

Table 4.

Model fit statistics from confirmatory factor analysis of GM@W questionnaire

	GM@W	GM@W (mod)*	Rule of Thumb**
Absolute fit
Chi-square statistic	9153 (1937 df)	8878 (1935 df)
	p < 0.001	p < 0.001	p > 0.05
Incremental fit
Comparative fit index	0.84	0.85	0.95 +
Non-normed fit index	0.81	0.81	0.95 +
Parsimony
RMSEA (upper 95% bound)	0.064 (0.066)	0.063 (0.065)	< 0.05 (< 0.08)

Employed Ontario respondents in workplaces with 5 or more employees (N = 900)

* GM@W (mod) involved correlating two error terms. One between items “My employer makes efforts to prevent harm to employees from harassment, discrimination or violence” and “My employer deals effectively with situations that may threaten or harm employees” within psychological protection dimension. Second between item “I am able to reasonably balance the demands of work and personal life” and “I have energy left at the end of most workdays for my personal life” within balance dimension

** For good model fit absolute fit, p-value should be above 0.05, for incremental fit values should be equal to or above 0.95, and for parsimony RMSEA values should be below 0.05, with upper bound below 0.08

Table 5 presents estimates for coefficient H reliability, average variance extracted and maximum shared variance for each of the 13 dimensions from the original model, with no correlated errors. Estimates for the model with correlated error terms were not appreciably different so are not presented, but are available from the authors on request. In general, the reliability of items within each dimension all exceeded acceptable levels. The AVE, which represents how much variance in the items for each construct that can be explained by the latent factor, were below 0.5 for three of the 13 dimensions, and above 0.6 for only 2 of the 13 dimensions. While it is recommended that the AVE should be greater than the MSV—which represents the maximum variance that the latent construct can explain in other latent construct—this was not the case for any of the 13 dimensions.

Table 5.

Coefficient H (reliability), Average Variance Extracted and Maximum Shared Variance for dimensions of the GM@W Questionnaire among Employed Ontario respondents working in workplaces with 5 or more employees

	Coefficient H (reliability*)	Average variance extracted**	Maximum shared variance
Psychological support	0.861	0.535	0.963
Organisational culture	0.888	0.588	0.999
Clear leadership & expectation	0.891	0.571	0.962
Civility and respect	0.874	0.565	0.999
Psychological competencies & requirements	0.804	0.426***	0.973
Growth and development	0.848	0.507	0.991
Recognition and reward	0.879	0.542	0.993
Involvement and influence	0.859	0.529	0.993
Workload management	0.811	0.449	0.908
Engagement	0.836	0.494	0.565
Balance	0.862	0.514	0.908
Psychological protection	0.889	0.608	0.963
Protection of physical safety	0.903	0.630	0.734

*Reliability: Coefficient H values should be 0.80 or higher

**Convergent validity: The average variance extracted (AVE) should be 0.5 or higher

***Italics indicates where reliability or convergent validity requirements are not met

Table 6 presents the correlation matrix between the 13 dimensions from the original model, and the square root of the AVE. For discriminant validity the square root of the AVE for a dimension should exceed each of the correlations between that dimension and other dimensions. As demonstrated by the grey shading, this criterion was met in nine of the 78 correlations.

Table 6.

Square root of the average variance extracted and inter-factor correlations for dimensions of the GM@W questionnaire among Employed Ontario respondents working in workplaces with 5 or more employees

	1	2	3	4	5	6	7	8	9	10	11	12	13
1. Psychological support	0.731*
2. Organisational culture	0.928**	0.767
3. Clear leadership & expectation	0.884	0.981	0.756
4. Civility and respect	0.890	0.999	0.917	0.751
5. Psychological competencies & requirements	0.900	0.983	0.943	0.945	0.653
6. Growth and development	0.920	0.919	0.927	0.862	0.986	0.712
7. Recognition and reward	0.906	0.921	0.926	0.856	0.956	0.995	0.736
8. Involvement and influence	0.918	0.941	0.970	0.894	0.969	0.989	0.996	0.728
9. Workload management	0.883	0.848	0.857	0.823	0.880	0.875	0.900	0.952	0.670
10. Engagement	0.659	0.679	0.622	0.613	0.752	0.695	0.696	0.656	0.694	0.703
11. Balance	0.937	0.840	0.847	0.801	0.862	0.857	0.884	0.891	0.953	0.633	0.717
12. Psychological protection	0.981	0.959	0.938	0.972	0.933	0.928	0.932	0.945	0.916	0.647	0.929	0.780
13. Protection of physical safety	0.807	0.747	0.734	0.780	0.753	0.763	0.751	0.773	0.746	0.529	0.725	0.857	0.794

^*Bold estimates in the correlation table are the square root of the AVE for each dimension

^**For divergent validity the square root of AVE should be greater than inter-construct correlations. Italics indicate where divergent validity requirements are not met

Discussion

Psychological health and safety is gaining increasing attention in workplaces across Canada and in other developed economies globally. The 2013 National Workplace Psychological Health and Safety Standard, produced by the Canadian Standard Association and commissioned by the Mental Health Commission of Canada (Canadian Standards Association 2013), recommends that workplaces looking to assess and address the 13 dimensions of the work environment as outlined in The Standard, should use the 65 item GM@W questionnaire. Yet, to date, limited information is available on the psychometric properties of the GM@W questionnaire, and if it can measure, and distinguish between the 13 dimensions of the work environment as outlined in The Standard. In a sample of 900 Ontario workers, from a variety of industries, we observed numerous measurement issues in the GM@W survey, specifically concerning highly correlated items which are theoretically influenced by different dimensions, and an inability of the items within the GM@W survey to be able to isolate particular dimensions from The Standard. In general, dimensions of The Standard measured with the GM@W survey demonstrate poor discriminant validity. That is, the ability of each dimension to explain the variance in the items associated with that dimension is lower than the ability of that dimension to explain variance in other dimensions. These limitations are important, as workplaces using the GM@W survey will not be able to isolate the different dimensions as outlined in The Standard, or assess changes in particular dimensions over time.

There are strengths and limitations which should be taken into account when interpreting our study findings. The response rate to our survey was low (approximately 12% of respondents in the EKOS panel who were approached to complete the survey). Compared to estimates from the employed labour force in Ontario from Statistics Canada’s Labour Force survey in February 2020, our sample had a higher proportion of females (54% versus 50%), fewer respondents under the age of 25 (4% versus 14%), slightly more permanent full-time workers (80% versus 76%), fewer respondents from the construction industry (3% versus 6%) and more respondents from the education (16% versus 9%) and public administration (14% versus 6%) industry groups. However, we were able to recruit workers across a variety of industry groups, age groups, with variation in assessments of the psychosocial work environment (22% of our sample described the psychological health and safety climate in their workplace as healthy or supportive, and 14% described their psychological health and safety climate as poor or toxic). We can compare our results to those previously published from a survey conducted in 2012 commissioned by The Great-West Life Centre for Mental Health in the Workplace. In this survey of 6,624 Canadian workers, using a different household panel, the most positively rated dimensions in The Standard were engagement, followed by protection of physical safety, while the least positively rated dimensions were organisational culture, growth and development, psychological support and psychological protection (Great-West Life Centre for Mental Health in the Workplace 2012). In our sample the same two dimensions were rated most positively, and organisational culture and psychological protection were amongst the least favourable. The levels of serious and significant concerns were similar, although slightly higher in our sample, compared to this earlier survey (Great-West Life Centre for Mental Health in the Workplace 2012). It is possible that the level of psychosocial work conditions that someone experiences can be related to their propensity to respond to a survey. However, it is likely this can work in both directions (i.e. people with more negative conditions can be more likely to respond, or people with more positive conditions may be more likely to respond). Given no information on the GM@W standard is available from a representative sample of the Canadian population, it is not possible for us to estimate which direction this bias might be present in our sample (if at all). If the differences in serious and significant concerns in our sample, compared to the previous Great-West Life Centre sample, are indicative that people with more negative psychosocial work environments were more likely to respond to our survey, the estimates of the potential ceiling effects in our sample may be underestimated. We used a Spearman correlation matrix to account for the non-normal distribution of responses across items in our CFA. We did compare these results to other potential approaches, such as a polychoric correlation matrix. In general, CFA model fit statistics when using a polychoric correlation matrix were worse than when using the Spearman correlation matrix (results not presented but available from authors on request). In addition, we have not assessed all psychometric attributes of the GM@W survey. We had originally also intended to conduct a test-retest assessment of each of the 65 items as part of the survey. However, the timing of the retest assessment was during the initial workplace closures in Ontario due to COVID-19, as such we did not have enough respondents where the psychosocial work environment had been stable between test and retest.

Comparing our results to previous examinations of the GM@W survey is challenging, given the limited number of peer-reviewed publications on this instrument. We do not think the poor model fit statistics observed in our study are specific to the EKOS panel or the Canadian labour force. For example, previous CFA examinations of the 19 dimensions contained within the Copenhagen Psychosocial Environment Questionnaire (COPSOQ), using the same panel for recruitment, have demonstrated good model fit (Ramkissoon et al. 2019). In addition, the high correlations between dimensions captured in the GM@W survey have been noted in a report from a Portuguese sample (Magalhães and Paul 2017), while the inability of the GM@W survey to reflect the 13 dimensions of The Standard has also been noted among a Canadian sample in a conference abstract (MacLellan et al. 2016).

Understanding the implications of poor model fit and lack of discriminant validity for the GM@W survey instrument for the use of The Standard in general is complex. Although in The Standard it is inferred that the 13 dimensions of the psychosocial work environment were informed by other available psychosocial models, in particular the demand-control model (Karasek et al. 1998; Karasek and Theorell 1990), the effort-reward imbalance model (Siegrist 1996), and the model of organisational justice (Elovainio et al. 2002), it is also stated that 12 of the 13 dimensions were those that were part of the GM@W instrument at that time. Further work should examine whether the extreme overlap between the 13 dimensions measured in our study reflects poor measurement development of the items contained within the GM@W survey instrument, or general lack of conceptual clarity in the development of the 13 dimensions in the actual Standard.

In conclusion, the results of our paper suggest caution is required if using the GM@W survey to assess the 13 dimensions of the psychosocial work environment as outlined in The Standard. Based on our results, the 13 dimensions as measured by the GM@W instrument are highly overlapping, and unable to be distinguished with the 65-item survey. Given the importance of the psychosocial work environment in contemporary labour markets and workplaces, it is important that progressive workplaces looking to address dimensions of the psychosocial work environment are provided with instruments with demonstrated psychometric properties. This is particularly important for the psychosocial work environment as there are a number of dimensions that a workplace could target, and due to resource constraints, workplaces might choose to focus on particular dimensions at different times. The inability of the GM@W instrument to be able to discriminate between dimensions presents a challenge for workplaces in this situation, both in terms of assessing their current psychosocial work environment, and measuring progress in addressing particular dimensions. We would suggest alternative instruments, where psychometric attributes have already been demonstrated, should be recommended, with appropriate concurrent changes to the names and number of psychosocial dimensions made, in future versions of The Standard.

Appendix

See Table 7

Table 7.

Percent of missing values and distribution of scores across all 65 items in the GM@W questionnaire. Employed Ontario respondents in workplaces with 5 or more employees (N = 1,006)

Dimension	Question	% Missing	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree	Pct agreement (%)
Psychological support	My employer offers services or benefits that adequately address my psychological and mental health	0.3	16.3	15.7	41.7	26.0	67.9
	My supervisor would say or do something helpful if I looked distressed while at work	0.2	10.3	20.9	37.5	31.1	68.7
	I feel supported in my workplace when I am dealing with personal or family issues	0.6	8.1	17.9	36.4	37.1	73.9
	My workplace supports employees who are returning to work after time off due to a mental health condition	1.2	9.2	18.2	40.8	30.6	72.2
	People in my workplace have a good understanding of the importance of employee mental health	0.4	11.9	21.7	40.4	25.7	66.3
Organizational culture	All people in our workplace are held accountable for their actions	0.3	17.0	25.9	37.4	19.4	56.9
	People at work show sincere respect for others’ ideas, values and beliefs	0.3	6.6	20.0	45.0	28.1	73.4
	Difficult situations at work are addressed effectively	0.4	16.2	28.7	37.6	17.1	54.9
	I feel that I am part of a community at work	0.4	9.0	15.9	41.6	33.2	75.1
	Employees and management trust one another	0.3	23.3	25.0	34.8	16.7	51.6
Clear leadership & expectations	In my job, I know what I am expected to do	0.2	1.7	7.4	35.5	55.3	90.9
	Leadership in my workplace is effective	0.5	19.5	26.1	36.5	17.4	54.2
	I am informed about important changes at work in a timely manner	0.3	17.7	25.7	36.3	20.1	56.5
	My supervisor provides helpful feedback on my performance	0.6	13.9	23.2	36.3	26.0	62.7
	My organization provides clear, effective communication	0.4	19.2	28.1	35.2	17.1	52.5
Civility & respect	People treat each other with respect and consideration in our workplace	0.1	6.4	13.7	47.8	32.0	79.9
	Our workplace effectively handles “people problems” that exist between staff	0.5	21.2	29.5	37.6	11.2	49.1
	People from all backgrounds are treated fairly in our workplace	0.4	7.2	9.5	35.8	47.1	83.2
	Unnecessary conflict is kept to a minimum in our workplace	0.3	11.6	19.7	41.7	26.7	68.6
	My workplace has effective ways of addressing inappropriate behaviour by customers or clients	0.6	14.2	23.3	40.2	21.8	62.3
Psychological competencies & requirements	Hiring/promotion decisions consider the “people skills” necessary for specific positions	0.3	15.0	24.4	40.8	19.6	60.5
	My company hires people who fit well within the organization	0.7	10.1	22.0	47.8	19.4	67.7
	I have the social and emotional skills needed to do my job well	0.3	1.5	4.9	32.0	61.3	93.6
	My supervisor believes that social skills are as valuable as other skills	0.6	8.8	17.4	41.6	31.7	73.7
	My position makes good use of my personal strengths	0.5	7.3	13.1	40.2	39.0	79.5
Growth & development	I receive feedback at work that helps me grow and develop	0.5	13.9	28.3	35.4	21.9	57.5
	My supervisor is open to my ideas for taking on new opportunities and challenges	0.8	10.1	16.6	39.7	32.8	73.0
	I have the opportunity to advance within my organization	0.4	19.5	23.6	34.8	21.8	56.8
	My company values employees’ ongoing growth and development	0.3	9.9	20.7	42.6	26.4	69.3
	I have the opportunity to develop my “people skills” at work	0.3	6.0	13.6	38.6	41.6	80.4
Recognition & reward	My immediate supervisor appreciates my work	0.3	8.1	11.4	35.6	44.6	80.5
	I am paid fairly for the work I do	0.2	11.0	14.8	36.0	38.0	74.1
	My company appreciates extra effort made by employees	0.4	17.9	23.8	34.4	23.6	58.2
	Our organization celebrates our shared accomplishments	0.7	10.2	19.3	39.9	29.9	70.3
	My employer values my commitment and passion for my work	0.4	10.6	19.3	36.7	33.0	70.0
Involvement & influence	I am able to talk to my immediate supervisor about how I do my work	0.3	6.2	11.1	34.2	48.2	82.7
	I have some control over how I organize my work	0.4	4.9	9.9	37.0	47.8	85.1
	My opinions and suggestions are considered at work	0.5	7.9	16.0	46.7	28.9	76.0
	I am informed of important changes that may impact how my work is done	0.3	12.2	25.6	38.4	23.6	62.1
	My employer encourages input from all staff on important issues related to their work	0.5	15.1	24.1	35.5	24.9	60.6
Workload management	The amount of work I am expected to do is reasonable for my position	0.4	10.2	17.7	35.8	35.9	72.0
	I can talk to my supervisor about the amount of work I have to do	0.7	8.9	13.1	37.1	40.3	77.9
	I have the equipment and resources needed to do my job well	0.3	5.6	17.0	44.7	32.4	77.4
	My work is free from unnecessary interruptions and disruptions	0.4	22.3	36.0	28.2	13.1	41.5
	I have control over prioritizing tasks and responsibilities when facing multiple demands at work	0.7	7.1	15.3	38.3	38.7	77.5
Engagement	I enjoy my work	1.0	5.6	10.5	40.0	42.9	83.7
	I am willing to give extra effort at work if needed	0.2	1.9	3.2	31.7	63.0	94.9
	My work is an important part of who I am	0.4	7.2	14.2	36.2	42.1	78.5
	I am committed to the success of my organization	0.2	3.4	8.1	37.3	51.1	88.5
	I am proud of the work I do	0.9	3.1	5.5	31.3	59.2	91.4
Balance	My employer encourages me to take my entitled breaks (e.g., lunchtime, sick time, vacation time, earned days off, parental leave)	0.4	15.1	18.9	29.4	36.2	65.9
	I am able to reasonably balance the demands of work and personal life	0.2	5.3	17.4	43.0	34.1	77.3
	My employer promotes work-life balance	1.0	13.7	24.8	37.6	23.0	61.1
	I can talk to my supervisor when I am having trouble maintaining work-life balance	0.3	11.7	19.6	36.1	32.3	68.6
	I have energy left at the end of most workdays for my personal life	0.5	15.5	27.0	36.4	20.6	57.2
Psychological protection	My employer is committed to minimizing unnecessary stress at work	0.5	21.4	30.3	36.1	11.7	48.1
	My immediate supervisor cares about my emotional well-being	0.6	11.1	15.6	39.6	33.1	73.1
	My employer makes efforts to prevent harm to employees from harassment, discrimination or violence	0.3	8.3	12.3	38.4	40.8	79.4
	I would describe my workplace as being psychologically healthy	0.3	19.9	23.0	36.4	20.5	57.0
	My employer deals effectively with situations that may threaten or harm employees (e.g., harassment, discrimination, violence)	0.5	10.9	17.9	35.4	35.3	71.0
Protection of physical safety	Management takes appropriate action to protect my physical safety at work	0.3	6.2	11.6	37.2	44.7	82.2
	My employer offers sufficient training to help protect my physical safety at work (emergency preparedness, safe lifting, violence prevention)	0.3	9.2	13.6	37.6	39.4	77.2
	When physical accidents occur or physical risks are identified, my employer responds effectively	0.7	6.7	10.4	37.6	44.6	82.8
	I have the equipment and tools I need to do my job in a physically safe way (protective clothing, adequate lighting, ergonomic seating)	0.4	5.4	10.2	36.2	47.8	84.3
	My employer responds appropriately when workers raise concerns about physical safety	0.9	8.5	12.5	36.6	41.6	78.8

Authors’ contributions

PS and JO participated in the conception and design of this work, and acquisition of the data. PS conducted the analyses. PS and JO interpreted the results. PS drafted the manuscript and JO provided critical revision. PS and JO approved the final version of the manuscript. PS is the guarantor for the paper.

Funding

The Institute for Work & Health, and the Occupational Health Clinics for Ontario Workers are supported through funding from the Ontario Ministry of Labour, Training and Skills Development (MLTSD). The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the MLTSD; no endorsement is intended or should be inferred. No specific funding was obtained to conduct this work.

Data availability

Data in aggregated form is available from the authors on request.

Code availability

All code used for the analyses is available from the authors on request.

Declarations

Conflict of interests

OHCOW has developed an alternative platform to assess the psychosocial work environment, called Stress Assess, which is based on the validated Copenhagen Psychosocial Questionnaire. The Stress Assess survey is offered at no cost to all users. The authors declare no other conflicts of interest.

Ethical approval

The work was performed at the Institute for Work and Health, Toronto, ON, Canada. This study was approved through the University of Toronto, Health Sciences Research Ethics Board (# 38935).

Consent to participate

All participants involved in the study provided informed consent through a checkbox provided.

prior to completing the internet-based survey.

Consent to publication

Both authors have reviewed the manuscript and consent to having it published. This manuscript or a similar manuscript it not submitted anywhere else.