Psychometric Evaluation of GHQ-12 as a Screening Tool for Psychological Impairment of Healthcare Workers Facing COVID-19 Pandemic

Abstract

Background:

The General Health Questionnaire (GHQ) is a widely used tool in clinical and research settings due to its brevity and easy administration. Researchers often adopt a dichotomous measurement method, considering a total score above or below a certain threshold, leading to an extreme simplification of the gathered data and, therefore, the loss of clinical details. In a multistep evaluation study aimed at assessing health care workers’ mental health during the COVID-19 pandemic, GHQ-12 proved to be the most effective tool to detect psychological distress compared to other scales. These results deepened the understanding of GHQ-12 properties through a statistical study focusing on items’ properties and characteristics.

Methods:

GHQ-12 responses were analyzed using Item Response Theory (IRT), a suitable method for scale assessment. Instead of considering the single overall score, in which each item accounts equally, it focuses on individual items’ characteristics. Moreover, IRT models were applied combined with the latent class (LC) analysis, aiming to determine subgroups of individuals according to their level of psychological distress.

Results:

GHQ-12 was administered to 990 healthcare workers, and responses were scored using the binary method (0-0-1-1). We applied the two-parameter logistic (2-PL) model, finding that the items showed different ways of responses and features. The latent class analysis classified subjects into three sub-groups according to their responses to GHQ-12 only: 47% of individuals with general well-being, 38% expressing signs of discomfort without severity, and 15% of subjects with a high level of impairment. This result almost reproduces the subjects’ classification obtained after administering the six questionnaires of the study protocol.

Conclusions:

Accurate statistical techniques and a deep understanding of the latent factors underlying the GHQ-12 resulted in more effective usage of such a psychometric questionnaire – i.e., a more refined gathering of data and significant time and resource efficiency. We underlined the need to maximize the extraction of data from questionnaires and the necessity of them being less lengthy and repetitive.

1. Introduction

The General Health Questionnaire (GHQ) aims to provide information about an individual’s mental well-being by identifying distressing symptoms [1]. Its shorter version (GHQ 12-item) has become one of the most widely used scales for assessing psychological distress and short-term changes in mental health, and its popularity can be mainly attributable to its brevity and easy administration [2].

GHQ-12 has shown strong psychometric properties and it is recommended as screening tool to detect common mental disorders as depressive, anxiety and somatic disorders [3, 4].

Several analyses explored its characteristics, especially the factor structure, mostly identifying a two-factor solution through Confirmatory Factor Analysis (CFA) techniques. The two factors commonly proposed were the Depression/Anxiety construct (related to the emotional component of psychological distress) and the Social Dysfunction construct (related to the social functioning component of the individual experiencing the distress) [5]. Other authors proposed a three-dimensional solution, comprising anxiety (4-item), social dysfunction (6-item), and loss of confidence (2-item) dimensions [6].

The use of GHQ-12 to measure the mental health status in healthcare populations is frequent, and several recent contributions gave examples of its application in analyzing psychological well-being during the pandemic [7-14]. In all these cases, the screening of the psychological status through GHQ-12 was determined according to its total score. The scores typically used are the binary scale (0-0-1-1) and the 4-point Likert-type scale (0-1-2-3). Responses to all items are summed up to a total score ranging from 0 to 12 (binary scale) or 0 to 36 (Likert scale), with higher scores indicating more severe impairment. A score above a specific cut-off (3/4 for bimodal and 13/14 for the Likert scale) indicates psychological distress and suggests further investigation for potential mental disorders [15].

A possible difference in items contribution can be lost through such a measurement method, in which each item counts the same. Indeed, every single item may have a different weight, expressing different severity of the psychological impairment measured by the test.

We proposed to analyze GHQ-12 data through Item Response Theory (IRT)-based methods as they provide more details about individual items. IRT is a specific statistical model for evaluating questionnaires, and it is a more suitable tool than the usual methodologies based on Classical Test Theory (CTT), whose use is still prevalent in the psychometric field. The strength of such a technique lies in its focus on items rather than individual scores, while in the CTT, the evaluation of test properties and item characteristics is not included.

From a statistical point of view, if the results of a test are reported as a single score, it is implicitly assumed that all the items are measuring the same trait equally therefore losing the complexity of underlying traits in psychological testing. IRT allows to evaluate individual-level distress and to describe the performances of the items on the questionnaire simultaneously may providing a better clinical insight on symptoms detected and associations with potential underlying mental disorders.

To our knowledge, few authors proposed analyzing the GHQ-12 scale via an IRT approach. In some cases, IRT was applied as a suitable tool to determine the factor structure of the scale [16, 17], and more recently through the multidimensional version of IRT [18]. For instance, the IRT approach used by Smith et al. [16] explored the fact that item phrasing, item variance and levels of respondents’ distress affect the factor structure observed for the GHQ-12 and may perhaps explain why different factor structures of the instrument have been found in different populations. Other uses of IRT on GHQ-12 regard computerized adaptive testing [19] or Mokken analysis [20].

However, no studies have studied the performance of GHQ-12 with IRT in an occupational setting during a pandemic. Therefore, this work aims to perform an IRT-based analysis on GHQ-12, investigating the methodological and clinical benefits of such an approach.

2. Methods

2.1 Population Study

We conducted a multistep epidemiological study within occupational health surveillance to systematically assess healthcare workers’ mental well-being during the COVID-19 pandemic in a large Hospital in Milan (Italy).

GHQ-12 was administered, jointly with the Impact of Event Scale (IES-R; post-traumatic distress, [21]) and General Anxiety Disorder Scale (GAD-7; anxiety, [22]) questionnaires, to assess the psychological impact of the pandemic and to identify possible signs of impairment, further investigated through psycho-diagnostic questionnaires and specialist evaluation.

The steps of such assessment were fully presented in a previous study [23]: for each worker, the psychological well-being was screened in three steps. The first-level questionnaire collected several personal information and data from three tests (i.e., GHQ-12, IES-R, and GAD-7). Workers who scored above the cut-off in at least one scale were further investigated by the second-level questionnaire composed of psycho-diagnostic scales to assess depressive symptoms (Patient Health Questionnaire-9; [24]), dissociative symptoms (Dissociative Experiences Scale –II; [25]) and other psychological symptoms (Symptoms Checklist-90 [26]). If the second level showed psychological impairments, an individual specialized treatment with a psychiatrist and psychologist (third-level) was offered.

In this framework, GHQ-12 (binary version) proved to be an effective screening tool, deserving a deeper investigation [27].

The occupational medicine unit, where workers underwent the periodical health surveillance already prescribed by the current Italian legislation, proposed the study protocol to all workers since July 2020. By July 2021, 990 subjects out of a total population of 1,610 had been enrolled in the study. The participation rate was 62%. In detail, 220 (13%) workers did not answer our calls or were unavailable and 400 (25%) refused to participate.

Participants were predominantly female (70%) with a mean age of 45 years (sd=11); nurses (42.5%) was the most prominent job category, followed by physicians (23.5%), administrative staff (12%), health assistants (6.5%) and other roles (16%). Four hundred and forty-six (45%) participants had the experience of working in a COVID-19 area: 25% were working with COVID-19 patients during data collection, and 20% had worked in a COVID-19 department before enrollment.

Six hundred and twenty-seven workers (63%) did not show signs of psychological impairment; 363 (37%) presented signs of psychological impairment at the first screening level (i.e., with scores above the cut-off in at least one scale among GHQ-12, IES-, R and GAD-7) and underwent the second level assessment. Among these, 231 (67%; 23% of the total sample) scored above the cut-off in at least one scale among PHQ-9, DES, and SCL-90. As a result, we were able to classify participants into three sub-groups, according to their scorings: a group with no evidence of psychological distress after first-level screening (Group 1, N=627), workers who expressed distress without severe symptoms (Group 2, N=132), and subjects who expressed signs of impairment and received psychological and/or psychiatric support (Group 3, N=231).

Out of the 363 subjects who showed psychological impairment at first-level screening, almost all (91%) scored above the cut-off (equal to 4) of GHQ-12, while about half of them over-passed the cut-off of IES-R and GAD-7 scales (53% and 56% respectively). This result suggested that GHQ-12 could determine the transition to the second level more effectively than the other scales.

Results obtained from the analysis of risk factors for psychological impairment were presented in detail in a previous paper [27].

2.2 Item Response Theory (IRT)

The basic assumption of IRT models is that a person’s interactions with test items can be represented according to probabilistic relations, containing a single parameter to describe the individual’s characteristics θ. The power of IRT is that it estimates item characteristics through some item parameters, which permit the calculation of the expected score at the item level (e.g., probability of 1 or correct answer if responses are binary or dichotomous) and at the test level. In addition, the person’s latent trait θ_i for an individual i is also estimated, considering specific item characteristics and how the person answers to each item.

We applied the so-called two parameters logistic (2-PL), suitable for binary data, which uses two parameters to describe each item j, corresponding to its “difficulty” and its “discrimination”. The item difficulty represents the level of latent trait for which one has a 50% probability of responding ‘correctly’ (or 1) to that item. In other words, if θ_i=β_j, then P(Y_ij=1)=0.5. If θ_i>β_j, then P(Y_ij=1)>0.5 and if θ_i<β_j, then P(Y_ij=1)<0.5. The discriminating parameter for item j, λ_j, estimates the capacity of the item to distinguish between subjects with different latent trait levels.

Two plots are typically employed in the IRT framework to visualize the analysis results. Item Characteristic Curves (ICCs) show the probability of answering 1 to an item at varying levels of the latent trait, specifying how well an item discriminates between respondents at various levels of the latent trait. The “easier” items functions are on the left side of the plot, in the lower regions of the latent trait scale, while the more “difficult” items are on the right (in our case, they are the items that underlie more severe impairment in mental health). The discrimination parameter represents the slope, which refers to how well the item response options discriminate (or differentiate) between subjects with high and low latent trait levels.

IICs show how well and precisely each item measures the latent trait at various attribute levels. Item Information measures the strength of the relationship between an item and the latent trait. Some items may provide more information at low levels of the attribute, while others may provide more information at higher levels of the attribute.

We applied the IRT model in one of its discrete versions, based on the so-called Latent Class (LC) analysis [28, 29], whose assumption is that the population under study is composed of homogeneous classes of individuals who have very similar unobservable characteristics [30, 31]. Data were collected through a computerized database generated by REDCap [32], which was subsequently analyzed by R software [33].

3. Results

Based on the dichotomous scored version of GHQ-12, we calculated Cronbach’s α equal to 0.87, indicating good internal consistency. The mean score was 3.31 (SD=3.45), with 37% of subjects scoring above the cut-off equal to 4 (indicating a general level of psychological distress). Table 1 presents the twelve questionnaire items with the distribution of the respective answers.

Table 1.

GHQ-12 answers distribution.

	0	1
Item 1 - Able to concentrate	75%	25%
Item 2 - Loss of sleep over worry	64%	36%
Item 3 - Playing a useful part	86%	14%
Item 4 - Capable of making decisions	85%	15%
Item 5 - Felt constantly under strain	49%	51%
Item 6 - Could not overcome difficulties	78%	22%
Item 7 - Able to enjoy day-to-day activities	55%	45%
Item 8 - Able to face problems	79%	21%
Item 9 - Feeling unhappy and depressed	68%	32%
Item 10 - Losing confidence	83%	17%
Item 11 - Thinking of self as worthless	92%	8%
Item 12 - Feeling reasonably happy	73%	27%

Item parameters estimation (difficulty and discrimination) of the 2-PL model are reported and graphically represented in Figure 1, which shows the ICCs.

Figure 1.

Item Characteristic Curves for GHQ-12 questionnaire.

Item 5 (followed by Item 7) has the lowest threshold parameter, while Item 3 (“feeling useless”) and Item 11 (“thinking of yourself as a worthless person”) are the most ‘difficult’ ones. Item 5 has a location (or difficulty) parameter equal to 0, meaning that a person with a latent trait θ at level 0 has the same probability of answering 0 or 1 to Item 5. On the contrary, a level of latent trait θ equal to 1.73 is needed for having an equal probability of answering less than/same as usual or more/much more than usual to Item 11. Concerning the discrimination parameter, Item 12 (feeling reasonably happy, all things considered) has the highest value, much greater than the others. On the other hand, Item 3 has the lowest discrimination parameter. The item provides sample information about differences across individuals when discrimination is high. Item 5 and Item 7 have the leftmost lines represented in the plot, while the curve of Item 3, which is also the less steep, is plotted on the right. The curve of Item 12 is indeed the steepest one. In the IICs plot in Figure 2, the curve of Item 12 gives much information around a value of θ between 0 and 1, while Item 3 (with the lowest discrimination parameter) gives more or less the same (low) information over a broader range.

Figure 2.

Item Information Curves for GHQ-12 questionnaire.

The latent class model reaches the best fit (calculated through BIC) with three latent classes. The weights and levels of the latent trait for each dimension and latent class are in Table 2. The latent model estimates three values for the θ finding support points equal to -3.3 (low level of distress) with weight equal to 0.47, a medium level (around 0) for 38% of subjects, and a higher level of distress (θ=1.15) with weight 0.15.

Table 2.

Latent class model results and percentage of subjects by multistep evaluation.

	θ	%	Group 1 (no distress)	Group 2 (psychological distress)	Group 3 (psychological impairment)
Class 1	-3.3	47%	97%	1%	2%
Class 2	-0.8	38%	44%	20%	36%
Class 3	1.2	15%	1%	14%	85%

We compared the three latent classes with the three groups resulting from questionnaire scorings (Group 1, Group 2, Group 3) in Table 2. Almost all (97%) subjects belonging to Class 1 did not undergo the second-level screening, i.e., they did not express any sign of discomfort through GHQ-12, IES-R, and GAD-7. On the contrary, most (85%) of those assigned to Class 3 needed psychological support, while only one-third of Class 2 required psychological therapy.

Table 3 shows the percentage of answers equal to 1 for each item, according to the latent class.

Table 3.

Probability of answering equally to 1 by class.

	ITEM
	1	2	3	4	5	6	7	8	9	10	11	12
Class 1	3%	4%	4%	2%	17%	1%	16%	1%	1%	-	-	-
Class 2	31%	52%	17%	17%	76%	24%	64%	21%	45%	15%	6%	33%
Class 3	76%	92%	38%	50%	96%	81%	90%	81%	95%	76%	41%	97%

Participants assigned to Class 1 answered 0 for almost all the items, with the highest percentages of the answer 1 occurring for Items 5 and 7 (but much less than in the general distribution). On the contrary, considering such items (Items 5 and 7), almost everyone who belongs to Class 3 answered 0. In addition, for the group with more severe signs of psychological distress, percentages of answer 1 were much higher (than the general distribution) up to the items found to be the most “difficult” (Item 3 and Item 11). For the second class, the distribution was more balanced, and more than half of the participants answered 1 only to Item 2, Item 5, and Item 7.

4. Discussion

The GHQ-12 is frequently used among different settings and populations, and its assessment methods adopt predominantly a dichotomous scoring, which may contribute to lose potential differences in items contribution; this rationale motivated us to a psychometric analysis to better clarify the methodological and clinical quality of this tool. The analysis was carried out within a study aimed to evaluate psychological well-being of healthcare workers in a large Hospital in Milan (Italy) facing COVID-19 pandemic.

In our scenario, the IRT was a suitable tool for scale assessment. Instead of considering the single overall score, in which each item accounts equally, the item-based analysis produced interesting results by identifying specific items able to detect psychological impairment effectively.

Such considerations are similar to those obtained in previous analyses on GHQ-12 based on IRT methods in other frameworks. For example, the approach of Smith and colleagues [18], within the multi-dimensionality assessment of GHQ-12, showed how the use of the summated scores for the GHQ-12 could potentially lead to an incorrect assessment of patients’ psychiatric morbidity.

The focus on items characteristics allowed us to deeply investigate how the mental health status was captured by GHQ-12 in our population, identifying different levels of severity (given by item difficulty) and quantifying the impact each item had on the measurement of general distress. We further specify that participants were healthcare workers involved in a disruptive pandemic, which imposed them unprecedented and heavy workloads coupled with lack pf preparation to cope with such demands. In such circumstances, questions about utility, capacity to make decisions, loss of trust and confidence showed peculiar responses, affecting in different way the psychological wellbeing; feeling useless (Item 3) and thinking of yourself as a worthless person (Item 11) caused more severe impairments than, for instance, feeling constantly under strain (Item 5) or being unable to enjoy day-to-day activities (Item 7).

The analysis on the item response patterns also allowed the classification of subjects according to different impairment levels: the first class with almost all responses equal to 0 (subjects without distress), the second class where percentages of responses equal to 1 were high only for Item 3, Item 5 and Item 9 (subjects with psychological distress) and the third class with huge percentages of responses equal to 1 (subjects with psychological impairment).

Such a classification agreed with previous results obtained by administering several other psychological questionnaires. Indeed, through an item-based latent class analysis, we could determine the screening outcome without considering the other questionnaires, previously part of the first-level evaluation (IES-R, GAD-7).

IRT was a helpful tool for identifying clinically meaningful subgroups in our population, recognizing distinct patient profiles, and tailoring effective interventions, whose importance was already underlined in previous works [34].

Thus, subjects’ classification based only on responses to GHQ-12 could potentially simplify workers evaluation. Results show that one step of the evaluation (i.e. second-level) is redundant and may be skipped. According to symptoms’ severity, immediate access to specialist evaluation can be planned for those with psychological impairment (Class 3), without testing them through second-level scales; subjects with less severity, i.e. psychological distress (Class 2), will be instead be monitored with a check evaluation after a certain period of time.

Our study is prone to potential biases as self-selection of respondents [35]. We managed to minimize that risk grounding our investigation on the occupational physician health surveillance, obtaining a very high participation rate and minimizing the risk of untrue or uncompleted answers.

We know that our results cannot be generalized, and neither are they comparable with results obtained in different scenarios. The pandemic’s consequences directly affected our population, and this exceptional situation should be carefully considered. However, in terms of psychological assessment, our results agree with findings obtained in similar populations of healthcare workers who expressed high levels of GHQ-12 during the COVID-19 pandemic in Italy (e.g., more than 50% above cut-off equal to 3 in Del Piccolo et al. [10]) and in other countries (e.g., 39% of subjects above cut-off equal to 3 in Dai et al. [8]).

5. Conclusions

The GHQ-12 is commonly analyzed and interpreted according to CTT rules and we decided to complement it by performing an analysis based on IRT. As outlined in our work, drawing on the strengths of IRT as an alternative to CTT analyses supported the development of rigorous measures and valuable interpretations. It was possible to classify the degree of severity of psychological impairment by administering only GHQ-12 questionnaire and according to response patterns, focusing on the way in answering each question more than the scored obtained as a sum of “positive responses”.

In light of these results, our approach may suggest simplifying the multistep protocol for evaluating mental health in occupational settings, recommending using GHQ-12 as a single measurement tool to be the most effective. Such a method may also meet the need for resources and time reduction when conducting studies and assessments involving workers.

Through such analysis, we gave an example of the utility of IRT in psychometric studies conducted among workers populations. The application of appropriate methodological tools to support the interpretation of questionnaires could sensibly discover their potential in simplifying the screening framework and saving one of the most important workers’ resources: their time. Even in questionnaire-based epidemiological studies, in many cases, less is more.

Funding:

The full cost of the study was covered by the Foundation IRCCS Ca’ Granda Ospedale Maggiore Policlinico (internal funding) and by a charitable donation to the University of Milan by the “Fondazione Romeo ed Enrica Invernizzi”.

Institutional review board statement:

The study was approved by the Hospital Ethical Committee (Milan Area 2 Ethical Committee, n.652_2020 of July 21, 2020) and was conducted in compliance with all local legal and regulatory requirements, Good Clinical Practice, the International Conference on Harmonisation document and the Declaration of Helsinki.

Informed consent statement:

The participation was voluntary; each subject read and signed an extended informed-consent to participate in the study.

Acknowledgments:

The Authors thank all workers who agreed to participate and undergo our multistep evaluation for giving us their precious time. Further, authors thank Ezio Belleri, Laura Chiappa, Dario Laquintana and all the IRCCS Cà Granda Fondazione Hospital managing team for their support in different phases of study design and implementation; Antonio Minolfi and Stefania Pazzi from the Pavia (Italy) Consorzio di Bioingenieria e Informatica Medica (CBIM) for the data collection software and their technical support; Chiara Moltrasio, MariaLuna Sandri and Francesca Maddalena for their technical support in contacting study subjects and arranging workers’ interview.

Declaration of interest:

The authors declare no conflict of interest.