Validation of the fear of COVID-19 scale in South Africa: Three complementary analyses

Abstract

Fear is the most common response to disease outbreaks. Persistent and prolonged fear can elevate the levels of psychological distress and aggravate preexisting mental health problems. Therefore, prompted by the central role of fear in psychological responses to COVID-19, the Fear of COVID-19 Scale was developed, which is the only instrument that can assess emotional fear reactions in relation to the current pandemic. In this study, we extend research on the psychometric properties of this instrument by adopting three complementary approaches: classical test theory, Rasch analysis, and Mokken analysis. Combining these methods allows for a more nuanced overview of the psychometric properties of the instrument. The sample comprised South African teachers (n = 355) who completed the Fear of COVID-19 Scale. All three approaches confirmed the reliability and the construct, convergent, and concurrent validity of the scale as used with South African teachers. In addition, all three approaches confirmed that the scale is sufficiently homogenous to be considered unidimensional.

With the emergence of the COVID-19 pandemic, given its rapid transmission and high mortality rate, many countries worldwide have started monitoring its spread and implementing a range of infection prevention measures. These measures included social-distancing policies, travel restrictions, compulsory wearing of masks, closure of all educational institutions and nonessential services [35], and mandatory vaccination for vulnerable population groups, such as healthcare workers [36]. However, despite these prevention efforts, the virus continued to spread unabated and has become an uncontainable stressor [4], [11].

Fear is one of the most common psychological responses to disease outbreaks [1]. It is typically an adaptive response to perceived or actual threat, and it serves to activate responses aimed at promoting safety [11]. However, in the presence of an unremitting stressor, constant fear can increase and sustain high levels of psychological distress [1]. Moreover, persistent fear of becoming infected with the virus, of loved ones becoming infected, and of transmitting the virus to significant others can adversely impact mental health [19]. Research has confirmed that fear of COVID-19 is associated with elevated levels of stress and anxiety as well as mood-related disorders and obsessive behaviors [33]. Excessive fear has also been found to aggravate several preexisting mental health conditions, such as anxiety and depression [3] and can have a further negative impact on pre-existing physical conditions [21]. Furthermore, intense fear of COVID-19 and of being in the vicinity of those who are infected with the virus has been found to be associated with the development of posttraumatic stress symptoms [37].

Given the central role of fear in psychological responses to pandemics, various instruments have been developed to assess fear and anxiety toward disease outbreaks. To our knowledge the Fear of COVID-19 Scale (FCV-19S) [1] is the only measure developed specifically to assess fear of COVID-19. The initial version of the scale was developed in Persian [1] and was subsequently translated into over 15 languages (e.g., Hebrew [4]; Turkish [10]; Spanish [20]). It has also been used in diverse cultural contexts (e.g., Greece [38]; India [5]; Japan [41]; Bangladesh [30]) and demonstrated sound psychometric properties (α = 87 [30], [38], [41]).

In a recent South African study, researchers assessed the psychometric properties of the FCV-19S [18] using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) and reported that the scale demonstrates a unidimensional factor structure and sound internal consistency reliability (α = .87,ω = .88) when used with a sample of university students. In the current study, we aim to extend this work by assessing the psychometric properties and dimensionality of the FCV-19S as used with teachers in the South African context with three different but complementary approaches: classical test theory (CTT), Rasch analysis (a parametric item response theory), and Mokken analysis (a nonparametric item response theory). In general, combining these methods allows for a more nuanced overview of the psychometric properties of an instrument [22]. While CTT allows computing a score for the FCV-19S and offers a global view of the respondents’ fear towardCOVID-19, the greater diagnostic power of Rasch and Mokken analyses allows identifying the items that are more likely to be endorsed by respondents with different levels of fear [22], [26], [45]. This can facilitate the generation of more precise respondent profiles and criterion-referenced interpretations [45]. Generally speaking, this type of information is important for identifying the significance of fear among various population groups as well as for targeted intervention efforts [25].

1. Method

1.1. Participants

Our participants were a convenience sample of 355 school teachers in South Africa. The majority of the sample where from the province of the Western Cape (82.3%), were female (76.6%), taught at a primary school level (61.1%), and lived in an urban area (61.7%). The mean number of years that the participants spent in the teaching profession was 15.7 (SD  = 11.8), whereas the mean age of the sample was 41.9 years (SD  = 12.4).

1.2. Measures

In addition to a demographic survey, the participants completed the FCV-19S [1] and the Perceived Vulnerability to Disease Scale (PVD-Q) [7]. The FCV-19S consists of seven items that measure emotional fear reactions toward the pandemic on a five-item Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). The total score ranges between seven and 35, with a higher sum score indicating a higher level of fear of COVID-19.

The PVD-Q is a 15-item self-report instrument consisting of two subscales. The first subscale assesses beliefs regarding one's own susceptibility to infectious diseases (perceived infectability: PI) and comprises eight items, whereas the other subscale assesses emotional discomfort in contexts that are associated with a high potential for disease transmission (germ aversion: GA) and comprises seven items. The participants respond to each item on a seven-point scale ranging from strongly disagree to strongly agree. Overall, the PVDS has demonstrated high internal consistency (α = .90 [7]; α = 73 [43]) and has been used with different samples in diverse contexts (e.g., Japan [44]; Germany [34]).

1.3. Procedure

An electronic web-based survey comprising a demographic survey, the FCV-19S and the PVD-Q was created using Google Forms and distributed to public school teachers in South Africa during the period from April to July 2021 via social media platforms. In addition, the University of the Western Cape's school liaison officer assisted with distributing the electronic survey to school teachers in the province via existing institutional networks.

1.4. Ethics

Ethical approval for the study was granted by the Humanities and Social Sciences Ethics Committee of the University of the Western Cape (ethics reference number: HS21/3/8). All participants completed informed consent forms and were provided with the contact details for psychological counseling support in case completing the survey evoked distress.

1.5. Data Analysis

CTT analyses focused on internal consistency (coefficient α), interitem correlations, corrected item-total correlation, standard error of measurement (SEM), average variance extracted (AVE), composite reliability (CR), and EFA (maximum likelihood). All of these were obtained using IBM SPSS Statistics version 26 for Windows (IBM Corp., Armonk, NY, USA). In addition, to examine the unidimensionality and concurrent validity of the scale, CFA with IBM SPSS Amosversion 26 (IBM Corp.) was performed. In general, a reliability coefficient of > 70, an interitem correlation of > 30, an item-total correlation of > 50 [9], a small SEM, and an AVE of > 50 [8] are considered acceptable. With respect to the CFA, several fit indices were used to test the proposed unidimensional structure of the FCV-19S, and concurrent validity was examined in terms of the relationship of the FCV-19S with PVD-Q. As suggested by Kline [14], the selected indices included the root-mean-square error of approximation (RMSEA, best if close to .08 or less), comparative fit index (CFI, best if close to .90 or greater), goodness-of-fit index (GFI, best if close to .95 or greater), and Tucker–Lewis index (TLI, best if close to .95 or greater; [13]. In addition, the Akaike information criterion (AIC), which allows for model comparisons, was included [2]. In general, the model with the lowest AIC value is considered to have the best fit. With respect to the CFA, two models of the structure were examined, namely, a one-factor model and a bifactor model, in which the FCV-19S was conceptualized as consisting of two subscales as well as a total scale. In addition, bifactor indices were calculated using the Bifactor Indices Calculator [6]. These indices include the explained common variance (ECV), which refers to the percentage of variance accounted for by the general factor; the omega hierarchical (OmegaH), which indicates the percentage of variance in total scores that is due to individual differences in the general factor; and the percentage of uncontaminated correlations (PUC), which measures the number of unique correlations among items that can be explained by the general factor. An OmegaHof > 80 means that the scale is essentially unidimensional [29]. It has also been suggested that the ECV, OmegaH, and PUC should be considered together to draw conclusions regarding the dimensionality of an instrument. In this regard, Reise et al. [28] suggested that a PUC of < 80, together with a general ECV of > 60 and Omega Hof > 70, would indicate the presence of some dimensionality which is not sufficient to discount the conclusion that the instrument is essentially unidimensional.

Rasch analysis was performed using Winsteps version 5.1.4 [16]. This included the infit and outfit mean square (MnSq), item and person separation index, and item and person separation reliability, as well as the eigenvalue of the unexplained variance of the “first contrast” obtained through a principal component analysis (PCA) of the residuals. Linacre [17] suggests that mean square values between 0.5 and1.5 are optimal, whereas person separation should ideally have an index of > 2 and reliability of > 8, demonstrating that the items can differentiate between different respondents. Item separation, however, confirms the item hierarchy (construct validity) of the instrument and should ideally have an index of >3 and reliability of > 8. Finally, it has been suggested that the presence or absence of unidimensionality in the Rasch analysis depends on the size of a possible second dimension, referred to as the first contrast. In this regard, the eigenvalue associated with a possible second dimension should ideally be < 2.

Mokken analysis was performed with R (R Core Team, 2017) [27] using the “Mokken” package [39], [40]. This analysis allows determining the unidimensionality, monotonicity, invariant item ordering (IIO), and reliability (MS_Rho). In terms of unidimensionality, Mokken analysis provides a scalability coefficient (H) for the total scale and one for each item (H_i). The following rule of thumb is typically applied when evaluating H-coefficients: H ≥ .50 represents a strong scale, .40 ≤ H < .50 represents a medium scale, and .30 ≤ H < .40 represents a weak scale [42]. In addition to the scalability coefficient, the Mokken package in R provides an automated algorithm for searching for unidimensional scales, called the Automated Item Selection Procedure (AISP). In general, the AISP provides an indication of whether all items load on one scale or whether there are groupings of items that might indicate multidimensionality. Monotonicity refers to the probability that for each item, the likelihood of a particular response level is a monotonically non-decreasing function of the underlying latent trait, in this case fear of COVID-19. It is also worth noting that the Mokken package provides an indication of the violation of monotonicity for each item, as well as the significance of such a violation. The absence of violations would suggest that all items discriminate well between those with high levels of fear of COVID-19 and those with low levels of fear of COVID-19. Finally, IIO refers to the extent to which items have the same order irrespective of the score on the latent variable [31]. Similar to the assumption of monotonicity, the Mokken package provides an indication of the violation of IIO for each item and the significance of such a violation. For items with such violations, it would indicate that respondents with the same level of fear of COVID-19 might have endorsed these particular items in significant different ways. In addition to identifying the IIO violations, the Mokken package provides a scalability coefficient (H^T) as an index of the accuracy of IIO. The rule of thumb for H^T is as follows: IIO has zero accuracy if H^T  < .3, low accuracy if .3 ≤ H^T  < .4, medium accuracy if .4 ≤ H^T  < .5, and high accuracy if H^T  ≥ .5 [15].

2. Results

Table 1 shows the CTT and Rasch and Mokken analyses at the item level. This includes factor loadings, interitem correlations, item-total correlation, infit and outfit MnSq, and the scalability coefficient (H_i) for the individual items of the FCV-19S.

Table 1.

Classical Test Theory, Rasch, and Mokken Indices at the Item Level.

Item	1	2	3	4	5	6	7
1. Afraid	—
2. Uncomfortable	.71	—
3. Hands clammy	.55	.65	—
4. Afraid of losing life	.69	.62	.61	—
5. Anxious about news	.58	.59	.55	.62	—
6. Cannot sleep	.44	.47	.66	.49	.53	—
7. Heart races	.49	.54	.70	.61	.62	.78	—
Mean	3.4	3.2	2.6	3.3	3.4	2.4	2.7
SD	1.3	1.3	1.3	1.4	1.2	1.2	1.3
Factor loadings	.74	.77	.82	.78	.75	.74	.82
Item-total correlations	.71	.74	.77	.75	.71	.68	.77
Infit MnSq (Rasch)	1.12	0.95	0.88	1.07	0.99	1.06	0.89
Outfit MnSq(Rasch)	1.16	1.02	0.86	1.00	1.02	1.04	0.90
Hi (Mokken)a	.62	.64	.67	.65	.63	.63	.67
SE of H (Mokken)	.03	.03	.03	.03	.03	.03	.03

^aScalability coefficient of individual items.

Factor analysis resulted in one factor extracted explaining 59.64% of the variance. All factor loadings (.74 to .82) and item-total correlations were found to be significant (.68 to .77), indicating that all items contributed significantly to the scale. All interitem correlations were above .30 (.44 to .71), providing some evidence for construct validity. The infit and outfit MnSq values were found to range from0.88 to 1.16, thus falling within the range of 0.5 to 1.5, which is deemed optimal. The H_i index for all items ranged between .62 and .67,which is above the rule of thumb of .30 proposed by Mokken [23]. Similar to the item-total correlation, these H_i values demonstrate that all items contribute to the measurement of the latent variable, namely, fear of COVID-19.

Table 2 shows the CTT and Rasch and Mokken indices at the scale level for the FCV-19S.

Table 2.

Classical Test Theory, Rasch, and Mokken Indices at the Scale Level.

Index	Value
Cronbach's α	.911
Composite reliability	.912
Average variance extracted	.60
Standard error of measurement	2.12
Item separation reliability (Rasch)	.99
Item separation index (Rasch)	9.70
Person separation reliability (Rasch)	.87
Person separation index (Rasch)	2.58
Unexplained variance in thefirst contrast (Rasch)	2.22a
Scale H (Mokken)	.64
Mokkenscale reliability (MSRho)	.922

^aEigenvalue.

Table 2 shows that the indices of reliability can be considered very good (α = .91, CR = .91, MS_Rho  = .92) and that AVE is above the .50 level (AVE = .60), which is considered acceptable. In addition, the SEM found can be considered to be low (SEM = 2.12). It is also possible to consider the Rasch indices as satisfactory; for instance, the person and item separation reliability values were .87 and .99, respectively, whereas the person and item index values were 2.58 and 9.70, respectively. Finally, PCA of the residuals indicated that a significant amount of variance was explained by the Rasch dimension (64.2%, eigenvalue = 12.57), and only 7.4% by a possible second dimension, consisting of two to three items (eigenvalue = 2.22). Visual inspection of the plot of standardized residuals allowed identifying three items that clustered together: “hands clammy” (Item 3), “cannot sleep” (Item 6), and “heart races” (Item7). However, the deattenuated correlation between this cluster of items and the rest of the items was positive and strong (.71), suggesting that this cluster measures the same latent construct.

In terms of the Mokken analysis, the AISP defined one dimension on which all items loaded. The scalability coefficients of individual items (H_i) were found to range between .62 and .67 (SE = .03), whereas the scalability coefficients for item pairs (H_ij) were found to range between .51 and .84 (SE = .04 to .05). These scalability coefficients support the inclusion of all items of the FCV-19S as part of a Mokken scale. In addition, the scalability coefficient for the entire scale (H) was found to be .64 with MS reliability of .92,which is above the threshold of H > 50 proposed in the literature for a strong scale [32]. Visual and numerical inspection, however, revealed no significant violation of monotonicity. With respect to item ordering, IIO resulted in an H^Tvalue of .39, thus indicating medium accuracy of item ordering. In particular, IIO indicated two significant violations relating to Items 4 and 5, namely, “afraid of losing life” and “anxious about COVID-19 news.” However, the deletion of these items did not lead to a significantly improved H^Tvalue (.42) and reduced the MS reliability (.89). These two violations do not detract from the overall excellent psychometric properties of the FCV-19S, as the literature indicates that even psychometric models that do not imply IIO may still be useful [31].

Since the Rasch analysis suggested an additional dimension consisting of three items, CFA was used to compare a one-factor model of the FCV-19S with a bifactor model consisting of one general factor (total scale) and two specific factors (subscales). As suggested by the Rasch analysis, the three items that clustered together (“hands clammy,” “cannot sleep,” and “heart races”) appear to reflect physiological fear reactions as opposed to the other four items that reflect emotional fear reactions. As such, the bifactor model conceptualized the FCV-19S as consisting of a total scale score (fear of COVID-19) as well as two subscale scores, namely, physiological fear reactions and emotional fear reactions. Fig. 1 shows the hypothesized unidimensional CFA model of the FCV-19S. In this figure, the seven items of the FCV-19S are presented as observed measurements and fear of COVID-19 is presented as a latent variable. In addition, to establish concurrent validity, the two subscales of PVD-Q were correlated with the FCV-19S. Fig. 2 shows the bifactor model.

Fig. 1.

One-Dimensional Model of the Fear of COVID-19 Scale. All regression weights are standardized. The rectangles are measured variables, whereas the ellipse is a latent construct. ***P < .001.

Fig. 2.

Bifactor Model of the Fear of COVID-19 Scale. All regression weights are standardized. The rectangles are measured variables, whereas the ellipse is a latent construct. ***P < .001.

Table 3 shows the goodness-of-fit and bifactor indices resulting from the CFA.

Table 3.

Bifactor and Fit Indices for Two Models of the Fear of COVID-19 Scale.

Index	Best fit indicator	One-factor model	Bifactor model
Goodness-of-fit
χ2(df)		63.93(21)	94.74(21)
P-value	Nonsignificant	P < .001	P < .001
GFI	> .95	.96	.94
TLI	> .90	.96	.93
CFI	> .90	.98	.96
RMSEA	< .08	.08	.10
AIC	Lower levels	111.93	142.74
Bifactor
ECV	—	—	.73
OmegaH	—	—	.84
PUC	—	—	.57

χ²: Chi² statistic; GFI: goodness-of-fit index; TLI: Tucker–Lewis index; CFI: comparative fit index; RMSEA: root-mean-square error of approximation; SRMR: standardized root-mean-square residual; AIC: Akaike information criterion; ECV: explained common variance; PUC:percentage of uncontaminated correlations.

The fit indices for the one-factor model were all found to be at or above the suggested cut-off points (RMSEA = .076, CFI = .975, GFI = 962, and TLI = .96), indicating that a one-dimensional model is an acceptable fit. Moreover, the factor loadings (regression weights) were all significant and above .50 (between .68 and .86), indicating that all items contribute significantly to the total scale. However, while the fit indices for the bifactor, with the exception of RMSEA (.10), can be considered acceptable (CFI = .94, GFI = 95, and TLI = .93), the pattern of factor loadings for the emotional fear reaction subscale was largely nonsignificant, whereas the loadings for the physiological fear reaction subscale were negative. The model comparison index, AIC, was also found to be lower for the one-factor model (AIC= 111.93) than for the bifactor model (AIC = 142.74). Moreover, bifactor analysis indicated that while the general factor (fear of COVID-19) explained 73% of the common variance and the two specific factors (physiological and emotional fear reactions) explained 27% of the variance, OmegaH was above the threshold of .80, indicating that the FCV-19S is essentially unidimensional. When the PUC, ECV, and OmegaHare considered together (PUC < .80, ECV > .60, and OmegaH > .70), it can be observed that the presence of some dimensionality is not enough to discount an interpretation of the scale as being unidimensional. It is also worth noting that the association between the FCV-19S and the two subscales of PVD-Q was significant (GA: β = .25, p  < .001; PI: β = .41, p  < .001).

3. Discussion

The aim of this study is to extend existing research on the FCV-19S by examining its psychometric properties, including its reliability, validity, and dimensionality, using three different complementary approaches, namely, CTT, Rasch analysis, and Mokken analysis. Since each of these approaches has its own strengths and limitations, combining them provides a comprehensive picture of the reliability, validity, and dimensionality of the scale [22].

All reliability indices were found to be satisfactory. The scale also demonstrated satisfactory internal consistency (coefficient αand MS_Rho), and the Rasch analysis confirmed that the scale can distinguish between different levels of “performers” (person separation reliability) and that a hierarchy of items exists (item separation reliability).

With regard to dimensionality, EFA and CFA confirmed that the scale can be considered unidimensional. In terms of the Rasch analysis, a PCA of residuals indicated that significant variance is explained by the Rasch dimension (64.2%, eigenvalue = 12.57) and only 7.4% (eigenvalue = 2.22) is explained by a possible second dimension. With respect to the Mokken analysis, the AISP identified one latent factor on which all items loaded. It also showed that the scalability coefficients (H and H_i) were all above the suggested threshold, indicating that the scale is sufficiently homogenous. Overall, the results obtained from the three approaches represent acceptable evidence of a unidimensional structure for the FCV-19S.

In terms of convergent validity, Ghadi [8] suggested that factor loadings obtained in CFA, CR, and AVE provide a basis for decisions regarding convergent validity. Notably, the factor loading associated with the FCV-19S, which was above .68, the CR of .91, and the AVE of .60 provide evidence of convergent validity.

The construct validity of the FCV-19S can also be considered satisfactory. According to Hajjar [9], an item-total correlation of > .50 and an interitem correlation of > .30 provide evidence of construct validity. In the current study, both the interitem correlations and item-total correlation exceeded this threshold. The item separation indices in the Rasch analysis provide further evidence of construct validity. For instance, Linacre (2021b) suggested that when the item separation index is > 3 and the item separation reliability is > 9, this confirms the existence of an item difficulty hierarchy, which evidences construct validity. However, the item separation indices obtained in the current study exceed these suggested values. Moreover, the infit and outfit MnSq values were also within the acceptable range, providing further evidence of validity. Finally, concurrent validity was demonstrated through the association between fear of COVID-19 and the two subscales of PVD-Q, namely, germ aversion and perceived infectability.

In general, while fear is typically an adaptive and transient response to threat, disease outbreaks can prolong fear and enhance psychological distress [5]. Fear ofCOVID-19 has been associated with anxiety and mood disorders and has been found to aggravate preexisting mental health problems [11]. Furthermore, fear of contagion as well as discrimination and stigma directed toward infected individuals or those perceived to be responsible for the spread of COVID-19 can potentially fragment communities [12]. Hence, there is a need for methodologically sound instruments that are capable of identifying distinct markers of fear among population groups for targeted interventions, such as public education and health campaigns [24].

4. Limitations

This study had certain limitations. First, although the survey was anonymous, which should encourage disclosure, the potential impact of self-reporting bias needs to be acknowledged [38]. Second, it is unclear how the sample would generalize to teachers in other contexts. Therefore, future studies using a larger and more diverse sample are needed to replicate these results.

5. Conclusion

In this study, we demonstrated that the FCV-19S is a unidimensional scale with sound psychometric properties. Our findings complement, support, and extend previous studies on the psychometric properties, validity, and reliability of this instrument for samples from diverse contexts.

Disclosure of interest

The authors declare that they have no competing interest.