Development and initial validation of the COVID Stress Scales

Highlights

• •A large population-representative SAMPLE was surveyed to better understand COVID-19-related DISTRESS.
• •We developed and validated the COVID Stress Scales, comprising 36 items on 5 scales.
• •The COVID Stress Scales offer promise for identifying people in need of pandemic-related mental health services.

Abstract

Research and clinical observations suggest that during times of pandemic many people exhibit stress- or anxiety-related responses that include fear of becoming infected, fear of coming into contact with possibly contaminated objects or surfaces, fear of foreigners who might be carrying infection (i.e., disease-related xenophobia), fear of the socio-economic consequences of the pandemic, compulsive checking and reassurance-seeking regarding possible pandemic-related threats, and traumatic stress symptoms about the pandemic (e.g., nightmares, intrusive thoughts). We developed the 36-item COVID Stress Scales (CSS) to measure these features, as they pertain to COVID-19. The CSS were developed to better understand and assess COVID-19-related distress. The scales were intentionally designed so they could be readily adapted for future pandemics. The CSS were developed and initially validated in population-representative samples from Canada (N = 3479) and the United States (N = 3375). A stable 5-factor solution was identified, corresponding to scales assessing COVID-related stress and anxiety symptoms: (1) Danger and contamination fears, (2) fears about economic consequences, (3) xenophobia, (4) compulsive checking and reassurance seeking, and (5) traumatic stress symptoms about COVID-19. The scales performed well on various indices of reliability and validity. The scales were intercorrelated, providing evidence of a COVID Stress Syndrome. The scales offer promise as tools for better understanding the distress associated with COVID-19 and for identifying people in need of mental health services.

1. Introduction

Emerging findings from China suggest that more than 25 % of the general population experienced moderate to severe levels of stress- or anxiety-related symptoms in response to COVID-19 (Qiu et al., 2020; Wang et al., 2020). These findings are similar to those reported during the SARS outbreak (Cheng, Wong, Tsang, & Wong, 2004) and in the 2009 H1N1 pandemic (Rubin, Amlôt, Page, & Wessely, 2009; Wheaton, Abramowitz, Berman, Fabricant, & Olatunji, 2012). Studies of previous epidemics and pandemics show that anxiety, or the lack thereof, is an important driver of behavior (Taylor, 2019). People with too little anxiety about a viral outbreak are less likely to engage in hygiene behaviors (e.g., handwashing), less likely to adhere to physical distancing mandates, and are less likely to get vaccinated if a vaccine is available (Taylor, 2019). On the other hand, people with excessive anxiety are more likely to engage in socially disruptive behaviors, such as panic buying and surging unnecessarily into hospitals and clinics when they misinterpret their minor ailments as signs of serious infection (Asmundson & Taylor, 2020a, 2020b).

Given the role that anxiety plays in shaping behavioral responses to viral outbreaks—both behaviors that can mitigate as well as those that can facilitate the spread of infection—it is critical that public health decision-makers, health officials, and health care providers understand the nature and degree of adverse psychological responses to the current COVID-19 crisis. To date, there has been little empirical attention devoted to this issue; indeed, while several measures of COVID-19-related fears and anxieties have recently emerged (e.g., Ahorsu et al., 2020; McKay, Yang, Elhai, & Asmundson, 2020), they tend to be unidimensional (i.e., focused on general fear aspects of COVID-19) and are based on limited psychometric evaluation. There is a pressing need to develop an empirically-sound measure of COVID-19-related stress and anxiety-related symptoms.

Research and clinical observations (e.g., Taylor, 2019) suggest that during times of pandemic many people exhibit fear and anxiety-related distress responses that include the following: Fear of becoming infected, fear of coming into contact with possibly contaminated objects or surfaces, fear of foreigners who might be carrying infection (i.e., disease-related xenophobia), fear of the socio-economic consequences of the pandemic (e.g., job loss), compulsive checking and reassurance-seeking regarding possible pandemic-related threats, and traumatic stress symptoms about the pandemic (e.g., nightmares, intrusive thoughts). The COVID-19 Stress Scales (CSS) were developed to measure the aforementioned features as well as to better understand and assess COVID-19-related distress. The scales were intentionally designed so they could be readily adapted for future pandemics. In the present study we examined the factor structure, reliability as internal consistency, and convergent and discriminant validity of the CSS. To determine the robustness (replicability) of the findings, results from a Canadian sample were replicated in an American sample.

2. Method

2.1. Sample and data collection procedures

Data were collected from Canada and the United States using an internet-based self-report survey delivered in English by Qualtrics, a commercial survey sampling and administration company, between March 21 and April 1, 2020. The data collection protocol was approved by the University of Regina Institutional Research Ethics Board and all respondents consented prior to beginning the survey. Participation was solicited by Qualtrics using sampling of web-panels to meet quotas based on age, sex, ethnicity, socioeconomic status, and geographic region within each country in order to obtain a population representative sample. Filters were used to eliminate data from careless or incomplete responders. The final sample comprised 6854 adults (United States: 3375; Canada: 3479). Respondents were aged 18–94 years (M = 49.8 years, SD = 16.2). Almost half (47 %) were female and most (52.3 %) were employed full- or part-time. A total of 10.1 % were on leave or unemployed, 25.0 % were retired, 4.3 % were homemakers, and 4.4 % were students. Most (78.8 %) had completed full or partial college, 17.6 % had only completed high school or equivalent, and 2.9 % did not graduate from high school. Most (68.1 %) were Caucasian, with the remainder being Asian (11.5 %), African American/Black (9.4 %), Latino/Hispanic (6.4 %), Native American/Indigenous (1.4 %), or other (3.2 %).

2.2. Measures

In addition to questions regarding COVID-19-related distress, the survey comprised measures regarding demographics, current anxiety and depression, and various trait characteristics. Data collection began after the first cases of COVID-19 had been confirmed in both the United States and Canada. In order to assess pre-COVID-19 trait characteristics, we instructed participants to respond to the trait measures (see 2.2.2 Validation Scales) as they would have prior to the COVID-19 outbreak.

2.2.1. Candidate CSS items

The CSS was constructed by examining the relevant literature (e.g., Taylor, 2019) and by consulting experts on health-related anxiety. The following domains were identified: (1) Fears about the dangerousness of COVID-19 (14 items), (2) fears about sources of COVID-19-related contamination (i.e., objects, surfaces; 8 items), (3) COVID-19-xenophobia (i.e., fears that foreigners are sources of COVID-19; 7 items), (4) fears about the personal social and economic consequences of COVID-19 (e.g., fears of disruption in the supply chain, fears of looting or rioting; 10 items), (5) COVID-19-related checking (e.g., checking news media or social media, seeking reassurance from friends or medical professionals; 7 items), and (6) traumatic stress symptoms related to COVID-19 (e.g., unwanted intrusive thoughts or nightmares relating to COVID-19; 7 items).1

Items assessing these domains were generated (58 items in all, listed in the appendix of supplementary materials). Five items were culled on rational bases (see supplement), leaving 53 items for analysis. To simplify the instructions, we referred to COVID-19 as “the virus.” COVID-19 actually refers to the disease and SARS-CoV-2 is the virus; however, we expected that many respondents would not be aware of this distinction. Based on feedback from pilot testing, respondents readily understood what we were referring to. We assessed a 7-day window because fears about COVID-19 may change over time as the pandemic unfolds. A 7-day window provides the ability to assess these changes while also keeping the window broadly consistent with the other symptom measures included in our assessment battery.

Instructions for the fear-related items (domains 1–4) were as follows: “The following questions ask about various kinds of worries that you might have experienced over the past seven days… about the virus.” Items were rated on a 5-point scale ranging from 0 (not at all) to 4 (extremely). We used the term “worries” to assess feared (anticipated) outcomes. The checking and traumatic stress items were rated on a 5-point scale ranging from 0 (never) to 4 (almost always).

2.2.2. Validation scales

Several measures of trait characteristic were used to assess convergent and discriminant validity. These measures were not tied to COVID-19. As mentioned, we asked respondents to respond to these measures as they would have before the outbreak of COVID-19. We also included a measure to assess a bias toward socially desirable responding.

2.2.2.1. Patient Health Questionnaire-4 (PHQ-4; Kroenke, Spitzer, Williams, & Lowe, 2009)

The PHQ-4 provides a brief 4-item measure of current anxiety and depression using a 4-point Likert scale ranging from 0 (not at all) to 3 (nearly every day). Participants were asked to rate their past week anxiety and depression. Factor analysis supported a two-factor structure corresponding to depression and anxiety. The PHQ-4 has demonstrated good reliability and validity in both clinical and non-clinical samples (Kroenke et al., 2009; Löwe et al., 2010). Internal consistency for the full scale in the present study was excellent (Cronbach alpha = .90).

2.2.2.2. Short Health Anxiety Inventory (SHAI; Salkovskis, Rimes, Warwick, & Clark, 2002)

The main subscale of the SHAI was used. This measures health anxiety independently of physical health status using 14 items rated on a 4-point frequency of occurrence scale (e.g., I do not, I occasionally, I spend much of my time, I spend most of my time) over the past six months. The SHAI has good reliability and validity in both clinical and non-clinical samples (Abramowitz, Deacon, & Valentiner, 2007; Salkovskis et al., 2002; Wheaton, Berman, Franklin, & Abramowitz, 2010). Internal consistency for the 14-item subscale in the present study was excellent (alpha = .90).

2.2.2.3. Obsessive Compulsive Inventory-Revised (OCI-R; Foa et al., 2002)

The OCI-R measures symptoms characteristic of obsessive-compulsive disorder (OCD). The two subscales of checking (e.g., “I check more things than necessary”) and washing (e.g., “I sometimes have to wash or clean myself simply because I feel contaminated”) were used in this study, each comprising three items rated on a 5-point Likert scale ranging from 0 (not at all) to 4 (extremely). The OCI-R demonstrates good reliabilityas well as good convergent and discriminant validity (Abramowitz & Deacon, 2006; Foa et al., 2002; Hajcak, Huppert, Simons, & Foa, 2004). Internal consistency in the present study was good for the subscales (alpha = .86 for both).

2.2.2.4. Xenophobia Scale (XS; van Zalk, Kerr, van Zalk, & Tattin, 2013)

The XS measures negative attitudes towards immigrants (e.g., “Immigrants increase criminality”) using 12 items rated on a 4-point Likert scale ranging from 1 (don’t agree at all) to 4 (agree completely). The original measure made reference to Sweden. For the current measure, all references to “Sweden” were changed to “our country” so that items were applicable to all participants. The XS has demonstrated good reliability, high temporal stability, and convergent validity (van Zalk & Kerr, 2014; van Zalk et al., 2013). One item from the Xenophobia Scale (Wilson-Daily, Kemmelmeier, & Prats, 2018), “Immigrants abuse the health system and fill up our emergency rooms,” was added to our modified XS, given its potential relevance to the COVID-19 pandemic. The addition of this item improved internal consistency of the xenophobia subscale from alpha = .86 to .90 and, therefore, was retained.

2.2.2.5. Marlowe Crowne Social Desirability Scale Short Form (MCSD-SF; Reynolds, 1982)

The MCSD-SF measures the tendency to respond in socially desirable ways using 13 items (e.g., “I sometimes try to get even rather than forgive and forget”) with “true” (1) or “false” (0) response options. The 13-item version has demonstrated to have better reliability and fit compared to the original Marlowe-Crowne Scale (Crowne & Marlowe, 1960); but, the shortened version is still highly correlated with the original scale (Fischer & Fick, 1993; Reynolds, 1982). Internal consistency for the full scale in the present study was acceptable (alpha = .72).

2.3. Scale construction and evaluation procedures

The goal was to construct short, internally consistent scales, which likely would be intercorrelated to form a coherent syndrome. Results were conducted separately for each country to determine whether the findings were robust (i.e., replicable). For each country and for each of the six scales of the CSS, an exploratory factor analysis was conducted using robust Maximum Likelihood (RML) using MPlus (Muthen & Muthen, 2017); that is, Maximum Likelihood using robust standard errors. RML was used because it is robust to departures from normality in the data distribution. For each scale, parallel analysis (Patil, Singh, Mishra, & Donavan, 2017) indicated a single factor; that is, each scale had a single underlying factor. For the factor for each scale, the items with the six highest loadings were retained. This resulted in the creation of six 6-item scales. Six items were selected because we had previously found, in studies of other anxiety-related phenomena, that 6-item subscales provided a good balance between brevity and reliability (Taylor et al., 2007). Item selection was replicated across countries; that is, for each scale, the items with the top six loadings in the Canadian sample were the same as the items with the top six loadings in the United States sample (see supplement).

Exploratory factor analysis of the resulting pool of 36 items was conducted using the Canadian sample. This was done using RML with oblique (Oblimin) rotation using MPlus. Parallel analysis was used to determine the number of factors to retain. The robustness (stability) of the multi-factor structure was then investigated by conducting RML confirmatory factor analysis using MPlus on data from the 36 items from the United States sample.

The selection of goodness-of-fit indices was based on conventional guidelines (Hu & Bentler, 1998). We used the standardized-root-mean-square residual (SRMR), the root-mean-square error of approximation (RMSEA), and the comparative fit index (CFI). The SRMR was used because it is among the most sensitive to misspecified factor correlations, and the RMSEA is sensitive to misspecified factor loadings (Hu & Bentler, 1998). To interpret whether a given factor model provided a good fit to the data, we used Hu and Bentler's (1999) empirically derived cut-off values. These values minimize errors in deciding whether a model provides a good fit to the data. Excellent fit is indicated by SRMR ≤ .08, RMSEA ≤ .06, and CFI ≥ .95. Good fit is indicated by CFI ≥ .90.

Reliability as internal consistency was assessed by Cronbach’s coefficient alpha. Correlational analyses were conducted to assess various indices of validity. Given the number of statistical analyses reported in this study, the alpha level for statistical significance was set at 0.01. To interpret the substantive significance of correlations, we used Cohen's (1988) criteria: r = 0.10 (small), 0.30 (moderate), 0.50 (large).

3. Results

3.1. Sample characterization

The study was conducted during the early stages of the pandemic in the United States and Canada, in which many people were experiencing emotional distress. Based on the cut-offs for the PHQ-4 (Kroenke et al., 2009), 28 % of our general population sample from Canada and the United States had elevated anxiety and 22 % were experiencing clinically significant depressive symptoms. For the total PHQ-4 scale (depression and anxiety), the proportions were as follows, as based on cutoffs reported by Kroenke et al.: Normal (54 %), mild symptoms (23 %), moderate symptoms (13 %), and severe symptoms (10 %). These findings are consistent with studies of responses to trauma (e.g., earthquakes, fires, floods), which show that most people are resilient to stress, although a significant minority are prone to experience stress-related psychopathology (Galatzer-Levy, Huang, & Bonanno, 2018). These findings are also consistent with studies that show approximately 25 % of the general population of China experienced moderate to severe levels of anxiety in response to COVID-19 (Qiu et al., 2020; Wang et al., 2020).

3.2. Exploratory factor analysis

Parallel analysis indicated a 5-factor solution, rather than a 6-factor solution in which each factor corresponded to each of the six scales of the CSS. That is, two of the subscales loaded on a single factor (see below). The first six eigenvalues were as follows: 15.84, 2.86, 2.06, 1.58, 1.55, and 0.88. The factors were correlated 0.29 to 0.49 with one another (see supplement for the full correlation matrix). The factors corresponded to (1) COVID danger and contamination fears, (2) COVID fears about economic consequences, (3) COVID xenophobia, (4) COVID compulsive checking and reassurance seeking, and (5) COVID traumatic stress symptoms. Factor loadings are shown in Table 1 . The 5-factor solution had an excellent simple structure; that is, each item had a salient loading on only one factor. Each factor corresponded to one of the scales, with the exception that the COVID-related danger and COVID-related contamination scales loaded on a single factor. Rather than reducing this factor to a 6-item scale, it was retained as a 12-item scale so that, if needed in future studies, it would be possible to assess danger separately from contamination.

Table 1.

Exploratory factor analysis (Canadian sample): Factor loadings.

Item	Scale	I	II	III	IV	V
I am worried about catching the virus	D	0.67	0.11	0.00	0.13	0.06
I am worried that I can’t keep my family safe from the virus	D	0.54	0.22	0.01	0.15	−0.01
I am worried that our healthcare system won’t be able to protect my loved ones	D	0.54	0.24	0.04	0.11	−0.08
I am worried that our healthcare system is unable to keep me safe from the virus	D	0.51	0.23	0.09	0.12	−0.07
I am worried that basic hygiene (e.g., handwashing) is not enough to keep me safe from the virus	D	0.51	0.23	0.07	0.15	0.01
I am worried that social distancing is not enough to keep me safe from the virus	D	0.50	0.19	0.11	0.13	0.04
I am worried about grocery stores running out of food	SE	0.05	0.80	0.02	0.03	−0.01
I am worried that grocery stores will close down	SE	0.02	0.72	0.03	0.05	−0.01
I am worried about grocery stores running out of cleaning or disinfectant supplies	SE	0.19	0.61	0.05	0.02	0.06
I am worried about grocery stores running out of cold or flu remedies	SE	0.07	0.61	0.09	0.06	0.15
I am worried about grocery stores running out of water	SE	−0.10	0.60	0.15	0.05	0.19
I am worried about pharmacies running out of prescription medicines	SE	0.15	0.58	0.06	0.08	0.04
I am worried that foreigners are spreading the virus in my country	X	−0.03	0.07	0.83	0.03	−0.02
If I went to a restaurant that specialized in foreign foods, I’d be worried about catching the virus	X	−0.13	0.10	0.79	0.04	0.03
I am worried about coming into contact with foreigners because they might have the virus	X	0.12	0.00	0.78	0.04	0.04
If I met a person from a foreign country, I’d be worried that they might have the virus	X	0.12	0.01	0.75	0.05	0.03
If I was in an elevator with a group of foreigners, I’d be worried that they’re infected with the virus	X	0.29	0.00	0.63	−0.02	0.02
I am worried that foreigners are spreading the virus because they’re not as clean as we are	X	0.09	0.10	0.59	0.06	0.12
I am worried that if I touched something in a public space (e.g., handrail, door handle), I would catch the virus	C	0.68	0.02	0.11	0.06	0.12
I am worried that if someone coughed or sneezed near me, I would catch the virus	C	0.64	0.04	0.16	0.04	0.06
I am worried that people around me will infect me with the virus	C	0.60	0.09	0.15	0.11	0.07
I am worried about taking change in cash transactions	C	0.57	0.01	0.12	0.05	0.17
I am worried that I might catch the virus from handling money or using a debit machine	C	0.56	0.07	0.14	0.07	0.15
I am worried that my mail has been contaminated by mail handlers	C	0.41	0.14	0.22	0.02	0.19
I had trouble concentrating because I kept thinking about the virus	T	0.09	0.00	0.00	0.85	0.01
Disturbing mental images about the virus popped into my mind against my will	T	−0.01	0.03	0.07	0.80	0.07
I had trouble sleeping because I worried about the virus	T	0.07	0.05	0.00	0.75	0.04
I thought about the virus when I didn’t mean to	T	0.16	0.03	−0.03	0.75	−0.01
Reminders of the virus caused me to have physical reactions, such as sweating or a pounding heart	T	−0.03	0.04	0.07	0.74	0.11
I had bad dreams about the virus	T	−0.11	0.06	0.06	0.69	0.15
Searched the Internet for treatments for COVID-19	CH	0.00	0.09	0.08	0.06	0.67
Asking health professionals (e.g., doctors or pharmacists) for advice about COVID-19	CH	−0.11	0.08	0.06	0.10	0.66
YouTube videos about COVID-19	CH	0.03	0.00	0.02	0.04	0.65
Checking your own body for signs of infection (e.g., taking your temperature)	CH	0.12	0.03	0.01	0.09	0.59
Seeking reassurance from friends or family about COVID-19	CH	0.13	0.02	−0.06	0.15	0.55
Social media posts concerning COVID-19	CH	0.24	−0.03	−0.09	0.04	0.46

Bold = salient (> .30) loading. D = danger, SE = socio-economic consequences, X = xenophobia, C = contamination, T = traumatic stress, CH = compulsive checking.

3.3. Confirmatory factor analysis

The 5-factor model, obtained in the exploratory factor analysis from the Canadian sample, was tested in RML confirmatory factor analysis in the United States sample. In the latter sample, the model performed well in terms of the goodness-of-fit indicates: RMSEA = 0.050 (90 % confidence interval: 0.049−0.051), SRMR = 0.042, and CFI = .93. To further test the robustness (replicability) of the factor structure, a 2-group confirmatory factor analysis was conducted in which the 5-factor model was simultaneously fitted to the United States and Canadian samples, with factor loadings and factor correlations constrained to be the same for each sample. The models performed well in goodness-of-fit: RMSEA = 0.050 (90 % confidence interval: 0.049−0.051), SRMR = 0.053, and CFI = .92. That is, constraining loadings and correlations to be equal across samples yielded a factor model that had a good fit to the data, indicating that the factor structure was replicable across countries. For the two samples, the correlations among factors ranged from .48 to .77 (see supplement for the complex matrix of correlations). The factors loaded on a single higher-order factor. The present study focuses on the psychometric properties of the lower-order factors.

3.4. Internal consistency of scales

Based on the results of the factor analysis, five scales were constructed to form the CSS. The complete CSS appears in the supplement. The scales were scored by adding the unit-weighted items together. Higher scores indicate greater levels of COVID-19-related distress. Items were unit-weighted instead of being weighted according to factor score coefficients, because unit-weightings are more likely to be reliable (i.e., replicable) in future studies (Cohen, 1990). Table 2 presents Cronbach alpha coefficients for each scale in each sample. Here, it can be seen that all of the coefficients were > .80, indicating good-to-excellent reliability as internal consistency (Tavakol & Dennick, 2011). Table 3 presents the correlations among the scales. Here, it can be seen that all the scales of the CSS were intercorrelated. This suggests, for people with high scores, that symptoms assessed in the CSS form a coherent COVID Stress Syndrome.

Table 2.

Reliability as internal consistency: Cronbach alphas.

	Canadian sample	U.S. sample
COVID danger and contamination	0.94	0.95
COVID socioeconomic consequences	0.90	0.91
COVID xenophobia	0.92	0.93
COVID traumatic stress	0.93	0.93
COVID compulsive checking	0.83	0.86

Table 3.

Correlations among the COVID Stress Scales: Canadian (and U.S.) samples.

	1	2	3	4
1. COVID danger and contamination	–
2. COVID socioeconomic consequences	.71 (.73)	–
3. COVID xenophobia	.65 (.66)	.58 (.60)	–
4. COVID traumatic stress symptoms	.62 (.62)	.55 (.57)	.43 (.48)	–
5. COVID compulsive checking	.53 (.54)	.49 (.53)	.41 (.48)	.58 (.63)

All p < .001.

3.5. Convergent validity

Table 4 shows the correlations of the scales of CSS with the pre-COVID trait measures of health anxiety and obsessive-compulsive (OC) contamination and checking symptoms. Here, it can be seen that all correlations were significant (p < .001) and almost all were medium-to-large in magnitude. These findings support the convergent validity of the CSS.

Table 4.

Tests of convergent validity: Correlations of the COVID Stress Scales with pre-COVID health anxiety and obsessive-compulsive checking and contamination symptoms.

	COVID danger and contamination	COVID socioeconomic consequences	COVID xenophobia	COVID traumatic stress symptoms	COVID compulsive checking
Canadian Sample
Pre-COVID health anxiety	0.41	0.37	0.29	0.43	0.31
Pre-COVID OC checking	0.37	0.40	0.35	0.38	0.41
Pre-COVID OC contamination	0.42	0.42	0.42	0.38	0.45
U.S. Sample
Pre-COVID health anxiety	0.47	0.42	0.34	0.48	0.40
Pre-COVID OC checking	0.43	0.44	0.43	0.44	0.46
Pre-COVID OC contamination	0.46	0.45	0.46	0.42	0.48

Correlations ≥0.30 in bold. For all r, p < .001. OC = obsessive-compulsive.

3.6. Discriminant validity

Due to the large sample sizes, the correlations between the five scales of the CSS and social desirability were statistically significant for each country (p < .001); but, they were substantively trivial in their absolute values, smaller than Cohen’s classification of “small” correlation (i.e., the correlations ranged from -.14 to -.05). This finding indicates that a socially desirable response set was essentially unrelated to scores on the CSS.

Table 5 shows the tests of the differences between correlations with general anxiety versus depression for each of the scales of the CSS. This is a highly stringent and, in some ways, contentious test of discriminant validity, in part because if a given variable leads to anxiety, depression is a common consequence. Accordingly, for many measures of anxiety-related symptoms (e.g., OC symptoms), it has historically been very difficult to show that they are more strongly correlated with anxiety than depression (e.g., Taylor, 1995). Nevertheless, Table 5 shows that for most scales of the CSS, the correlations with current anxiety were significantly greater than the correlations with current depression. This finding supports the discriminant validity of the CSS.

Table 5.

Tests of discriminant validity: Comparison of correlations with current anxiety and depression.

	Current anxiety	Current depression	Significance of difference between rs: Z
Canadian Sample
COVID danger and contamination	0.50	0.44	5.39***
COVID socioeconomic consequences	0.40	0.40	0.00
COVID xenophobia	0.27	0.28	0.81
COVID traumatic stress symptoms	0.62	0.57	5.03***
COVID compulsive checking	0.39	0.35	3.38***
U.S. Sample
COVID danger and contamination	0.49	0.42	6.48***
COVID socioeconomic consequences	0.45	0.42	2.74**
COVID xenophobia	0.30	0.29	0.85
COVID traumatic stress symptoms	0.59	0.54	5.07***
COVID compulsive checking	0.39	0.37	1.77

Correlations ≥0.30 in bold. *p < .01, **p < .005, 8**p < .001.

Table 6 provides further support for discriminant validity. Here, the mean correlations for general distress (i.e., correlations of the CSS with the measures of general anxiety, depression, and the pre-COVID trait measures of health anxiety and OC symptoms) were compared to the correlations with general, pre-COVID xenophobia as measured using the XS. These correlations were conducted for each of the scales of the CSS. Table 6 shows that the pattern of correlations supports the discriminant validity of the CSS. That is, the COVID xenophobia scale was more strongly correlated with the general xenophobia scale versus the mean of the distress scales. Table 6 shows that the converse pattern was observed for the other CSS scales. Those scales were more strongly correlated with distress than general xenophobia. In addition, the COVID xenophobia scale, as compared to the other CSS scales, was more strongly correlated with general xenophobia: Canadian sample, Z = 26.75, p < .001; United States sample, Z = 23.46, p < .001.

Table 6.

Tests of discriminant validity: Comparison of the mean correlations for distress measures versus correlation with general xenophobia.

	Mean r: distress measures	r: general xenophobia	Significance of difference between rs: Z
Canadian Sample
COVID danger and contamination	0.43	0.18	11.87
COVID socioeconomic consequences	0.40	0.22	8.20
COVID xenophobia	0.33	0.53	10.57
COVID traumatic stress symptoms	0.49	0.07	19.85
COVID compulsive checking	0.38	0.04	15.92
U.S. Sample
COVID danger and contamination	0.46	0.13	15.33
COVID socioeconomic consequences	0.43	0.15	13.33
COVID xenophobia	0.36	0.48	5.93
COVID traumatic stress symptoms	0.50	0.14	17.40
COVID compulsive checking	0.42	0.09	15.30

Correlations ≥0.30 in bold. For all Z, p < .001.

4. Discussion

The CSS were developed and initially validated in large, population-representative samples from Canada and the United States. A stable 5-factor solution was identified, corresponding to five scales assessing COVID-19 stress and anxiety symptoms: (1) COVID danger and contamination fears, (2) COVID fears about economic consequences, (3) COVID xenophobia, (4) COVID compulsive checking and reassurance seeking, and (5) COVID traumatic stress symptoms. The scales performed well on various indices of reliability and validity. The scales were intercorrelated, loading on a single higher-order factor, thereby providing evidence of a COVID-19 Stress Syndrome. The scales offer promise as tools for better understanding the distress associated with COVID-19 and for identifying people in need of mental health services. The CSS can also be used in studies to predict which people are most likely to engage in safety behaviors. For example, the CSS could be used to investigate which people are most likely to engage in hygiene behaviors, social distancing, and the uptake of a vaccine, when one becomes available.

It is anticipated that when this pandemic passes, significant mental health needs will emerge in the public. These predictions are based on prior pandemics, where anxiety, depression, and traumatic reactions were observed (such as following quarantine due to SARS; Hawryluck et al., 2004; Taylor, 2019). Accordingly, the development of a pandemic-specific measure such as the CSS can serve to aid in identifying individuals at risk for adverse emotional reactions both during and post-pandemic. This can then aid public health officials in allocating resources for mental health interventions. The measure can also be further investigated for its predictive utility for returning to functioning post-pandemic. This is also, to our knowledge, the first assessment of pandemic-related emotional responses that includes specific evaluation of xenophobia as a contributory factor in fear and avoidance. This is an important consideration in any preparation for public health officials in addressing emotional and behavioral responses to potential pandemics. The role of xenophobic reactions, in conjunction with other emotional indicators such as those identified in the CSS (i.e., COVID danger and contamination, COVID socioeconomic consequences, COVID traumatic stress, and COVID checking and reassurance seeking) deserves additional investigation, such as with individuals at risk for post-pandemic adverse reactions.

In terms of limitations, the present study did not include structured diagnostic assessments (i.e., DSM-5 or ICD-11 diagnoses), which would have been useful in evaluating criterion-related (known-groups) validity of the CSS. It would be predicted that scores on the CSS would be higher in people with current anxiety-related disorders (e.g., generalized anxiety disorder, OCD), as compared to nonclinical controls. Future research is needed to investigate this issue. A further limitation was that we were unable to compare the CSS to other COVID-related anxiety measures (e.g., Ahorsu et al., 2020), because the latter scales had not been published when we were constructing our study. Despite these limitations, the present findings provide encouraging support for the CSS as a brief, multidimensional measure of COVID-related stress and anxiety.

This study is also limited by the reliance on an online survey method of evaluation and self-report measurement. Additional research involving expanded breadth of content may provide incremental increases in the validity of the CSS. Likewise, additional methods of assessment, such as interviews to evaluate the scope of avoidance and the inclusion of new safety behaviors, may reveal additional indicators of emotional responses to pandemic. Notwithstanding these limitations, the introduction of a robust instrument to assess COVOD-19-related stress reactions, developed and evaluated with a large bi-national community sample during the peak period of COVID-19, is of considerable importance. It is expected that the CSS will lead to important new empirical findings on the nature of reactions to COVID-19 in particular and future pandemics in general.

Author note

Drs. Taylor and Asmundson contributed equally to this article. This research was supported by funding from the Canadian Institutes of Health Research and the University of Regina. Dr. Taylor receives financial support through royalties from various book publishers and from editorial duties as Associate Editor of the Journal of Obsessive-Compulsive and Related Disorders. Dr. Asmundson is the Editor-in-Chief of the Journal of Anxiety Disorders and Development Editor of Clinical Psychology Review. He receives financial support through payments for his editorial work on the aforementioned journals and royalties from various book publishers. Dr. Asmundson was blinded to the review process for this article. Dr. McKay is Associate Editor of Behavior Therapy and the Bulletin of the Menninger Clinic. He receives royalties through his editorial work on the aforementioned journals, as well as from various book publishers. Dr. McKay also receives funding from a private investment company for the development of technology assisted methods for the treatment of fear of public speaking.

Appendix A. Supplementary data

The following are Supplementary data to this article: