Abstract
Objective
To examine the psychometric properties of three measures, the Perceived Stress Scale (PSS), Pregnancy Experience Scale (PES), and State Trait Anxiety Inventory (STAI), for assessing stress and anxiety during pregnancy among a large sample of nulliparous women.
Methods
The sample included 10,002 pregnant women participating in the Nulliparous Pregnancy Outcomes Study: Monitoring Mothers-to-Be (nMoM2b). Internal consistency reliability was assessed with Cronbach’s alpha and factorial validity with confirmatory factor analyses. Intraclass correlations (ICCs) were calculated to determine stability of PSS scales over time. Psychometric properties were examined for the overall sample, as well as subgroups based on maternal age, race/ethnicity, and language.
Results
All three scales demonstrated good internal consistency reliability. Confirmatory factor analyses supported the factor structures of the PSS and the PES. However, a one-factor solution of the trait-anxiety subscale from the STAI did not fit well; a two-factor solution, splitting the items into factors based on direction of item wording (positive vs. negative) provided a better fit. Scores on the PSS were generally stable over time (ICC=0.60). Subgroup analyses revealed a few items that did not perform well on Spanish versions of the scales.
Conclusion
Overall, the scales performed well, suggesting they could be useful tools for identifying women experiencing high levels of stress and anxiety during pregnancy and allowing for the implementation of interventions to help reduce maternal stress and anxiety.
Keywords: Stress, anxiety, parity, psychometrics
INTRODUCTION
Stress and anxiety during pregnancy have been associated with adverse outcomes among mothers and their children. Mothers experiencing greater antenatal stress are more likely to deliver babies preterm and at lower birth weights [1–4]. In turn, children born preterm or at low birth weight are at increased risk for poor neurodevelopmental and health outcomes [5,6]. Antenatal anxiety is likely to continue following delivery [7] and significantly increases risk for postpartum depression in the mother [8–10], as well as for emotional and behavioral problems in the child at 4 years of age [11].
Reliable and valid measures are needed for use in clinical settings to identify pregnant women experiencing high levels of stress and anxiety. Women experiencing their first pregnancy (i.e., nulliparous women) may be particularly vulnerable to stress and anxiety as they transition to new motherhood and incorporate their role as mother into other areas of their lives. A study of first-time mothers found that women began the transition to motherhood early in their pregnancy when they reported feeling a loss of control over their lives [12]. In a qualitative study of women’s work lives, Messias and colleagues [13] found that “a first pregnancy transformed women’s work as they added pregnancy to the mix and experienced significant shifts and transformations in personal and social identities and in the meanings, values, and priorities they attached to work” (p. 41). In addition, women pregnant with their first child were more likely to experience reductions in satisfaction with their roles over the transition to motherhood compared with women having their second child [14].
Nulliparous women do not have prior pregnancy experience to draw on; therefore, their pregnancy experiences and feelings of stress and anxiety during this time period may be different from those experienced by women who have already gone through a pregnancy. DiPietro and colleagues [15] found differing trajectories of stress and anxiety during pregnancy and postpartum based on parity. Ideally, the psychometric properties of a measure should be evaluated among the intended population. However, studies of the reliability and validity of stress and anxiety measures for use in pregnancy have not focused on nulliparous women. A review of stress measures in pregnancy identified only two measures that had been assessed among nulliparous women [16]. A small study of 94 nulliparous women assessed the Pregnancy Experience Scale [17], and a few small studies (sample sizes of 112 to 230) examined properties of the Pregnancy-Related Anxiety Questionnaire among Dutch-speaking women [18–20].
The Nulliparous Pregnancy Outcomes Study: Monitoring Mothers-to-Be (nuMoM2b) is a large prospective cohort study of approximately 10,000 pregnant nulliparous women sponsored by the Eunice Kennedy Shriver National Institute of Child Health and Human Development. The goal of the current analysis was to examine the psychometric properties of two stress measures, the Perceived Stress Scale (PSS) [21] and the Pregnancy Experience Scale (PES) [17], and one anxiety measure, the trait-anxiety subscale from the State Trait Anxiety Inventory (STAI) [22], among the large sample of nulliparous women participating in the nuMoM2b study. In addition, we examined properties of these measures among subgroups based on age, race/ethnicity, and language.
METHODS
Study Design
The nuMoM2b is a prospective cohort study designed to identify maternal, fetal, and placental growth parameters associated with adverse pregnancy outcomes. Data were collected by interviews, self-administered questionnaires, clinical measurements, ultrasounds, medical records review, and biospecimen collections through four study visits: Visit 1 (6 weeks 0 days through 13 weeks 6 days gestation), Visit 2 (16 weeks 0 days through 21 weeks 6 days gestation), Visit 3 (22 weeks 0 days through 29 weeks 6 days gestation), and Visit 4 (time of delivery). Study protocols and procedures were approved by each participating center’s institutional review board.
Participants
Participants were recruited through eight academic medical centers in the United States between October 2010 and September 2013. Participants were eligible for this observational cohort study if they had a viable singleton gestation, were between 6 weeks 0 days and 13 weeks 6 days of pregnancy based on ultrasound, and had not had a previous pregnancy lasting 20 or more weeks. Women were excluded if they were less than 13 years of age, had 3 or more spontaneous abortions, participated in a prior study that may impact outcomes, planned to terminate the pregnancy, and/or could not provide informed consent or if there was likely fetal malformation, known fetal aneuploidy, and/or the pregnancy was due to assisted reproduction with donor egg. A total of 10,038 women were enrolled in the nuMoM2b study. The current analyses were restricted to 10,002 of these women who completed at least one of the three measures. Participant demographic characteristics are shown in Table 1.
Table 1.
Characteristic | N | % |
---|---|---|
Age | ||
13–17 | 244 | 2 |
18–24 | 3,344 | 33 |
25–29 | 2,922 | 29 |
30–34 | 2,562 | 26 |
35–39 | 786 | 8 |
40+ | 144 | 1 |
Race/Ethnicity | ||
Non-Hispanic white | 5,987 | 60 |
Non-Hispanic black | 1,416 | 14 |
Hispanic | 1,679 | 17 |
Asian | 407 | 4 |
Other | 513 | 5 |
Language | ||
English | 9,783 | 98 |
Spanish | 218 | 2 |
Education | ||
Less than high school graduate | 816 | 8 |
High school graduate/GED | 1,162 | 12 |
Some college | 1,943 | 19 |
Associate/technical degree | 1,004 | 10 |
Bachelor’s degree | 2,767 | 28 |
Graduate degree | 2,308 | 23 |
Annual household income (% of Federal poverty level) | ||
High (> 200%) | 5,664 | 57 |
Medium (100–200%) | 1,170 | 12 |
Low (< 100%) | 1,294 | 13 |
Not reported | 1,874 | 19 |
Measures
The stress and anxiety-related measures were administered during interviews at the clinics.
Perceived Stress Scale
The PSS is a general measure of stress that captures how often individuals felt stress in their daily lives within the past month [21]. The scale contains 10 items with response options ranging from 1 (never) to 5 (very often) and was administered at Visits 1 and 3 (Table 2). Items 4, 5, 7, and 8 of the scale were reverse coded for analysis, so higher values indicate more stress, consistent with other items.
Table 2.
Factor Loadings | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
||||||||||
Age | Race/Ethnicity | Language | ||||||||
|
||||||||||
Item | Mean (SD) | All | < 30 | 30+ | Black | White | Hispanic | Asian | English | Spanish |
Visit 1 (8–13 weeks) (N=9,989) | ||||||||||
1. Upset because of something that happened unexpectedly | 2.61 (1.00) | 0.70 | 0.71 | 0.66 | 0.69 | 0.70 | 0.68 | 0.68 | 0.70 | 0.61 |
2. Unable to control the important things in your life | 2.15 (1.05) | 0.72 | 0.72 | 0.71 | 0.72 | 0.73 | 0.70 | 0.67 | 0.72 | 0.72 |
3. Nervous and stressed | 3.00 (0.98) | 0.66 | 0.67 | 0.66 | 0.71 | 0.68 | 0.68 | 0.65 | 0.66 | 0.65 |
4. Confident about ability to handle personal problems | 1.85 (0.87) | 0.48 | 0.47 | 0.50 | 0.40 | 0.55 | 0.37 | 0.36 | 0.49 | 0.22 |
5. Felt things were going your way | 2.15 (0.92) | 0.60 | 0.58 | 0.59 | 0.52 | 0.64 | 0.52 | 0.42 | 0.60 | 0.50 |
6. Could not cope with all of the things that you had to do | 2.08 (0.97) | 0.67 | 0.66 | 0.66 | 0.68 | 0.68 | 0.57 | 0.62 | 0.68 | 0.40 |
7. Able to control irritations in your life | 2.31 (0.95) | 0.51 | 0.49 | 0.51 | 0.42 | 0.54 | 0.42 | 0.43 | 0.52 | 0.22 |
8. Felt on top of things | 2.19 (0.87) | 0.55 | 0.53 | 0.59 | 0.48 | 0.63 | 0.43 | 0.43 | 0.56 | 0.32 |
9. Angered because of things outside of your control | 2.63 (1.04) | 0.69 | 0.71 | 0.63 | 0.69 | 0.67 | 0.71 | 0.64 | 0.69 | 0.76 |
10. Felt difficulties were piling up so high that you could not overcome them | 1.87 (0.97) | 0.76 | 0.76 | 0.73 | 0.75 | 0.76 | 0.74 | 0.65 | 0.76 | 0.68 |
Model Fit Indices | ||||||||||
Comparative Fit Index (CFI) | — | 0.95 | 0.95 | 0.93 | 0.95 | 0.95 | 0.94 | 0.91 | 0.95 | 0.91 |
Tucker-Lewis Fit Index (TLI) | — | 0.92 | 0.93 | 0.90 | 0.93 | 0.93 | 0.91 | 0.87 | 0.92 | 0.88 |
Standardized Root Mean Square Residual (SRMR) | — | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.05 | 0.06 | 0.04 | 0.06 |
Cronbach’s alpha | — | 0.88 | 0.88 | 0.87 | 0.86 | 0.89 | 0.85 | 0.83 | 0.88 | 0.79 |
| ||||||||||
Visit 3 (22–29 weeks) (N=9,202) | ||||||||||
1. Upset because of something that happened unexpectedly | 2.37 (0.95) | 0.73 | 0.74 | 0.71 | 0.71 | 0.73 | 0.72 | 0.71 | 0.73 | 0.64 |
2. Unable to control the important things in your life | 1.97 (0.98) | 0.73 | 0.72 | 0.75 | 0.72 | 0.74 | 0.71 | 0.77 | 0.73 | 0.69 |
3. Nervous and stressed | 2.77 (0.95) | 0.69 | 0.69 | 0.69 | 0.71 | 0.71 | 0.69 | 0.72 | 0.70 | 0.53 |
4. Confident about ability to handle personal problems | 1.83 (0.87) | 0.49 | 0.46 | 0.54 | 0.32 | 0.56 | 0.43 | 0.43 | 0.49 | 0.35 |
5. Felt things were going your way | 2.03 (0.89) | 0.60 | 0.58 | 0.63 | 0.54 | 0.64 | 0.52 | 0.52 | 0.61 | 0.41 |
6. Could not cope with all of the things that you had to do | 1.96 (0.92) | 0.67 | 0.66 | 0.67 | 0.61 | 0.68 | 0.65 | 0.61 | 0.66 | 0.77 |
7. Able to control irritations in your life | 2.23 (0.95) | 0.52 | 0.50 | 0.56 | 0.45 | 0.56 | 0.42 | 0.42 | 0.53 | 0.17 |
8. Felt on top of things | 2.08 (0.84) | 0.60 | 0.58 | 0.62 | 0.51 | 0.65 | 0.52 | 0.40 | 0.60 | 0.30 |
9. Angered because of things outside of your control | 2.45 (1.00) | 0.71 | 0.73 | 0.67 | 0.73 | 0.69 | 0.72 | 0.74 | 0.71 | 0.59 |
10. Felt difficulties were piling up so high that you could not overcome them | 1.75 (0.90) | 0.76 | 0.77 | 0.73 | 0.76 | 0.76 | 0.77 | 0.73 | 0.76 | 0.83 |
Model Fit Indices | ||||||||||
Comparative Fit Index (CFI) | — | 0.95 | 0.94 | 0.95 | 0.93 | 0.96 | 0.93 | 0.88 | 0.95 | 0.94 |
Tucker-Lewis Fit Index (TLI) | — | 0.92 | 0.92 | 0.93 | 0.90 | 0.94 | 0.91 | 0.84 | 0.92 | 0.91 |
Standardized Root Mean Square Residual (SRMR) | — | 0.04 | 0.04 | 0.04 | 0.05 | 0.03 | 0.05 | 0.07 | 0.04 | 0.07 |
Cronbach’s alpha | — | 0.89 | 0.88 | 0.89 | 0.86 | 0.90 | 0.87 | 0.86 | 0.89 | 0.82 |
Pregnancy Experience Scale
The brief version of the PES [17] was administered at Visit 3. The PES contains two subscales, the Uplifts subscale, which asks women about how often 10 specific experiences make them feel “happy, positive, or uplifted,” and the Hassles subscale, which asks women about 10 experiences that may make them feel “unhappy, negative, or upset” (Table 3). The response options for each item range from 1 (not at all) to 4 (a great deal).
Table 3.
Factor Loadings | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
||||||||||
Age | Race/Ethnicity | Language | ||||||||
|
||||||||||
Item | Mean (SD) | All | < 30 | 30+ | Black | White | Hispanic | Asian | English | Spanish |
Visit 3 (22–29 weeks) (N=8,801) | ||||||||||
Factor 1: Uplifts | ||||||||||
1. How much the baby is moving | 3.67 (0.57) | 0.49 | 0.47 | 0.53 | 0.42 | 0.52 | 0.45 | 0.47 | 0.49 | 0.40 |
2. Discussions with spouse about baby names | 3.19 (0.91) | 0.59 | 0.58 | 0.59 | 0.55 | 0.62 | 0.54 | 0.64 | 0.59 | 0.42 |
3. Comments from others about your pregnancy/appearance | 3.03 (0.89) | 0.65 | 0.66 | 0.65 | 0.68 | 0.64 | 0.66 | 0.63 | 0.65 | 0.58 |
4. Making or thinking about nursery arrangements | 3.09 (0.92) | 0.67 | 0.67 | 0.65 | 0.70 | 0.65 | 0.67 | 0.67 | 0.67 | 0.64 |
5. Feelings about being pregnant at this time | 3.25 (0.83) | 0.73 | 0.73 | 0.72 | 0.74 | 0.73 | 0.75 | 0.72 | 0.72 | 0.80 |
6. Visits to obstetrician/midwife | 2.94 (0.92) | 0.69 | 0.70 | 0.69 | 0.69 | 0.70 | 0.70 | 0.68 | 0.69 | 0.73 |
7. Spiritual feelings about being pregnant | 2.86 (1.07) | 0.68 | 0.69 | 0.65 | 0.75 | 0.64 | 0.74 | 0.73 | 0.68 | 0.74 |
8. Courtesy/assistance from others because you are pregnant | 2.88 (0.95) | 0.68 | 0.69 | 0.65 | 0.74 | 0.66 | 0.72 | 0.68 | 0.68 | 0.69 |
9. Thinking about the baby’s appearance | 3.36 (0.81) | 0.65 | 0.63 | 0.68 | 0.61 | 0.66 | 0.64 | 0.72 | 0.65 | 0.60 |
10. Discussions with spouse about pregnancy/childbirth issues | 2.89 (0.99) | 0.61 | 0.61 | 0.63 | 0.63 | 0.61 | 0.59 | 0.64 | 0.62 | 0.34 |
Factor 2: Hassles | ||||||||||
1. Getting enough sleep | 1.88 (0.81) | 0.53 | 0.53 | 0.51 | 0.52 | 0.53 | 0.53 | 0.49 | 0.53 | 0.53 |
2. Physical intimacy | 1.67 (0.79) | 0.49 | 0.50 | 0.49 | 0.49 | 0.49 | 0.50 | 0.54 | 0.49 | 0.52 |
3. Normal discomforts of pregnancy | 1.94 (0.81) | 0.56 | 0.55 | 0.60 | 0.50 | 0.59 | 0.55 | 0.62 | 0.57 | 0.42 |
4. Your weight | 1.90 (0.86) | 0.59 | 0.59 | 0.58 | 0.66 | 0.57 | 0.61 | 0.55 | 0.59 | 0.68 |
5. Body changes due to pregnancy | 1.88 (0.82) | 0.64 | 0.64 | 0.63 | 0.67 | 0.64 | 0.64 | 0.57 | 0.64 | 0.69 |
6. Thoughts about whether the baby is normal | 1.85 (0.84) | 0.58 | 0.59 | 0.56 | 0.62 | 0.56 | 0.60 | 0.62 | 0.58 | 0.79 |
7. Thinking about your labor and delivery | 2.09 (0.92) | 0.63 | 0.64 | 0.61 | 0.68 | 0.60 | 0.69 | 0.63 | 0.63 | 0.79 |
8. Ability to do physical tasks/chores | 1.77 (0.80) | 0.66 | 0.68 | 0.63 | 0.66 | 0.65 | 0.67 | 0.70 | 0.66 | 0.73 |
9. Concerns about physical symptoms | 1.53 (0.74) | 0.58 | 0.59 | 0.57 | 0.56 | 0.57 | 0.61 | 0.67 | 0.58 | 0.46 |
10. Clothes/shoes don’t fit | 1.76 (0.85) | 0.62 | 0.63 | 0.60 | 0.65 | 0.61 | 0.62 | 0.60 | 0.62 | 0.63 |
Model Fit Indices | ||||||||||
Comparative Fit Index (CFI) | — | 0.93 | 0.94 | 0.92 | 0.94 | 0.92 | 0.93 | 0.91 | 0.93 | 0.91 |
Tucker-Lewis Fit Index (TLI) | — | 0.93 | 0.93 | 0.91 | 0.94 | 0.91 | 0.92 | 0.90 | 0.92 | 0.90 |
Standardized Root Mean Square Residual (SRMR) | — | 0.04 | 0.03 | 0.04 | 0.04 | 0.04 | 0.04 | 0.05 | 0.04 | 0.07 |
Cronbach’s alpha | ||||||||||
Uplifts | — | 0.88 | 0.87 | 0.87 | 0.88 | 0.87 | 0.88 | 0.88 | 0.88 | 0.84 |
Hassles | — | 0.85 | 0.85 | 0.84 | 0.85 | 0.84 | 0.86 | 0.86 | 0.85 | 0.87 |
State Trait Anxiety Inventory (Trait-Anxiety Subscale)
Participants responded to the trait-anxiety (Y-2) subscale of the STAI [22] at the first visit. The trait-anxiety subscale assesses components of anxiety that are expected to persist over time. Participants respond to a series of 20 statements, using response options of 1 (almost never) to 4 (almost always) (Table 4). Items 1, 3, 6, 7, 10, 13, 14, 16, and 19 were reverse coded for analysis, so higher values indicate greater anxiety, consistent with the other items; items were then summed to compute the scale score.
Table 4.
Factor Loadings | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
||||||||||
Age | Race/Ethnicity | Language | ||||||||
|
||||||||||
Item | Mean (SD) | All | < 30 | 30+ | Black | White | Hispanic | Asian | English | Spanish |
Visit 1 (8–13 weeks) (N=8,723) | ||||||||||
Factor 1: Anxiety | ||||||||||
1. Pleasant | 1.77 (0.74) | 0.67 | 0.67 | 0.66 | 0.69 | 0.67 | 0.65 | 0.69 | 0.68 | 0.53 |
2. Nervous and restless | 1.79 (0.70) | 0.49 | 0.47 | 0.52 | 0.40 | 0.53 | 0.42 | 0.52 | 0.49 | 0.41 |
3. Satisfied with myself | 1.82 (0.78) | 0.71 | 0.70 | 0.73 | 0.69 | 0.73 | 0.69 | 0.72 | 0.71 | 0.56 |
4. Wish I could be as happy as others | 1.60 (0.82) | 0.55 | 0.55 | 0.55 | 0.49 | 0.61 | 0.44 | 0.39 | 0.58 | 0.10 |
5. Feel like a failure | 1.22 (0.51) | 0.53 | 0.54 | 0.50 | 0.49 | 0.57 | 0.48 | 0.41 | 0.54 | 0.39 |
6. Rested | 2.51 (0.79) | 0.49 | 0.48 | 0.49 | 0.48 | 0.50 | 0.45 | 0.53 | 0.49 | 0.44 |
7. Calm, cool, and collected | 2.07 (0.78) | 0.65 | 0.65 | 0.64 | 0.68 | 0.65 | 0.64 | 0.64 | 0.65 | 0.54 |
8. Difficulties are piling up | 1.40 (0.62) | 0.54 | 0.55 | 0.50 | 0.48 | 0.56 | 0.51 | 0.44 | 0.54 | 0.49 |
9. Worry too much | 1.82 (0.77) | 0.49 | 0.49 | 0.48 | 0.44 | 0.51 | 0.48 | 0.47 | 0.49 | 0.49 |
10. Happy | 1.54 (0.70) | 0.73 | 0.73 | 0.74 | 0.72 | 0.74 | 0.70 | 0.72 | 0.73 | 0.60 |
11. Disturbing thoughts | 1.24 (0.54) | 0.38 | 0.38 | 0.37 | 0.37 | 0.37 | 0.39 | 0.46 | 0.38 | 0.43 |
12. Lack self-confidence | 1.56 (0.70) | 0.50 | 0.49 | 0.52 | 0.43 | 0.53 | 0.51 | 0.52 | 0.49 | 0.63 |
13. Secure | 1.69 (0.79) | 0.70 | 0.69 | 0.73 | 0.67 | 0.72 | 0.68 | 0.73 | 0.70 | 0.74 |
14. Make decisions easily | 2.16 (0.85) | 0.55 | 0.54 | 0.57 | 0.56 | 0.55 | 0.55 | 0.57 | 0.55 | 0.47 |
15. Inadequate | 1.40 (0.64) | 0.47 | 0.44 | 0.54 | 0.27 | 0.54 | 0.39 | 0.43 | 0.48 | 0.28 |
16. Content | 1.72 (0.79) | 0.74 | 0.72 | 0.76 | 0.67 | 0.78 | 0.66 | 0.74 | 0.74 | 0.66 |
17. Unimportant thoughts | 1.66 (0.70) | 0.46 | 0.47 | 0.43 | 0.49 | 0.46 | 0.46 | 0.41 | 0.46 | 0.49 |
18. Can’t put disappointments out of mind | 1.65 (0.77) | 0.41 | 0.40 | 0.42 | 0.29 | 0.44 | 0.37 | 0.40 | 0.41 | 0.54 |
19. Steady person | 1.77 (0.75) | 0.68 | 0.67 | 0.70 | 0.65 | 0.71 | 0.61 | 0.62 | 0.68 | 0.57 |
20. State of tension or turmoil | 1.50 (0.69) | 0.55 | 0.54 | 0.55 | 0.47 | 0.57 | 0.52 | 0.53 | 0.55 | 0.55 |
Model Fit Indices | ||||||||||
Comparative Fit Index (CFI) | — | 0.87 | 0.87 | 0.88 | 0.84 | 0.89 | 0.82 | 0.87 | 0.88 | 0.71 |
Tucker-Lewis Fit Index (TLI) | — | 0.86 | 0.85 | 0.86 | 0.82 | 0.87 | 0.80 | 0.85 | 0.86 | 0.67 |
Standardized Root Mean Square Residual (SRMR) | — | 0.05 | 0.06 | 0.05 | 0.07 | 0.05 | 0.07 | 0.06 | 0.05 | 0.10 |
Cronbach’s alpha | — | 0.91 | 0.90 | 0.91 | 0.89 | 0.91 | 0.89 | 0.90 | 0.91 | 0.86 |
| ||||||||||
Visit 1 (8–13 weeks) (N=8,723) | ||||||||||
Factor 1: Absent | ||||||||||
1. Pleasant | 1.77 (0.74) | 0.71 | 0.71 | 0.69 | 0.72 | 0.71 | 0.70 | 0.71 | 0.71 | 0.60 |
3. Satisfied with myself | 1.82 (0.78) | 0.72 | 0.71 | 0.75 | 0.71 | 0.75 | 0.71 | 0.73 | 0.73 | 0.64 |
6. Rested | 2.51 (0.79) | 0.51 | 0.51 | 0.51 | 0.50 | 0.51 | 0.49 | 0.55 | 0.51 | 0.47 |
7. Calm, cool, and collected | 2.07 (0.78) | 0.67 | 0.67 | 0.66 | 0.70 | 0.67 | 0.68 | 0.66 | 0.67 | 0.59 |
10. Happy | 1.55 (0.70) | 0.75 | 0.74 | 0.76 | 0.74 | 0.76 | 0.71 | 0.74 | 0.75 | 0.63 |
13. Secure | 1.69 (0.79) | 0.72 | 0.71 | 0.73 | 0.68 | 0.73 | 0.70 | 0.74 | 0.72 | 0.75 |
14. Make decisions easily | 2.16 (0.85) | 0.56 | 0.56 | 0.57 | 0.57 | 0.56 | 0.57 | 0.56 | 0.56 | 0.49 |
16. Content | 1.72 (0.79) | 0.76 | 0.75 | 0.79 | 0.70 | 0.80 | 0.69 | 0.76 | 0.77 | 0.72 |
19. Steady person | 1.77 (0.75) | 0.70 | 0.70 | 0.71 | 0.68 | 0.73 | 0.64 | 0.63 | 0.70 | 0.66 |
Factor 2: Present | ||||||||||
2. Nervous and restless | 1.79 (0.70) | 0.55 | 0.53 | 0.58 | 0.48 | 0.58 | 0.50 | 0.60 | 0.55 | 0.48 |
4. Wish I could be as happy as others | 1.60 (0.82) | 0.59 | 0.59 | 0.59 | 0.58 | 0.64 | 0.51 | 0.43 | 0.62 | 0.24 |
5. Feel like a failure | 1.22 (0.51) | 0.60 | 0.61 | 0.56 | 0.56 | 0.63 | 0.56 | 0.46 | 0.60 | 0.50 |
8. Difficulties are piling up | 1.40 (0.62) | 0.61 | 0.63 | 0.56 | 0.60 | 0.61 | 0.62 | 0.51 | 0.61 | 0.62 |
9. Worry too much | 1.82 (0.77) | 0.57 | 0.57 | 0.55 | 0.56 | 0.57 | 0.57 | 0.55 | 0.56 | 0.66 |
11. Disturbing thoughts | 1.24 (0.54) | 0.46 | 0.46 | 0.45 | 0.49 | 0.41 | 0.54 | 0.53 | 0.45 | 0.64 |
12. Lack self-confidence | 1.56 (0.70) | 0.53 | 0.53 | 0.54 | 0.49 | 0.56 | 0.58 | 0.54 | 0.53 | 0.64 |
15. Inadequate | 1.40 (0.64) | 0.54 | 0.51 | 0.59 | 0.37 | 0.60 | 0.48 | 0.49 | 0.54 | 0.42 |
17. Unimportant thoughts | 1.66 (0.70) | 0.54 | 0.55 | 0.50 | 0.61 | 0.52 | 0.60 | 0.49 | 0.53 | 0.71 |
18. Can’t put disappointments out of mind | 1.65 (0.77) | 0.49 | 0.50 | 0.49 | 0.43 | 0.50 | 0.49 | 0.51 | 0.49 | 0.55 |
20. State of tension or turmoil | 1.50 (0.69) | 0.64 | 0.64 | 0.64 | 0.59 | 0.65 | 0.63 | 0.65 | 0.64 | 0.64 |
Model Fit Indices | ||||||||||
Comparative Fit Index (CFI) | — | 0.94 | 0.95 | 0.93 | 0.94 | 0.94 | 0.95 | 0.93 | 0.94 | 0.86 |
Tucker-Lewis Fit Index (TLI) | — | 0.93 | 0.94 | 0.92 | 0.93 | 0.93 | 0.94 | 0.92 | 0.93 | 0.84 |
Standardized Root Mean Square Residual (SRMR) | — | 0.03 | 0.03 | 0.04 | 0.04 | 0.03 | 0.03 | 0.04 | 0.03 | 0.07 |
Cronbach’s alpha | ||||||||||
Absent | — | 0.89 | 0.88 | 0.89 | 0.88 | 0.89 | 0.87 | 0.88 | 0.88 | 0.84 |
Present | — | 0.83 | 0.83 | 0.83 | 0.81 | 0.84 | 0.82 | 0.81 | 0.83 | 0.82 |
Spanish Translation of Measures
For Spanish-speaking respondents, we utilized the Mexican-Spanish version of the Perceived Stress Scale [23]. No Spanish version of the Pregnancy Experience Scale was available. To develop the Spanish version for this study, the PES was translated and then the translation was reviewed by an American Translators Association-certified translator and a language methodologist who made adjustments as needed. The Spanish version of the trait-anxiety subscale of the State-Trait Anxiety Inventory was developed in one of our prior studies based on a series of forward and backward translations which were reviewed by the original scale developer. Analyses of data for the prior study identified two problematic items which were then revised and re-translated before use in the current study.
Psychometric Analyses
For each scale, we conducted a series of item- and scale-level analyses. Descriptive statistics were calculated for each item, including means and standard deviations and item-total correlations. Cronbach’s alphas were calculated to assess the internal consistency reliability of the scales. A common rule of thumb indicates that alphas of 0.70 or higher indicate acceptable internal consistency for group-level comparisons [24]. We conducted confirmatory factor analyses to assess the factorial validity of each scale and determine if the items cluster into factors as expected based on the scale manual or prior research. Values of 0.95 or higher for the comparative fit index (CFI) and Tucker-Lewis Fit Index (TLI) generally indicate good model fit [25] with values of 0.90 or higher indicating acceptable fit. Values of 0.08 or lower for the Standardized Root Mean Square Residual (SRMR) indicate good model fit [25]. For the PSS, which was administered at two time points (Visits 1 and 3), we additionally calculated intra-class correlations (ICCs) to estimate the relationship between scores over time.
In addition to the properties for the overall sample, we compared the psychometric properties of scales stratified by age (< 30 vs. ≥ 30 years), race/ethnicity (non-Hispanic black, non-Hispanic white, Hispanic, and Asian) and language of interview (English and Spanish). When reverse coding was required, original items were used to calculate the means and standard deviations for ease of interpretation, while reverse-coded items were used to calculate the item-total correlations, alphas, and factor analyses. All analyses except the factor analyses were conducted using SAS version 9.3; the factor analyses were conducted using Mplus version 6.12.
RESULTS
Perceived Stress Scale
On average, participants reported the lowest frequency of occurrence for Item 4 (confident about ability to handle personal problems) at Visit 1 (mean=1.85) and Item 10 (felt difficulties were piling up so high that you could not overcome them) at Visit 3 (mean=1.75) (Table 2). In contrast, they reported the greatest frequency for Item 3 (nervous and stressed) (mean=3.00 at Visit 1 and mean=2.77 at Visit 3). The values for the Cronbach’s alphas indicate high internal consistency reliability for the overall sample (alpha=0.88 at Visit 1 and alpha=0.89 at Visit 3). Although the Cronbach’s alphas for the Spanish speakers were lower than those for the other groups, they remain in the acceptable range (alpha=0.79 at Visit 1 and alpha=0.82 at Visit 3).
The one-factor confirmatory factor model fit well in the overall sample, and all items had moderate to high factor loadings (Table 2). The model also fit well for most subgroups except for Spanish-speaking participants at Visit 1 and Asian participants at Visits 1 and 3. Items that had to be reverse coded (items 4, 5, 7, and 8) generally had lower factor loadings, particularly among Spanish speakers. For example, the factor loadings for item 7 (able to control irritations in your life) on the Spanish version of the scale were 0.22 at Visit 1 and 0.17 at Visit 3 (Table 2).
Scores on the PSS were correlated between Visits 1 and 3 with an ICC of 0.60 among the overall sample. ICC values by demographic group are as follows: age less than 30 years (ICC=0.60), age 30 years or older (ICC=0.60), non-Hispanic black (ICC=0.55), non-Hispanic white (ICC=0.64), Hispanic (ICC=0.52), Asian (ICC=0.45), English language (ICC=0.61), and Spanish language (ICC=0.48).
Pregnancy Experience Scale
Item 7 (spiritual feelings about being pregnant) had the lowest mean of the Uplifts items (mean=2.86), while Item 1 (how much the baby is moving) had the highest mean (mean=3.67). On the Hassles subscale, Item 9 (concerns about physical symptoms) had the lowest mean (mean=1.53), and Item 7 (thinking about your labor and delivery) had the highest mean (mean=2.09). The item-total correlations were high for all items (not shown) and the alphas indicated high internal consistency reliability overall and across all subgroups (Table 3). The two-factor confirmatory factor model also fit well across all groups. Only one item, item 10 (discussions with spouse about pregnancy/childbirth issues), had a factor loading less than 0.40, with a value of 0.34 among Spanish speakers.
State Trait Anxiety Inventory (Trait-Anxiety Subscale)
Table 4 presents the means, standard deviations, and factor loadings for the STAI trait-anxiety subscale. On average, Item 5 (feel like a failure) occurred least frequently (mean=1.22), while Item 6 (rested) occurred most frequently (mean=2.51). All item-total correlations, except Item 11 (disturbing thoughts), were higher than 0.40; Item 11 was close to the 0.40 cut-off with an item-total correlation of 0.38 (not shown). Cronbach’s alphas indicated good internal consistency across all groups with values of 0.86 or higher (Table 4).
The one-factor confirmatory factor analyses did not fit the data well with values less than 0.90 for the CFI and TLI (Table 4). The scale scoring manual indicates that the scale developers also tested the following two factors based on the direction of the wording of the items: Trait-Anxiety—Present (positively worded items) and Trait-Anxiety—Absent (negatively worded items) [22]. Because respondents often do not use the entire response scale in the same way, it is common to find lower correlations among positively and negatively worded items than items worded in the same direction. Based on the model fit indices, two-factor models with one factor of positively-worded items (Trait Anxiety – Present) and another factor with negatively-worded items (Trait Anxiety – Absent) provided a much better fit to the data for all but the participants responding to the Spanish instrument (Table 4). One item in particular, item 4 (wish I could be as happy as others) had a low factor loading among Spanish speakers (loading=0.24). The two factors were highly correlated (r=0.75 in overall sample).
Relationship among Scales
Table 5 presents correlations among the scales. The PSS was positively correlated with both the STAI trait-anxiety subscale (r=0.68 at Visit 1 and r=0.62 at Visit 3) and the PES Hassles subscale (r=0.30 at Visit 3). The STAI Trait-Anxiety subscale and the PES Hassles subscale were also positively correlated with each other (r=0.38). As expected, the correlations between the PES Uplifts subscale and the other scales were negative, although the magnitudes were quite small.
Table 5.
Scale | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
---|---|---|---|---|---|---|---|
Visit 1 (8–13 weeks) | |||||||
1. Perceived Stress Scale | 1.00 | ||||||
2. State Trait Anxiety Inventory (Trait-Anxiety) | 0.68* | 1.00 | |||||
3. State Trait Anxiety Inventory (Trait-Anxiety—Absent) | 0.62* | 0.91* | 1.00 | ||||
4. State Trait Anxiety Inventory (Trait-Anxiety—Present) | 0.61* | 0.89* | 0.63* | 1.00 | |||
Visit 3 (22–29 weeks) | |||||||
5. Perceived Stress Scale | 0.62* | 0.59* | 0.54* | 0.52* | 1.00 | ||
6. Pregnancy Experience Scale—Uplifts (frequency) | −0.14* | −0.17* | −0.17* | −0.14* | −0.15* | 1.00 | |
7. Pregnancy Experience Scale—Hassles (frequency) | 0.30* | 0.38* | 0.34* | 0.34* | 0.39* | 0.01 | 1.00 |
p < 0.001
DISCUSSION
Overall, the three stress and anxiety scales performed well among this sample of nulliparous women, demonstrating good internal consistency reliability as measured by Cronbach’s alphas. The confirmatory factor analyses supported the one-factor structure of the PSS and a two-factor structure for the PES reflecting its two components (hassles and uplifts). However, contrary to expectations, the one-factor solution for the STAI Trait-Anxiety subscale did not fit the data well, although a two-factor solution splitting the items into one factor with negatively worded items and another with positively worded items fit well, consistent with results reported by the scale developer [22]. This finding suggests that the two-factor division may simply be an artifact of the direction of the wording of the items representing a spurious “difficulty” or “item direction” factor [26, 27] and do not imply that the scale itself if psychometrically invalid. The large Cronbach’s alpha for the overall scale and the strong correlation between the two factors suggest that the items are measuring the same construct and support the use of the overall scale.
Subgroup analyses revealed a small number of items within the scales that did not perform well among participants completing the interviews in Spanish. For example, Item 4 (confident about ability to handle personal problems) and Item 7 (able to control irritations in your life) on the PSS both had low factor loadings among Spanish-speaking respondents, suggesting that perhaps there may have been difficulties in translating these items or the construct has a different meaning among Spanish speakers. Cognitive interviewing [28] with Spanish-speaking women could be conducted to explore potential reasons for the poor performance of these items, such as problems with item wording. However, it is important to note that this study included only a small number of participants who completed the interview in Spanish (N=218). Further research is needed to examine the performance of these items in a larger sample of Spanish speakers.
Strengths of this study include the large sample of nulliparous women and inclusion of multiple stress- and anxiety-related measures, as well as repeated measurements for the PSS. A limitation of the study is the inclusion of only nulliparous women, which did not permit comparisons of psychometric properties of the measures by parity.
Based on the overall demonstrated reliability and validity of the measures, they appear to be valid tools for evaluating stress and anxiety among nulliparous women in clinical settings. Mothers reporting greater pregnancy-specific stress are less likely to exercise and take prenatal vitamins and are more likely to smoke and have unhealthy eating habits [2]. Identifying these mothers early would allow for the implementation of interventions to help reduce maternal stress and anxiety, modify health-related behaviors, and possibly improve pregnancy outcomes.
The potential impact of these types of interventions is likely even greater given the correlation of PSS scores throughout pregnancy from Visit 1 (8 through 13 weeks gestation) through Visit 3 (22 through 29 weeks gestation) for each of the demographic groups, suggesting women who experience greater stress earlier in their pregnancy tend to have stress later in pregnancy. These results are similar to the findings of DiPietro and colleagues [15] who reported stability of PSS and STAI trait-anxiety scores from 28 to 38 weeks of pregnancy. Interventions could potentially alter this trajectory by providing mothers with needed resources and effective stress management strategies, thereby reducing stress and improving longer-term outcomes among at-risk mothers and their babies.
CURRENT KNOWLEDGE ON THE SUBJECT.
Stress and anxiety during pregnancy have been associated with poor outcomes among mothers and their children.
Trajectories of stress through the prenatal to postpartum periods have been shown to differ among women based on parity.
Reliable and valid measures are needed to identify pregnant women experiencing high levels of stress and anxiety.
The psychometric properties of most stress and anxiety measures have not been assessed among nulliparous women.
WHAT THIS STUDY ADDS.
Two stress-related measures (Perceived Stress Scale and Pregnancy Experience Scale) and one anxiety measure (State-Trait Anxiety Inventory-Trait-Anxiety subscale) demonstrated good psychometric properties among a large sample of nulliparous women.
The measures generally perform0d well across age and racial/ethnic subgroups.
Scores on the Perceived Stress Scale were positively correlated between 8–13 weeks gestation and 22–29 weeks gestation.
Footnotes
DECLARATION OF INTEREST STATEMENT
The authors report no conflicts of interest. This study is supported by grant funding from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD): U10 HD063036, RTI International; U10 HD063072, Case Western Reserve University; U10 HD063047, Columbia University; U10 HD063037, Indiana University; U10 HD063041, University of Pittsburgh; U10 HD063020, Northwestern University; U10 HD063046, University of California Irvine; U10 HD063048, University of Pennsylvania; and U10 HD063053, University of Utah.
Contributor Information
Carla M. Bann, Division of Statistical and Data Sciences, RTI International, Research Triangle Park, NC.
Corette B. Parker, Biostatistics and Epidemiology Division, RTI International, Research Triangle Park, NC.
William A. Grobman, Department of Maternal & Fetal Medicine, Northwestern University, Chicago, IL.
Marian Willinger, Eunice Kennedy Shriver National Institute of Child Health and Human Development, Bethesda, MD.
Hyagriv N. Simhan, Department of Maternal & Fetal Medicine, University of Pittsburgh, Pittsburgh, PA.
Deborah A. Wing, Department of Maternal Fetal Medicine, University of California at Irvine, Irvine, CA.
David M. Haas, Department of Obstetrics & Gynecology, Indiana University, Indianapolis, IN.
Robert M. Silver, Department of Obstetrics & Gynecology, University of Utah, Salt Lake City, UT.
Samuel Parry, Department of Maternal & Fetal Medicine, University of Pennsylvania, Philadelphia, PA.
George R. Saade, Department of Maternal & Fetal Medicine, University of Texas Medical Branch-Galveston, Galveston, TX.
Ronald J. Wapner, Department of Maternal Fetal Medicine, Columbia University, New York, NY.
Michal A. Elovitz, Department of Maternal & Fetal Medicine, University of Pennsylvania, Philadelphia, PA.
Emily S. Miller, Department of Obstetrics & Gynecology, Northwestern University, Chicago, IL.
Uma M. Reddy, Eunice Kennedy Shriver National Institute of Child Health and Human Development, Bethesda, MD.
References
- 1.Dole N, Savitz DA, Hertz-Picciotto I, Siega-Riz AM, McMahon MJ, Buekens P. Maternal stress and preterm birth. Am J Epidemiol. 2003;157:14–24. doi: 10.1093/aje/kwf176. [DOI] [PubMed] [Google Scholar]
- 2.Lobel M, Cannella DL, Graham JE, DeVincent C, Schneider J, Meyer BA. Pregnancy-specific stress, prenatal health behaviors, and birth outcomes. Health Psychol. 2008;27(5):604–615. doi: 10.1037/a0013242. [DOI] [PubMed] [Google Scholar]
- 3.Paarlberg KM, Vingerhoets A, Passchier J, Dekker G, van Geijn H. Psychosocial factors and pregnancy outcome: A review with emphasis on methodological issues. J Psychosom Res. 1995;39:563–595. doi: 10.1016/0022-3999(95)00018-6. [DOI] [PubMed] [Google Scholar]
- 4.Rondo PH, Ferreira RF, Nogueira F, Ribeiro MC, Lobert H, Artes R. Maternal psychological stress and distress as predictors of low birth weight, prematurity, and intrauterine growth retardation. Eur J Clin Nutr. 2003;57(2):266–272. doi: 10.1038/sj.ejcn.1601526. [DOI] [PubMed] [Google Scholar]
- 5.Hack M, Taylor G, Drotar D, Schluchter M, Cartar L, Andreias L, Wilson-Costello D, Klein N. Chronic conditions, functional limitations, and special health care needs of school-aged children born with extremely low birth weight in the 1990s. JAMA. 2005;294(3):318–325. doi: 10.1001/jama.294.3.318. [DOI] [PubMed] [Google Scholar]
- 6.Vohr BR, Wright LL, Dusick AM, Mele L, Verter J, Steichen JJ, Simon NP, Wilson DC, Broyles S, Bauer CR, et al. Neurodevelopmental and functional outcomes of extremely low birth weight infants in the National Institute of Child Health and Human Development Neonatal Research Network, 1993–1994. Pediatrics. 2000;105(6):1216–1226. doi: 10.1542/peds.105.6.1216. [DOI] [PubMed] [Google Scholar]
- 7.Grant KA, McMahon C, Austin MP. Maternal anxiety during the transition to parenthood: A prospective study. J Affect Disord. 2008;108:101–111. doi: 10.1016/j.jad.2007.10.002. [DOI] [PubMed] [Google Scholar]
- 8.Austin MP, Tully L, Parker G. Examining the relationship between antenatal anxiety and postnatal depression. J Affect Disord. 2007;101(1–3):169–174. doi: 10.1016/j.jad.2006.11.015. [DOI] [PubMed] [Google Scholar]
- 9.Heron J, O’Connor TG, Evans J, Golding J, Glover V. The course of anxiety and depression through pregnancy and the postpartum in a community sample. J Affect Disord. 2004;80:65–73. doi: 10.1016/j.jad.2003.08.004. [DOI] [PubMed] [Google Scholar]
- 10.Matthey S, Barnett B, Howie P, Kavanagh DJ. Diagnosing postpartum depression in mothers and fathers: whatever happened to anxiety? J Affect Disord. 2003;74:139–147. doi: 10.1016/s0165-0327(02)00012-5. [DOI] [PubMed] [Google Scholar]
- 11.O’Connor TG, Heron J, Golding J, Beveridge M, Glover V. Maternal antenatal anxiety and children’s behavioural/emotional problems at four years: Report from the Avon Longitudinal Study of Parents and Children. Br J Psychiatry. 2002;180:502–508. doi: 10.1192/bjp.180.6.502. [DOI] [PubMed] [Google Scholar]
- 12.Darvill R, Skirton H, Farrand P. Psychological factors that impact on women’s experiences of first-time motherhood: A qualitative study of the transition. Midwifery. 2010;26:357–366. doi: 10.1016/j.midw.2008.07.006. [DOI] [PubMed] [Google Scholar]
- 13.Messias DKH, DeJoseph JF. The personal work of a first pregnancy: Transforming identities, relationships, and women’s work. Women Health. 2007;45(4):41–64. doi: 10.1300/j013v45n04_03. [DOI] [PubMed] [Google Scholar]
- 14.Krieg DB. Does motherhood get easier the second-time around? Examining parenting stress and marital quality among mothers having their first or second child. Parent Sci Pract. 2007;7(2):149–175. [Google Scholar]
- 15.DiPietro JA, Costigan KA, Sipsma HL. Continuity in self-report measures of maternal anxiety, stress, and depressive symptoms from pregnancy through two years postpartum. J Psychosom Obstet Gynecol. 2008b;29(2):115–124. doi: 10.1080/01674820701701546. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Alderice F, Lynn F, Lobel M. A review and psychometric evaluation of pregnancy-specific stress measures. J Psychosom Obstet Gynecol. 2012;33(2):62–77. doi: 10.3109/0167482X.2012.673040. [DOI] [PubMed] [Google Scholar]
- 17.DiPietro JA, Christensen AL, Costigan KA. The pregnancy experience scale-brief version. J Psychosom Obstet Gynecol. 2008a;29(4):262–267. doi: 10.1080/01674820802546220. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Gutteling BM, de Weerth C, Zandbelt N, Mulder EJ, Visser GH, Buitelaar JK. Does maternal prenatal stress adversely affect the child’s learning and memory at age six? J Abnorm Child Psychol. 2006;34:789–798. doi: 10.1007/s10802-006-9054-7. [DOI] [PubMed] [Google Scholar]
- 19.Huizink AC, de Medina PG, Mulder EJ, Visser GH, Buitelaar JK. Psychological measures of prenatal stress as predictors of infant temperament. J Am Acad Child Adolesc Psychiatry. 2002;41:1078–1085. doi: 10.1097/00004583-200209000-00008. [DOI] [PubMed] [Google Scholar]
- 20.Huizink AC, Mulder EJ, Robles de Medina PG, Visser GH, Buitelaar JK. Is pregnancy anxiety a distinctive syndrome? Early Hum Dev. 2004;79:81–91. doi: 10.1016/j.earlhumdev.2004.04.014. [DOI] [PubMed] [Google Scholar]
- 21.Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983;24(4):385–396. [PubMed] [Google Scholar]
- 22.Spielberger CD, Gorsuch RL, Lushene R, Vagg PR, Jacobs GA. Manual for the State-Trait Anxiety Inventory for Adults. Palo Alto, CA: Mind Garden, Inc; 1983. [Google Scholar]
- 23.Ramirez MTG, Hernandez RL. Factor structure of the Perceived Stress Scale (PSS) in a sample from Mexico. Span J Psychol. 2007;10(1):199–206. doi: 10.1017/s1138741600006466. [DOI] [PubMed] [Google Scholar]
- 24.Reeve B, Wyrwich K, Wu A, Velikova G, Terwee C, Snyder C, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 2013;22(8):1889–905. doi: 10.1007/s11136-012-0344-y. [DOI] [PubMed] [Google Scholar]
- 25.Hua L-T, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal. 1999;6(1):1–55. [Google Scholar]
- 26.Gorsuch RL. Factor analysis. 2. Hillsdale, NJ: Lawrence Erlbaum Associates; 1983. [Google Scholar]
- 27.Spector PE, Van Katwyk PT, Brannick MT, Chen PY. When two factors don’t reflect two constructs: how item characteristics can produce artifactual factors. J Manage. 1997;23(5):659–677. [Google Scholar]
- 28.Willis GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications, Inc; 2005. [Google Scholar]